Skip to content
Lime-it.us
  1. Home
  2. Categories
  3. FreeBSD Notes
  4. Bhyve Hypervisor Freebsd ZFS

Bhyve Hypervisor Freebsd ZFS

FreeBSD Notes
  • rickR

    Get updated to at least Freebsd 11.0

    Bhyve will function on previous releases after 10.2, maybe.

    Install bhyve and bhyve grub:

    pkg install vm-bhyve grub2-bhyve

    Load kernel modules:

    add these directives into /boot/loader.conf:

    if_bridge_load=”YES”
if_tap_load=”YES”
nmdm_load=”YES”
vmm_load=”YES”

    We could load the modules manually, however I’ve had issues doing this the first time, but go ahead and try, maybe it will ‘just work’ for you.

    kldload if_bridge if_tap nmdm vmm

    When that does not work, (you might not even know at this point) just reboot and the modules will be loaded at that time.

    Add the following lines to /etc/rc.conf:

    vm_enable=”YES”
vm_dir=”zfs:zroot/vms”
vm_list=””
vm_delay=”5″

    If you’ve not already setup storage, lets do it now. If you’ve not changed the default name for ZFS, it should be ‘zroot’, find out by running:

    zfs list

    On this machine the output is:

    NAME                 USED  AVAIL  REFER  MOUNTPOINT
bootpool             144M  1.72G   142M  /bootpool
zroot               30.8G   860G    88K  /zroot
zroot/ROOT          17.4G   860G    88K  none
zroot/ROOT/default  17.4G   860G  17.4G  /
zroot/tmp           40.3M   860G  40.3M  /tmp
zroot/usr           11.6G   860G    88K  /usr
zroot/usr/home      8.55G   860G  8.55G  /usr/home
zroot/usr/ports     2.45G   860G  2.45G  /usr/ports
zroot/usr/src        633M   860G   633M  /usr/src
zroot/var           10.5M   860G    88K  /var
zroot/var/audit       88K   860G    88K  /var/audit
zroot/var/crash       88K   860G    88K  /var/crash
zroot/var/log        692K   860G   692K  /var/log
zroot/var/mail       244K   860G   244K  /var/mail
zroot/var/tmp       9.33M   860G  9.33M  /var/tmp
zroot/vms           1.62G   860G   500M  /vms
zroot/vms/freebsd1  1.13G   860G  1.13G  /vms/freebsd1

    As we can see, ‘zroot’ is the pool name. !Important If the pool is named something other than ‘zroot’ , you must change the directives we did a minute ago within /etc/rc.conf to reflect the proper zpool.

    Create the storage:

    zfs create -o mountpoint=/vms zroot/vms

    Initialize:

    vm init

    If you get the error message “$vm_enable is not enabled in /etc/rc.conf!” ect… , this is due to the modules not being loaded as described earlier, just reboot and come back. Bookmark this page or check your history.

    Now we copy the templates to the /vms directory we’ve created:

    cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/

    Creating the network bridge VMs will use:

    First lets find out what our inteface is called:

    ifconfig

    In this case the interface is called ‘em0’

    So:

    vm switch create public

    vm switch add public em0

    Edit the template config files to use zvol instead of the IMG or iso You can use the iso, however we have ZFS for a reason don’t we.

    Templates are located where we copied them to earlier: /vms/.templates

    List all config files in the .templates directory:

    ls /vms/.templates

    Here are the default conf files listed:

    alpine.conf		centos7.conf		debian.conf		freebsd-zvol.conf	openbsd.conf		windows.conf
centos6.conf		config.sample		default.conf		netbsd.conf		ubuntu.conf

    Have a look at the debian.conf template:

    loader="grub"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
grub_run_partition="1"
grub_run_dir="/boot/grub"

    You can edit what you wish, but know what your editing, and what it will do.

    To make use of zvol we edit both “disk0_name” & “disk0_dev” from the above to look like this:

    loader="grub"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0"
disk0_dev="sparse-zvol"
grub_run_partition="1"
grub_run_dir="/boot/grub"

    Time to grab an iso, lets use freebsd: Head over to https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/ and grab 11.1

    Copy the URL of the iso Running the following will snatch the image and deposit it locally.

    vm iso https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso

    Time to create the VM: This will create a VM of freebsd named ‘freebsd11’ with a disk size of ten gigs

    vm create -s 10G freebsd11

    Now install the image we just downloaded:

    vm -f install freebsd11 FreeBSD-11.1-RELEASE-amd64-disc1.iso

    You will be asked to select [vt100] for the console type, unless you know what’s up otherwise, choose it. Now continue with installation as you would normally.

    Once install is complete, log in and play, then type:

    poweroff

    List all VM’s:

    vm list

    NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE
freebsd11        default         bhyveload   1      256M      -                    No           Running (1923)

    Start the VM:

    vm start freebsd11

    To destroy the VM:

    vm destroy freebsd11

    To edit the config file for specific VM:

    vm configure freebsd11

    This will open the config file in your default editor.

    0
    • lime-itL

      ISO from DVD to local file

      Linux Systems Guides
      2
      +0
      0 Votes
      2 Posts
      1k Views
      rickR

      FYI for FreeBSD the driver only supports block size chunks, therefore:

      dd if=/dev/cd0 of=/name-the.iso bs=2048
    • rickR

      zpool destroy /dev/ada0 operation not permitted

      FreeBSD Notes
      2
      0 Votes
      2 Posts
      1k Views
      rickR

      If attempting to install FreeBSD on a disk which previously belongs to a ZFS, and you get this error: Before installing, select the option ‘shell’

      Once in the shell, remove geom protections by running:

      sysctl kern.geom.debugflags=0x10

      When your finished, type exit and return to the install / configure screen.

    • rickR

      Blank and Burn ISO Freebsd

      FreeBSD Notes
      1
      +0
      0 Votes
      1 Posts
      816 Views
      No one has replied
    • rickR

      Format USB device under FreeBSD

      FreeBSD Notes
      2
      +0
      0 Votes
      2 Posts
      1k Views
      rickR

      Here is the Freebsd manpage for the newfs command:

      NEWFS(8) FreeBSD System Manager's Manual NEWFS(8) NAME newfs -- construct a new UFS1/UFS2 file system SYNOPSIS newfs [-EJNUjlnt] [-L volname] [-O filesystem-type] [-S sector-size] [-T disktype] [-a maxcontig] [-b block-size] [-c blocks-per-cylinder-group] [-d max-extent-size] [-e maxbpg] [-f frag-size] [-g avgfilesize] [-h avgfpdir] [-i bytes] [-k held-for-metadata-blocks] [-m free-space] [-o optimization] [-p partition] [-r reserved] [-s size] special DESCRIPTION The newfs utility is used to initialize and clear file systems before first use. The newfs utility builds a file system on the specified spe- cial file. (We often refer to the ``special file'' as the ``disk'', although the special file need not be a physical disk. In fact, it need not even be special.) Typically the defaults are reasonable, however newfs has numerous options to allow the defaults to be selectively over- ridden. The following options define the general layout policies: -E Erase the content of the disk before making the filesystem. The reserved area in front of the superblock (for bootcode) will not be erased. This option is only relevant for flash based storage devices that use wear-leveling algorithms. Erasing may take a long time as it writes to every sector on the disk. -J Enable journaling on the new file system via gjournal. See gjournal(8) for details. -L volname Add a volume label to the new file system. -N Cause the file system parameters to be printed out without really creating the file system. -O filesystem-type Use 1 to specify that a UFS1 format file system be built; use 2 to specify that a UFS2 format file system be built. The default format is UFS2. -T disktype For backward compatibility. -U Enable soft updates on the new file system. -a maxcontig Specify the maximum number of contiguous blocks that will be laid out before forcing a rotational delay. The default value is 16. See tunefs(8) for more details on how to set this option. -b block-size The block size of the file system, in bytes. It must be a power of 2. The default size is 32768 bytes, and the smallest allow- able size is 4096 bytes. The optimal block:fragment ratio is 8:1. Other ratios are possible, but are not recommended, and may produce poor results. -c blocks-per-cylinder-group The number of blocks per cylinder group in a file system. The default is to compute the maximum allowed by the other parame- ters. This value is dependent on a number of other parameters, in particular the block size and the number of bytes per inode. -d max-extent-size The file system may choose to store large files using extents. This parameter specifies the largest extent size that may be used. The default value is the file system blocksize. It is presently limited to a maximum value of 16 times the file system blocksize and a minimum value of the file system blocksize. -e maxbpg Indicate the maximum number of blocks any single file can allo- cate out of a cylinder group before it is forced to begin allo- cating blocks from another cylinder group. The default is about one quarter of the total blocks in a cylinder group. See tunefs(8) for more details on how to set this option. -f frag-size The fragment size of the file system in bytes. It must be a power of two ranging in value between blocksize/8 and blocksize. The default is 4096 bytes. -g avgfilesize The expected average file size for the file system. -h avgfpdir The expected average number of files per directory on the file system. -i bytes Specify the density of inodes in the file system. The default is to create an inode for every (2 * frag-size) bytes of data space. If fewer inodes are desired, a larger number should be used; to create more inodes a smaller number should be given. One inode is required for each distinct file, so this value effectively specifies the average file size on the file system. -j Enable soft updates journaling on the new file system. This flag is implemented by running the tunefs(8) utility found in the user's $PATH. -k held-for-metadata-blocks Set the amount of space to be held for metadata blocks in each cylinder group. When set, the file system preference routines will try to save the specified amount of space immediately fol- lowing the inode blocks in each cylinder group for use by meta- data blocks. Clustering the metadata blocks speeds up random file access and decreases the running time of fsck(8). By default newfs sets it to half of the space reserved to minfree. -l Enable multilabel MAC on the new file system. -m free-space The percentage of space reserved from normal users; the minimum free space threshold. The default value used is defined by MINFREE from <ufs/ffs/fs.h>, currently 8%. See tunefs(8) for more details on how to set this option. -n Do not create a .snap directory on the new file system. The resulting file system will not support snapshot generation, so dump(8) in live mode and background fsck(8) will not function properly. The traditional fsck(8) and offline dump(8) will work on the file system. This option is intended primarily for memory or vnode-backed file systems that do not require dump(8) or fsck(8) support. -o optimization (space or time). The file system can either be instructed to try to minimize the time spent allocating blocks, or to try to mini- mize the space fragmentation on the disk. If the value of min- free (see above) is less than 8%, the default is to optimize for space; if the value of minfree is greater than or equal to 8%, the default is to optimize for time. See tunefs(8) for more details on how to set this option. -p partition The partition name (a..h) you want to use in case the underlying image is a file, so you do not have access to individual parti- tions through the filesystem. Can also be used with a device, e.g., newfs -p f /dev/da1s3 is equivalent to newfs /dev/da1s3f. -r reserved The size, in sectors, of reserved space at the end of the parti- tion specified in special. This space will not be occupied by the file system; it can be used by other consumers such as geom(4). Defaults to 0. -s size The size of the file system in sectors. This value defaults to the size of the raw partition specified in special less the reserved space at its end (see -r). A size of 0 can also be used to choose the default value. A valid size value cannot be larger than the default one, which means that the file system cannot extend into the reserved space. -t Turn on the TRIM enable flag. If enabled, and if the underlying device supports the BIO_DELETE command, the file system will send a delete request to the underlying device for each freed block. The trim enable flag is typically set when the underlying device uses flash-memory as the device can use the delete command to pre-zero or at least avoid copying blocks that have been deleted. The following options override the standard sizes for the disk geometry. Their default values are taken from the disk label. Changing these defaults is useful only when using newfs to build a file system whose raw image will eventually be used on a different type of disk than the one on which it is initially created (for example on a write-once disk). Note that changing any of these values from their defaults will make it impos- sible for fsck(8) to find the alternate superblocks if the standard superblock is lost. -S sector-size The size of a sector in bytes (almost never anything but 512). EXAMPLES newfs /dev/ada3s1a Creates a new ufs file system on ada3s1a. The newfs utility will use a block size of 32768 bytes, a fragment size of 4096 bytes and the largest possible number of blocks per cylinders group. These values tend to pro- duce better performance for most applications than the historical defaults (8192 byte block size and 1024 byte fragment size). This large fragment size may lead to much wasted space on file systems that contain many small files. SEE ALSO fdformat(1), geom(4), disktab(5), fs(5), camcontrol(8), dump(8), dumpfs(8), fsck(8), gpart(8), gjournal(8), growfs(8), gvinum(8), makefs(8), mount(8), tunefs(8) M. McKusick, W. Joy, S. Leffler, and R. Fabry, "A Fast File System for UNIX", ACM Transactions on Computer Systems 2, 3, pp 181-197, August 1984, (reprinted in the BSD System Manager's Manual).
    • rickR

      List block devices FreeBSD

      FreeBSD Notes
      1
      +0
      0 Votes
      1 Posts
      793 Views
      No one has replied
    • rickR

      ssh_copy_id Freebsd

      FreeBSD Notes
      2
      0 Votes
      2 Posts
      1k Views
      rickR

      The scp way:

      You could type the path to id_rsa.pub, or cd into the users home directory:

      scp id_rsa.pub user@ipaddress:.ssh/authorized_keys
    • rickR

      Openvpn client Freebsd

      FreeBSD Notes
      2
      0 Votes
      2 Posts
      1k Views
      rickR

      Hint:

      kldload if_tun
    • rickR

      rc.conf read only

      FreeBSD Notes
      1
      0 Votes
      1 Posts
      2k Views
      No one has replied
  • rickR

    Alternatively or in addition to the above to install bhyve:

    pkg install vm-bhyve bhyve-firmware bhyve-rc-3 grub2-bhyve

    Output:

    To ensure binaries built with this toolchain find appropriate versions of the necessary run-time libraries, you may want to link using

    -Wl,-rpath=/usr/local/lib/gcc48

    For ports leveraging USE_GCC, USES=compiler, or USES=fortran this happens transparently.

    ===> NOTICE:

    This port is deprecated; you may wish to reconsider installing it:

    Unsupported by upstream. Use GCC 6 or newer instead… Message from vm-bhyve-1.1.8_1:

    To enable vm-bhyve, please add the following lines to /etc/rc.conf, depending on whether you are using ZFS storage or not. Please note that the directory or dataset specified should already exist.

    vm_enable="YES"
vm_dir="zfs:pool/dataset"

    OR

    vm_enable="YES"
vm_dir="/directory/path"

    Then run :

    vm init

    If upgrading from 1.0 or earlier, please note that the ‘guest’ configuration option is no longer used.

    Guests that are not using UEFI boot will need either loader=“grub” or loader=“bhyveload” in their configuration in order to make sure the correct loader is used.

    Message from bhyve-rc-3:

    Configuration is done completely though rc.conf. The rc script won’t touch any devices for you (neither disk, nor tap) so you need to make sure all of those have been initialized properly.

    General setup:

    kldload vmm
net.link.tap.up_on_open=1

    Make it persistent:

    echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
cat >> /boot/loader.conf << EOF
vmm_load="YES"
EOF

    Minimal example:

    cat >> /etc/rc.conf << EOF
cloned_interfaces="tap0 bridge0"
bhyve_enable="YES"
bhyve_diskdev="/dev/zvol/anything/bhyve/virt"
EOF

    ifconfig tap0 create
ifconfig bridge0 create

    service bhyve start
tmux list-sessions
tmux attach -t bhyve
service bhyve status
service bhyve stop

    Multi profile configuration example:

    cat >> /etc/rc.conf << EOF
cloned_interfaces="tap0 tap1 bridge0"
bhyve_enable="YES"
bhyve_profiles="virt1 virt2"
bhyve_virt1_diskdev="/dev/zvol/anything/bhyve/virt1"

bhyve_virt2_tapdev="tap1"
bhyve_virt2_diskdev="/dev/zvol/anything/bhyve/virt2"
bhyve_virt2_memsize="8192"
bhyve_virt2_ncpu="4"
EOF

    ifconfig tap0 create
ifconfig tap1 create
ifconfig bridge0 create

    service bhyve start # start all
service bhyve start virt2 # start individual
tmux attach -t bhyve_virt1
tmux attach -t bhyve_virt1
service bhyve stop virt2 # stop individual
service bhyve stop # stop all

    (by default ctrl-b d detaches from tmux).

    0

FreeBSD Notes
  • rickR
    rick

    Print command name running on port

    sudo lsof -iTCP -sTCP:LISTEN -n -P | awk 'NR>1 {print $9, $1, $2}' | sed 's/.*://' | while read port process pid; do echo "Port $port: $(ps -p $pid -o command= | sed 's/^-//') (PID: $pid)"; done | sort -n
    read more
  • rickR
    rick

    Locate devices:

    camcontrol devlist

    Output; in this case only:

    <ST3500418AS CC35> at scbus3 target 0 lun 0 (pass0,ada0) <ST500DM002-1BD142 KC45> at scbus5 target 0 lun 0 (pass1,ada1) <AHCI SGPIO Enclosure 1.00 0001> at scbus9 target 0 lun 0 (ses0,pass2) <Generic STORAGE DEVICE 1532> at scbus10 target 0 lun 0 (da0,pass3) <Generic STORAGE DEVICE 1532> at scbus10 target 0 lun 1 (da1,pass4)

    Where ada0 and ada1 are mechanical drives, da0 is a miniSD card in a USB enclosure da1

    Or to print all partitions:

    gpart show

    Output (after formatting USB device):

    => 63 976773105 ada0 MBR (466G) 63 1 - free - (512B) 64 976773096 1 freebsd [active] (466G) 976773160 8 - free - (4.0K) => 0 976773096 ada0s1 BSD (466G) 0 4194304 1 freebsd-zfs (2.0G) 4194304 4194304 2 freebsd-swap (2.0G) 8388608 968384480 4 freebsd-zfs (462G) 976773088 8 - free - (4.0K) => 63 976773105 ada1 MBR (466G) 63 1 - free - (512B) 64 976773096 1 freebsd [active] (466G) 976773160 8 - free - (4.0K) => 0 976773096 ada1s1 BSD (466G) 0 4194304 1 freebsd-zfs (2.0G) 4194304 4194304 2 freebsd-swap (2.0G) 8388608 968384480 4 freebsd-zfs (462G) 976773088 8 - free - (4.0K) => 32 2012128 da0 MBR (983M) 32 2012128 1 fat32 (982M)

    List partitions on dev da0:

    gpart show da0

    Delete existing partitions:

    gpart delete -i da0

    Destroy label:

    gpart destroy da0

    Create new mbr spanning entire disk:

    gpart create -s mbr da0

    Create new fat32 partition spanning entire disk:

    gpart add -t fat32 da0

    Initialize fat32 file system:

    newfs_msdos -F32 /dev/da0s1

    Lets break something!

    Don’t do any of this unless you are prepared to break it all, or better yet, you read the man pages and find out what they actually do, very useful tools however.

    I’m just making notes from other notes, various resources on the net.

    gpart destroy -F da0

    Zero out the drive === !!!Don’t do this jazz regularly on any USB!!! The type of memory has a finite read/write number===

    dd if=/dev/zero of=/dev/da0 bs=2m count=1

    Format the drive

    newfs_msdos -F32 /dev/da0s1
    read more
  • rickR
    rick

    Alternatively or in addition to the above to install bhyve:

    pkg install vm-bhyve bhyve-firmware bhyve-rc-3 grub2-bhyve

    Output:

    To ensure binaries built with this toolchain find appropriate versions of the necessary run-time libraries, you may want to link using

    -Wl,-rpath=/usr/local/lib/gcc48

    For ports leveraging USE_GCC, USES=compiler, or USES=fortran this happens transparently.

    ===> NOTICE:

    This port is deprecated; you may wish to reconsider installing it:

    Unsupported by upstream. Use GCC 6 or newer instead… Message from vm-bhyve-1.1.8_1:

    To enable vm-bhyve, please add the following lines to /etc/rc.conf, depending on whether you are using ZFS storage or not. Please note that the directory or dataset specified should already exist.

    vm_enable="YES" vm_dir="zfs:pool/dataset"

    OR

    vm_enable="YES" vm_dir="/directory/path"

    Then run :

    vm init

    If upgrading from 1.0 or earlier, please note that the ‘guest’ configuration option is no longer used.

    Guests that are not using UEFI boot will need either loader=“grub” or loader=“bhyveload” in their configuration in order to make sure the correct loader is used.

    Message from bhyve-rc-3:

    Configuration is done completely though rc.conf. The rc script won’t touch any devices for you (neither disk, nor tap) so you need to make sure all of those have been initialized properly.

    General setup:

    kldload vmm net.link.tap.up_on_open=1

    Make it persistent:

    echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf cat >> /boot/loader.conf << EOF vmm_load="YES" EOF

    Minimal example:

    cat >> /etc/rc.conf << EOF cloned_interfaces="tap0 bridge0" bhyve_enable="YES" bhyve_diskdev="/dev/zvol/anything/bhyve/virt" EOF ifconfig tap0 create ifconfig bridge0 create service bhyve start tmux list-sessions tmux attach -t bhyve service bhyve status service bhyve stop

    Multi profile configuration example:

    cat >> /etc/rc.conf << EOF cloned_interfaces="tap0 tap1 bridge0" bhyve_enable="YES" bhyve_profiles="virt1 virt2" bhyve_virt1_diskdev="/dev/zvol/anything/bhyve/virt1" bhyve_virt2_tapdev="tap1" bhyve_virt2_diskdev="/dev/zvol/anything/bhyve/virt2" bhyve_virt2_memsize="8192" bhyve_virt2_ncpu="4" EOF ifconfig tap0 create ifconfig tap1 create ifconfig bridge0 create service bhyve start # start all service bhyve start virt2 # start individual tmux attach -t bhyve_virt1 tmux attach -t bhyve_virt1 service bhyve stop virt2 # stop individual service bhyve stop # stop all

    (by default ctrl-b d detaches from tmux).

    read more
  • rickR
    rick

    If attempting to install FreeBSD on a disk which previously belongs to a ZFS, and you get this error: Before installing, select the option ‘shell’

    Once in the shell, remove geom protections by running:

    sysctl kern.geom.debugflags=0x10

    When your finished, type exit and return to the install / configure screen.

    read more
  • rickR
    rick

    Which means geom is protecting the disk.

    Running the following clears the protection:

    sysctl kern.geom.debugflags=0x10

    Output:

    kern.geom.debugflags: 0 -> 16

    Clearing MBR and partitions:

    dd if=/dev/zero of=/dev/ada0 bs=512 count=1 conv=notrunc
    read more