ZFS Listing Pools

ZFS has been amazing - the ability to snapshot, rollback, clone and create incremental updates is amazing!

When you're using ZFS, you may want to look at and sort your snapshots. But first, you may need to know what pools you have zpool list.

ZFS Listing Datasets

Once you know the pool you want see the datasets - well that's just zfs list rpool (or bpool, but if you're using ZFS on Ubuntu 22.04 you probably are more interested in rpool).

ZFS Listing Snapshots

ZFS allows you to select what entity types you'd like returned with the -t flag. By passing -t snapshot you list ZFS snapshots.

For example, zfs list  -t snapshot -r rpool

Further, you can sort the snapshots by, say, name with zfs list  -t snapshot -s name -r rpool

If you have tens of thousands of snapshots (trust me, it happens) you might want to limit the fields that are returned (for instance, if you have hundreds of thousands of snapshots it might take minutes for your OS to calculate the available and used space of all snapshots.

So - you can speed up listing the snapshots by only returning the name of the snapshot (with the -o name option).

zfs list  -t snapshot -o name -s name -r rpool

Find the dataset with the most snapshots

If you want to find datasets with the most snapshots, see my post here:

ZFS Finding Datasets with the most snapshots

Following up on Removing old ZFS snapshots, I had way too many snapshots... These commands will find all snapshots within the pool tank and count them by dataset.

/dev/stderrroot

The three main steps are

1. Resize the disk in the hypervisor
2. Update the partition table to take up the full disk
3. Increase the size of the filesystem to take up the full partition

Resize the disk in the hypervisor

First, you need to increase the size of the disk in whatever virtualization software you're using.

In Proxmox, select the VM, go to the Hardware tab, select the drive and hit "Disk Action." Then hit "Resize" and enter the number of GB you'd like to add to the disk. Note, you can only increase the size, not decrease it, so don't go too crazy! I just need a little room so I'm adding 4GB.

Update the partition table

lsblk
df -h

fdisk /dev/sda

You may see a warning like this:

This disk is currently in use - repartitioning is probably a bad idea.
It's recommended to umount all file systems, and swapoff all swap
partitions on this disk.

Again, continue at your own risk, and only if your data is backed up.

># Command (m for help): p _{# select p for partitions to see your partitions - again make SURE you know what partition you want to resize}

Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7BB1AE60-B6B0-6743-6C49-1ACCAD9BF33C

Device       Start      End  Sectors  Size Type
/dev/sda1     2048  1050623  1048576  512M BIOS boot
/dev/sda2  1050624 12580863 11530240  5.5G Linux filesystem

># Command (m for help): d _{# select d for delete}

># Partition number (1,2, default 2): 2 _{# select 2 to delete partition 2}

># Partition 2 has been deleted.

># Command (m for help): n _{# select n to create a new partition}

># Partition number (2-128, default 2): 2 _{# 2 to number the new partition 2}

># First sector (1050624-20971486, default 1050624):
># Last sector, +/-sectors or +/-size{K,M,G,T,P} (1050624-20971486, default 20969471): ENTER _{just hit the enter key to use the default}

># Created a new partition 2 of type 'Linux filesystem' and of size 9.5 GiB.
># Partition #2 contains a ext4 signature.

># Do you want to remove the signature? [Y]es/[N]o: n _{n to leave ext4 signature as is - we don't want to delete anything, just make it larger!}

># Command (m for help): p _{# select p for partitions to see your partitions again - make sure the starting location of the second partition is where it was when you first ran p and that the first partition is unchanged}  

Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7BB1AE60-B6B0-6743-6C49-1ACCAD9BF33C

Device       Start      End  Sectors  Size Type
/dev/sda1     2048  1050623  1048576  512M BIOS boot
/dev/sda2  1050624 20969471 19918848  9.5G Linux filesystem

_{And only when your SURE  things line up}

># Command (m for help): w _{# select w to write the new partition table}

># Command (m for help): d _{# select d for delete}

lsblk

Increase the size of the filesystem to take up the full partition

resize2fs /dev/sda2

df -h

A lot of this information came from AskUbuntu, but was modified slightly for Arch Linux. Please see below for the Ubuntu directions:

How can I resize an ext root partition at runtime?

How can I increase the size of the root partition of a system at runtime? I have a partition that is not allocated after the root partition (which is also ext4), how can I add that unallocated sp...

Ask UbuntuBonboBingo

Luckily, we can activate the keybindings on FreeBSD/Truenas that we're used to on other platforms.

Just add the keybindings to ~/.inputrc and then source it in ~/.zshrc!

setopt interactivecomments;

case "${TERM}" in
  cons25*|linux) # plain BSD/Linux console
    bindkey '\e[H'    beginning-of-line   # home \
    bindkey '\e[F'    end-of-line         # end  \
    bindkey '\e[5~'   delete-char         # delete\
    bindkey '[D'      emacs-backward-word # esc left\
    bindkey '[C'      emacs-forward-word  # esc right\
    ;;
  *rxvt*) # rxvt derivatives
    bindkey '\e[3~'   delete-char         # delete\
    bindkey '\eOc'    forward-word        # ctrl right\
    bindkey '\eOd'    backward-word       # ctrl left\
    # workaround for screen + urxvt\
    bindkey '\e[7~'   beginning-of-line   # home\
    bindkey '\e[8~'   end-of-line         # end\
    bindkey '^[[1~'   beginning-of-line   # home\
    bindkey '^[[4~'   end-of-line         # end\
    ;;
  *xterm*) # xterm derivatives
    bindkey '\e[H'    beginning-of-line   # home\
    bindkey '\e[F'    end-of-line         # end\
    bindkey '\e[3~'   delete-char         # delete\
    bindkey '\e[3;5~' delete-word         # delete-word\
    bindkey '\e[1;5C' forward-word        # ctrl right\
    bindkey '\e[1;5D' backward-word       # ctrl left\
    # workaround for screen + xterm\
    bindkey '\e[1~'   beginning-of-line   # home\
    bindkey '\e[4~'   end-of-line         # end\
    ;;
  screen)
    bindkey '^[[1~'   beginning-of-line   # home\
    bindkey '^[[4~'   end-of-line         # end\
    bindkey '\e[3~'   delete-char         # delete\
    bindkey '\eOc'    forward-word        # ctrl right\
    bindkey '\eOd'    backward-word       # ctrl left\
    bindkey '^[[1;5C' forward-word        # ctrl right\
    bindkey '^[[1;5D' backward-word       # ctrl left\
    ;;
esac

echo 'source ~/.inputrc' >> ~/.zshrc

Or - if you want to activate them immediately

source ~/.inputrc

Kudos to vermaden at Server Fault for creating the list! Please upvote him if you find his answer helpful (I only found it and wanted to make it easier to find for those using Truenas)!

Getting ;5D when hitting ctrl + arrow key in a Terminal on FreeBSD

On centos I can skip a word by hitting ctrl + arrow (left or right) in a terminal. When I ssh into a FreeBSD box and I try the same pattern I get: $ tail -f 20120412.log;5D;5D;5D (each try = ;5D...

Server Faultjdorfman

I have a MacOS VM running on Proxmox installed by using the instructions from Nicholas Sherlock

Installing macOS 12 “Monterey” on Proxmox 7 – Nicholas Sherlock

Nicholas Sherlock Menu and widgetsNicholas Sherlock

I ran out of disk space for the VM and needed to add more.

1. Shutdown the MacOS VM
2. Increase disk size in the Proxmox GUI
3. Start the MacOS instance and connect to it
4. Open Disk Utility
5. Run "First Aid" on the disk
6. Open the partitioning tool
7. Type in the disk size you want (dragging didn't seem to work correctly)
   The "unused disk" fraction (grey) may be WRONG - but that's okay. As the resize runs MacOS makes sure the new disk size will work.
8. Boom! You should now be all set with your larger disk!

On Proxmox I recent updated a debian container - and when I rebooted it it wouldn't start!

In the browser I could see an error but no details.

So I tried launching from the browser as

pct start 103

Only to receive the error

Failed to run lxc.hook.pre-start for container "103"

But when I ran the verbose (debug version) of pct start

pct start 103 --debug 1

I could see that a mount point didn't exist!

DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 103 lxc pre-start produced output: directory '/mnt/pve/MyMountPoint' does not exist

running in debug more

Easiest way to install Arch - a prebuilt qcow2 file!

This takes a just a couple minutes and requires minimal thought and effort... but the hard drive will be 40GB. If that's a dealbreaker see below on how to install your own instance of Arch.

1. Download the basic qcow2 image from https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/106936/artifacts/browse/output to your Proxmox server
   e.g., wget https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/106936/artifacts/raw/output/Arch-Linux-x86_64-basic-20221201.106936.qcow2?inline=false
2. In the PRoxmox GUS Click Create: VM
3. On the OS Selection page choose "Do not use any media"
4. Continue the VM creation flow
5. Once complete, go to the Hardware tab and select the hard drive - remove it!
6. From the terminal run (where 134 is the appropriate instance id and local-zfs the location you want the virtual drive stored).
   qm importdisk 134 Arch-Linux-x86_64-basic-20221105.99990.qcow2 local-zfs
7. Start the VM and log in with the username/password arch/arch

Harder but more flexible: Install your own instance of Arch

Download the Arch installable iso

1. Create: VM
2. Select a disk size of whatever you want (but at the very least 2 GiB - I'd suggest something like 6 GiB so you have some flexibility if you don't expect to need a lot of space.
3. Select at least 750 MB ram (while 512M should work, I receive an error "Waiting 30 seconds for device....device did not show up..."
4. Select the Arch iso as an attached disk
5. Once the bootable iso boots run:
   cfdisk /dev/sda
   then run the following commands (note we're installing a Bootable

> dos
> hit "Enter" on free space
> hit new > (continue using the remainder of the disk) 
> primary
> left arrow to Bootable
> Enter on Bootable (should now see an asterisk in the table under "Boot")
> write > yes > quit

6. to confirm, you can run lsblk to see your partitions

mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt

7. update package lists

pacman -Syy

8. Install needed packages to your new virtual hard drive

pacstrap /mnt base linux linux-firmware sudo nano openssh networkmanager grub os-prober mtools

# OPTIONAL: if you run into an error about invalid keys then make sure you have up to date keys
pacman -Sy archlinux-keyring

pacstrap /mnt base linux linux-firmware sudo nano openssh networkmanager grub os-prober mtools

9. Create the fstab

genfstab -U /mnt >> /mnt/etc/fstab

10. Change root to /mnt so you can configure your new system

arch-chroot /mnt

11. Configure and install grub (again, this is WITHIN chroot session)

grub-install /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

12. Set the root password and create your own user (if you want)

# set your own root password
passwd

#create a new user and set it's password
useradd -m -G wheel myuser
passwd myuser

13. Enable NetworkManager so  you have internet access once you reboot

systemctl enable NetworkManager

13. Exit chroot and shutdown

exit

shutdown

14. Remove the arch install CD (iso) and start up the new VM!

After install you might be able to lower memory to even 300MB!

While I didn't have GUI access, I could still SSH. It turns out, OPNsense's main configuration file is /conf/config.xml.

So, first I backed up the configuration cp /conf/config.xml /conf/config.xml.bak.

Then I opened it in vi to find my recent changes (I had enabled a route and intervace named OpenVPN) vi /conf/config.xml.

After reversing my edits I saved the file :wq and ran reboot - I was back in!

I had much better luck running a pfSense VM with OpenVPN installed on it and setting that pfSense instance as the gateway for my VM.  Pretty extreme, but it was the best I could get... Please share links or suggestions if you have any!

My main Proxmox hard drive was filling up recently...

NAME    SIZE  ALLOC FREE CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
rpool   451G   396G 55G        -         -    32%    89%  1.00x    ONLINE

but I realized the partition was only half the size of the disk!  (Proxmox was originally installed on a smaller disk and copied to it's current disk with dd.)

So I pulled up the drive partitions:

Disk /dev/sdh: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device       Start       End   Sectors   Size Type
/dev/sdh1       34      2047      2014  1007K BIOS boot
/dev/sdh2     2048   1050623   1048576   512M EFI System
/dev/sdh3  1050624 937703054 936652431 446.6G Solaris /usr & Apple ZFS

Disk: /dev/sdh

Partition I wanted to expand /dev/sdh3

[For spoiler on the one line resize scroll to the end.]

I started to go through the fdisk flow of resizing the partition but paniced once I realized I had to delete the partition, re-write it, and then saw messages like Created a new partition 3 of type 'Linux Filesystem' and of size 884.4 GiB. and Partition #3 contains a zfs_member signature. I wasn't sure if the partiton was right or not!

Device       Start        End    Sectors   Size Type
/dev/sdh1       34       2047       2014  1007K BIOS boot
/dev/sdh2     2048    1050623    1048576   512M EFI System
/dev/sdh3  1050624 1855717376 1854666753 884.4G Linux filesystem

I quickly quit out of fdisk without rewriting the partition. I had remembered something easier from Ubuntu and finally found it! resizepart!

Simple 1 line partition resize to 100%:

I simply ran parted /dev/sdh resizepart 3 100% and the partition grew to 100% like magic! (Obviously use your appropriate disk letter/partition number.)

I made sure the pool would expand

zpool set autoexpand=on rpool
zpool online -e rpool /dev/sdh3

And confirmed it did by rerunning zpool list!

NAME  SIZE  ALLOC FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH 
rpool 900G   396G 504G        -         -    32%    44%  1.00x    ONLINE 

Note: the proxmox gui didn't recognize the resize until reboot, but as long as zpool did I was confident things were good!

Simply running zfs list  -t snapshot -o name -s name -r nas resulted in tens of thousands of results I couldn't scroll through. And I wanted to be cautious where I ran zfs destroy on snapshots!

So I decided to create a series of commands that let me count the number of snapshots each dataset has and sort them!

These commands will find all snapshots within the pool tank and count them by dataset

zfs list  -t snapshot -o name -s name -r tank | sed 's/@.*//' | sort | uniq -c | sort

While those are sorted in ascending order, that works nice for me in the terminal. If you prefer just the top datasets in descending order just reverse the final sort and pipe to head!

zfs list  -t snapshot -o name -s name -r tank | sed 's/@.*//' | sort | uniq -c | sort -r | head

 23161 tank/dataset1
 11353 tank/dataset2 
 1126 tank/backup/ubuntu_g1f5ds/var/lib/230948234aef887o90897f987e987e
 1073 tank/dataset4
 953 tank/dataset5

Warning, this should not be piped to zfs destroy, xargs, or anything like that. Here sed removes the snapshot name leaving just the dataset name.

I use a Wireguard VPN to tunnel my traffic when on public wifi - but I couldn't use Wireguard from a recent network!  This specific network was causing Wireguard to fail with the error message "RTNETLINK answers: No such device."  

I hadn't changed any configurations, so I was very confused why a device was missing!

I  found the answer on the WireGuard mailing list, IPv6 requires MTU of at least 1280 - and when I ran ip a I could see the public wifi only allowed up to 1270!

The fix for "RTNETLINK answers: No such device":

Disable IPv6 altogether (to prevent a leak if you do somehow switch back to IPv6

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1; sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1; sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

And update the AllowedIPs Wireguard config to exclude IPv6.

AllowedIPs = 0.0.0.0/0

And you should be able to connect!

Reminder

If you need to re-enable IPv6 remember to revert the changes to your config file!  The disabling of IPv6 DOES NOT persist between reboots. (You can disable IPv6 permanantly in /etc/sysctl.conf if needed.)

I back up multiple computers to my Truenas instance hourly, but these snapshots hang around without an easy way to prune them (computers I back up include Ubuntu and Proxmox).

But over a couple years one of my datasets seemed to grow snapshots faster than bunny rabbits reproduce to over 400k snapshots!

zfs list -t snapshot -o name -s name -r tank/UBUNTU_DATASET_NAME | wc -l

Not only that, but the above command took several minutes!!! (Taking minutes to list the snapshots was actually how I knew something was wrong.)

Sanoid and syncoid are great, but I was sending too many snapshots over without cleaning them up!

How to clean up snapshots

If you do this be VERY careful that you know what you're deleting. Step 2 is included to make sure we actually look at what we're deleting - don't skip it!

I am not responsible for any commands you run. These commands are not suggestions, just a list of what I ran.

For the below, I'm assuming my pool is named tank and the dataset I want to clean up is called UBUNTU_DATASET_NAME.

1. I needed to run these as root so I switched to root

sudo su

2. Find the snapshots I need to delete

zfs list -t snapshot -o name -s name -r tank/UBUNTU_DATASET_NAME | grep '@'  | grep _hourly

• zfs list -t snapshot -o name -s name -r tank/UBUNTU_DATASET_NAME recursively lists the snapshots
• grep '@' is my safety check to make sure I'm listing snapshots
• grep _hourly makes sure I'm only deleting the hourly snapshots

3. Pipe those to xargs -n1 zfs destroy

zfs list -t snapshot -o name -s name -r tank/UBUNTU_DATASET_NAME | grep '@'  | xargs -n1 zfs destroy