LXC – improved clone support

Recently I took some time to work on implementing container clones through the lxc API. lxc-clone previously existed as a shell script which could create snapshot clones of lvm and btrfs containers. There were several shortcomings to this:

1. clone was not exportable through the API (to be used in python, lua, go and c programs). Now it is, so a Go program can create a container clone in one function call.
2. expanding the set of supported clone types became unsavory
3. overlayfs was only supported as ‘ephemeral containers’, which could be made persistent through the use of pre-mount hooks. They were not first class citizens. Now they are.

The result is now in upstream git as well as in the packages at the ubuntu-lxc/daily ppa. Supported backing store types currently include dir (directory), lvm, btrfs, overlayfs, and zfs. Hopefully loop and qemu-nbd will be added soon. They each are somewhat different due to the nature of the backing store itself, so I’ll go over each. However in my opinion the coolest thing you can do with this is:

# create a stock directory backed container
sudo lxc-create -t ubuntu -n dir1
# create an overlayfs snapshot of it
sudo lxc-clone -s -B overlayfs dir1 s1

The -s argument asks for a snapshot (rather than copy) clone, and -B specifies the backing store type for the new container. When container s1 starts, it will mount a private writeable overlay (/var/lib/lxc/dir1/delta0) over a readonly mount of the original /var/lib/lxc/dir1/rootfs.

Now make some changes to start customizing s1. Checkpoint that state by cloning it:

sudo lxc-clone -s s1 s2

This will reference the same rootfs (/var/lib/lxc/dir1/rootfs) and rsync the overlayfs delta from s1 to s2. Now you can keep working on s1, keeping s2 as a checkpoint. Make more changes, and create your next snapshot

sudo lxc-clone -s s1 s3

sudo lxc-clone -s s1 s4

If at some point you realize you need to go back to an older snapshot, say s3, then you can

sudo lxc-clone -s s1 s1_bad # just to make sure
sudo lxc-destroy -n s1
sudo lxc-clone -s s3 s1

and pick up where you left off. Finally, if you’re happy and want to tar up what you have to ship it or copy to another machine, clone it back to a directory backed container:

sudo lxc-clone -B dir s1 dir_ship
sudo tar zcf /var/lib/lxc/dir_ship.tgz /var/lib/lxc/dir_ship

So far I’ve shown dir (directory) backing store and overlayfs. Specific to directory backed containers is that they cannot be snapshotted, except by converting them to overlayfs backed containers. Specific to overlayfs containers is that the original directory backed container must not be deleted, since the snapshot depends on it. (I’ll address this soon, marking the snapshotted container so that lxc-destroy will leave it alone, but that is not yet done)

To use btrfs containers, the entire lxc configuration path must be btrfs. However since the configuration path is flexible, that’s not as bad as it used to be. For instance, I mounted a btrfs at $HOME/lxcbase, then did

sudo lxc-create -t ubuntu -P $HOME/lxcbase -n b1

(The ‘-P’ argument chooses a custom ‘lxcpath’, or lxc configuration path, than the default /var/lib/lxc. You can also specify a global default other than /var/lib/lxc in /etc/lxc/lxc.conf.) lxc-create detects the btrfs and automatically makes the container a new subvolume, which can then be snapshotted

sudo lxc-clone -s b1 b2

For zfs, a zfsroot can be specified in /etc/lxc/lxc.conf. I created a zfs pool called ‘lxc’ (which is actually the default for the lxc tools, so I did not list it in /etc/lxc/lxc.conf), then did

sudo lxc-create -B zfs -t ubuntu -n z1
sudo lxc-clone -B zfs dir1 z1

This created ‘lxc/z1’ as a new zfs fs and mounted it under /var/lib/lxc/z1/rootfs. Next I could

sudo lxc-clone -s z1 z2

Now lxc-destroy needs some smarts still built-in to make zfs backed containers easier to destroy. That is because when lxc-clone creates z2 from z1, it must first create a snapshot ‘lxc/z1@z2’, then clone that to ‘lxc/z2’. So before you can destroy z1, you currently must

sudo lxc-destroy -n z2
sudo zfs destroy lxc/z1@x2

Finally, you can also use LVM. LVM snapshot container clones have been supported longer than any others (with btrfs being second). I like the fact that you can use any filesystem inside the LV. However, the two major shortcomings are that you cannot snapshot a snapshot, and that you must (depending at least on the filesystem type) choose a filesystem size in advance.

To clone LVM conatiners, you either need a vg called ‘lxc’, or you can specify a default vg in /etc/lxc/lxc.conf. You can create the initial lvm container with

sudo lxc-create -t ubuntu -n lvm1 –fssize 2G –fstype xfs
sudo lxc-clone -B lvm dir1 lvm1

Then snapshot it using

sudo lxc-clone -s lvm1 lvm2

Note that unlike overlayfs, snapshots in zfs, btrfs, and lvm are safe from having the base container destroyed. In btrfs, that is because the btrfs snapshot is metadata based, so destroying the base container simply does not delete any of the data in use by the snapshot container. LVM and zfs both will note that there are active snapshots of the base rootfs and prevent the base container from being destroyed.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

12 Responses to LXC – improved clone support

  1. Interesting reading, thanks! I look forward to giving this a try programatically. Additionally, it always seems that btrfs has shortfalls – whenever someone tries to use it for something, but this seems to suggest it is well suited – so i’ll use that!. 🙂

  2. Hi,
    the new lxc-clone has broken support for LVM cloning, I had to go back to the old 0.8 script.
    Otherwise, good job!

    • s3hh says:

      Which lxc package were you using? Note that the ubuntu raring and saucy packages don’t yet have the new support. AFAIK only ppa:ubuntu-lxc/daily has it. It will land in saucy the next time we merge upstream, but that should not happen until I fix a few api issues…

      • I am using ppa:ubuntu-lxc/daily, actually. 0.9 has it broken too, though.
        I’m not the only one experiencing this as you can see below.

      • s3hh says:


        I’m not sure why I can’t reply to your post below…

        I don’t see anyone below having a problem with the ubuntu-lxc/daily ppa. If it isn’t working for you, could you please send an email to the lxc-users or lxc-devel list with more details? It should be working. I am using vg ‘vg0’ (defined as the default vg in my /etc/lxc/lxc.conf).

  3. Norberto says:


    I was about to use lxc-clone (ubuntu 12.04 and 13.04) but to my surprise, it didn’t like my lvm volume names.

    I create my containers like this:

    $ sudo lxc-create -n name -t ubuntu -B lvm –lvname lxc-name-root …

    because I want to know what a particular lvm volume holds (the vg is also used to store kvm images).

    I’ve tried lxc-clone from ppa:ubuntu-lxc/daily but it’s worse since it hangs for ever (lxc-clone from 12.04 at least shows an error message).

    I can hack the bash scripts shipped with precise and raring to handle this case, but do you think it is worth to add support to name the snapshot volumes to the main branch of lxc-clone?

    For my case, I was thinking in something like this:

    $ sudo lxc-clone -o name -n newname -l lxc-newname-root

    Thank you very much!

    PS: yes, I know about -p switch on raring, but I need a subfix too 🙂

    • Norberto says:

      replying to myself :-/

      lxc-clone from current ppa:ubuntu-lxc/daily does exactly what I want on raring (I was testing it on precise).

      I’m really sorry for the spam.

      Thanks so much for all the work you’re doing on lxc!

      • s3hh says:

        Hm, I can’t offhand think of any reason it should fail on precise kernel. In fact I use it on precise kernel all the time.

      • Norberto says:

        Hi again,

        I just tested ppa:ubuntu-lxc/daily again on precise, and now lxc-clone -s, with lvm backend (thru iscsi), does work here (does what I want). Maybe I was just tired and made some mistake somewhere (I’ve been testing this all week).

        Thanks again for all your work!

        Best regards,

  4. Looks like lxc-clone may be broken when used with vagrant-lxc. The root issue seems to occur when lxc-clone greps the old container config file for lxc.rootfs:

    oldroot=`grep lxc.rootfs *=’ $lxc_path/$lxc_orig/config | awk -F= ‘{ print $2 ‘}`

    However some of the vagrant-lxc images create comments that include the phrase “lxc rootfs”, which ends up matching the grep expression. Changing the regex to something more specific like:

    oldroot=`grep ‘^lxc\.rootfs *=’ $lxc_path/$lxc_orig/config | awk -F= ‘{ print $2 ‘}`

    resolves the issue

    • s3hh says:

      Try the git head or ppa:ubuntu-lxc/daily – lxc-clone has become a c program (using the clone api function), and the problem you’re having should mostly be solved now, as we’ve made it less ‘magic’ and it should now only replace the actual name.

  5. Pingback: Hardware? LXC (Linux containers) | develox.org

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s