Sharing mounts with a container

Sometimes you might want to mount a device or directory tree into a running container. Since the container is in its own mounts namespace, you can’t just mount it under /var/lib/lxc/container/rootfs – that mount won’t get forwarded to the container.

Mounts propagation however will let you implement a nice simple solution. Below we’ll set up a container o1, such that anything you mount on the host under /shared/o1/ will show up under /shared in the container.

First, create and set up /shared. (I’m showing you how to do it the first time. It’s probably simplest, every time you reboot the host, to rm -rf /shared and re-run this segment – in an upstart job for instance).

mkdir /shared
mount –bind /shared /shared
mount –make-unbindable /shared
mount –make-shared /shared
mkdir /shared/o1

(In case you are wondering, we first have to turn the directory into a bind mount, as there must be a vfsmount for us to set its propagation. Next we make it unbindable so that /shared won’t get bound into any containers by accident. Then we make it shared, which creates a new mounts peer group with only a single entry – /shared. We could be more fine-grained here and make only /shared/o1 itself shared, and then, *after* it gets bound into the container, turn the container’s mount into a slave so that the container can’t push mounts back into the host’s /shared/o1.)

Now if you haven’t yet, create a container:

lxc-create -f /etc/lxc.conf -t ubuntu -n o1

Create the /shared directory in the container:

mkdir /var/lib/lxc/o1/rootfs/shared

And now add an lxc.fstab entry to bind mount the shared directory:

cat >> /var/lib/lxc/o1/fstab << EOF
/shared/o1 /var/lib/lxc/o1/rootfs/shared none bind 0 0
EOF

That's it. Now, when you do

mkdir /shared/o1/lib
mount –bind /lib /shared/o1/lib

on the host then container o1 will see that mount as /shared/lib

Likewise, anything the container mounts under /shared will show up under /shared/o1 on the host. This could be useful for /proc or /dev/pts, for instance.

For details, you can see the in-kernel documentation. (There used to be a developerworks article by Ram Pai and myself showing more use cases, but that appears to have disappeared.)

[Update: the DeveloperWorks article has re-appeared. You can see it here.]

Advertisements
This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

39 Responses to Sharing mounts with a container

  1. Arie Skliarouk says:

    I had more success with using the tmpfs for holding mountpoints (the rootfs based /zfs was not visible at all by guests:
    mount -t tmpfs share /zfs
    mount –make-rshared /zfs
    # Mount a different filesystem on the /zfs/media
    mkdir -p /zfs/media
    mount /dev/aa-mapper/media /zfs/media

    Still, in both cases the /zfs/media filesystem becomes invisible by guest until I run “mount /zfs/media on the host.

    BTW, the commands you used should be a bit different:
    mount -o bind /shared /shared
    mount -o make-unbindable,make-shared /shared

  2. Ivan Vilata says:

    Hi S3hh, I’ve found your instructions very useful, thanks! However, I also found that starting other containers unmounts all bind mounts under the shared mount. You may be interested in the discussion on the lxc-users list.

    • s3hh says:

      Like I said in the email, I’d like to support this feature in lxc itself through a config option. I won’t have time in the next few weeks for it, but if noone else beats me to it I intend to do it.

      • Ivan Vilata says:

        Heh, I wondered that you may be the author of both, but I still considered interesting to link the discussion here just in case. đŸ™‚ It’d be great to have that option available!

      • Did anything along these lines ever enter upstream LXC code? The discussion link above doesn’t work anymore, unfortunately…

      • s3hh says:

        No, we’ve never yet found a cleaner way to pass a mount into a running container. There is now a new setns() system call which is used by lxc-attach to let you enter namespaces, but because mount() does not allow mixing different namespaces between the mount source and target and because current’s root is switched on setns to a new mounts namespace, it hasn’t sufficed to solve this. It’s possible that some sort of fdmount() system call would give us what we need, but that does not currently exist.

        So right now we still have two main ways of doing this – either mount them at container startup, or use mounts propagation.

        Actually there might be something spiffy we could do with 9pfs, but I’ve not looked into that.

  3. FYI. I ended up writing set of scripts (“LLS”) that enable basic manipulation of LVM partitions from inside of the LXC guest:
    http://skliarie.blogspot.com/2011/11/llslxclvmsnapshots.html

  4. bmullan says:

    S3hh I tried you example and it does work.
    However, for my purposes I found a simpler way of doing this using the container’s .conf file
    I wanted to create multiple containers and then have each container (as well as the host) have a common shared directory they could each read/write to.
    $ sudo mkdir /shared
    and make the /shared directory r/w so all containers can r/w to it
    $ sudo chmod 666 /shared
    Next, create a container using lxc-create:
    $ sudo lxc-create -t ubuntu -n lxc1
    after the container has been created you can further configure it if you want to add packages etc and plan to create duplicates of a “base” type container but when done just stop it
    $ sudo lxc-stop -n lxc1
    then …
    If you want to clone the original container… clone it and give the new container a name (lxc2)
    $ sudo lxc-clone -o lxc1 -n lxc2
    after lxc2 has been created stop it also
    $ sudo lxc-stop -n lxc2
    Finally, I edit the “config” file for both the lxc1 and the lxc2 containers:
    $ sudo nano /var/lib/lxc//config
    and add basically the same entry at the end of both container’s “config” file except for the substitution of the appropriate :
    # create a shared directory w/host
    lxc.mount.entry=/shared /var/lib/lxc//rootfs/shared none defaults,bind 0 0
    Now..
    Restart both containers (lxc-start) and now both containers as well as the host will all have r/w access to the host’s /shared directory.

    Brian Mullan

    • bmullan says:

      Sorry but in my post the “lxc.mount.entry” got mangled by the blog software.
      The word “container_name” should have appeared between the two slashes in the lines
      $ sudo nano /var/lib/lxc//config
      and
      $ lxc.mount.entry=/shared/var/lib/lxc//rootfs/shared none defaults,bind 0 0
      but because I used “” brackets to highlight word “container_name”… the blog software just dropped those words.
      Those two lines should read:
      $ sudo nano /var/lib/lxc/container_name/config
      lxc.mount.entry=/shared /var/lib/lxc/container_name/rootfs/shared none defaults,bind 0 0

      • s3hh says:

        Hey,

        love that you’re experimenting đŸ™‚ Two quick comments:

        1. in the lxc.mount.entry, it’s generally recommented now to use the relative pathname for the target. So ‘/shared’ rather than ‘/var/lib/lxc/container/rootfs/shared’. There are several reasons for this, not just that it makes cloning containers easier.

        2. I wasn’t clear enough about the goal of the original post. It wasn’t just to share a directory – but to actually share the mount events thereunder. So if the container does a mount under /shared, the host will see it under /shared/container/.

        Thanks for the feedback!

  5. bmullan says:

    Thanks s3hh..
    Your comment about using the relative pathname helps me alot as I AM using a lot of lxc-clone and having to modify each container’s /var/lib/lxc/[container_name]/config file entry was going to be the only really manual step but if what you indicate works then I can just make the initial “base” container and clone the rest to all have the same “shared” directory… which is a big help to me.

  6. bmullan says:

    s3hh – ok I tried your suggestion to use the relative pathname but there seems to be a problem or maybe a bug.
    I edited the “config” file of a container to use just /shared

    lxc.rootfs = /var/lib/lxc/lxc-bos-3/rootfs
    # shared directory
    xc.mount.entry=/shared /shared none defaults,bind 0 0

    then started the container and logged in.
    Once logged into the container I checked the permissions of the /shared directory

    $ ls -lg /
    :
    drwxrw-rw- 2 root 4096 Sep 25 09:29 shared
    :

    The /shared directory had been set to 766 permissions.
    While “in” the container I tried to change directory to /shared to make sure I could still access it:

    $ cd shared
    $ -bash: cd: shared: Permission denied

    So I changed the permissions of /shared to also add execute permissions for everyone

    $ sudo chmod 777 /shared

    Again tried to change directory to /shared and now it worked?

    $ cd /shared
    /shared$

    Now I was able to change directory to /shared but why would adding execute permissions have made a difference?

    Brian

  7. bmullan says:

    Maybe I am not understanding your previous comment about using the “relative” path in the

    > 1. in the lxc.mount.entry, it’s generally recommented now to use the relative pathname for the > target. So ‘/shared’ rather than ‘/var/lib/lxc/container/rootfs/shared’. There are several reasons > for this, not just that it makes cloning containers easier.

    If I use the following in the LXC containers config file then access to the /shared directory works correctly (my container name is “lxc-bos-1”:

    lxc.mount.entry=/shared /var/lib/lxc/lxc-bos-1/rootfs/shared none defaults,bind 0 0

    and you are saying to just change this to:

    lxc.mount.entry=/shared /shared non defaults,bind 0, 0

    Is this correct?

    • s3hh says:

      Sorry – leave out the leading ‘/’ in the target, to make it a relative path. So it should be

      lxc.mount.entry = /shared shared none defaults,bind 0 0

  8. bmullan says:

    I found my own answer with a google search which found an email response on gmane to the same question which was answered by Serge Hallyn. My mistake was prefacing the target with the “/”.

    http://permalink.gmane.org/gmane.linux.kernel.containers.lxc.general/3559

  9. bmullan says:

    I guess we posted the same answer at the same time.
    However, I still find that the cloned container cannot access the shared directory unless it is set to 777 permissions. It would seem that 666 should work but it doesn’t.

    • Serge Hallyn says:

      Is that as root in the container, or an unprivileged user? 666 does not include ‘x’ – ‘r’ is enough to view the file listing, but ‘x’ is needed to actually descend and see the file contents (including file metadata).
      If you have a problem there may be a bug, please feel free to submit a bug or ask in #lxcontainers.

      • bmullan says:

        it was as an unpriviliged user. I’d then do: sudo chmod 777 /shared
        and after that have no problem accessing the /shared directory as an unpriivliged user.

      • s3hh says:

        Right, then not being able to see things under /shared with 666 makes sense. If you want readonly, use 555. See the chmod manpage for more.

  10. ahes says:

    Hey s3hh. I couldn’t find your e-mail address anywhere on this site. I would like to contact you to share some LXC experience such as sharing root filesystem in read-only mode (mount bind template) between multiple containers and running huge amount of LVM volumes – 300 containers on single machine.

    • s3hh says:

      I can be reached at serge d o t hallyn a t ubuntu d o t com. You also could use the lxc-users or lxc-devel mailing list, as others will certainly be interested.

  11. HI,

    I am trying to use your solution to shared some directory by using fstab. I am able to use it bu running mount command :
    * mount –bind /data /data
    * mount –make-unbindable /data
    * mount –make-shared /data

    But I am not able to get the same result by using fstab. Syntax that I used :

    /data /data none bind 0 0
    none /data none make-shared 0 0
    none /data none make-unbindable 0 0

    Could you help me on this ?

    Thanks & Regards,
    Arthur.

  12. s3hh says:

    Great – could you share what format worked? đŸ™‚

    • Of course, I will share it with pleasure. I will give you the good syntax on Monday. I have not the code with me right now. And thank you for you post that was very helfull for me đŸ™‚

      • So for example to share /data though fstab entry I just need to put this line :
        /data /data none bind 0 0
        14 none /data none make-unbindable 0 0
        15 none /data none make-shared 0 0

        Which work exactly like the mount command give by your post

  13. Paul Healy says:

    Thanks this post was very useful.

    Note that from mount(8) – ‘… Since util-linux 2.23 mount command allows to use more propaga-
    tion flags together and with another mount operations. This fea-
    ture is EXPERIMENTAL. …’

    This works for me:
    /shared /shared none bind,unbindable,shared 0 0

  14. Tomi Ollila says:

    I did this:

    sudo mkdir /lxc-shared
    sudo mount –bind /lxc-shared /lxc-shared
    sudo mount –make-unbindable /lxc-shared
    sudo mount –make-shared /lxc-shared
    sudo mkdir /lxc-shared/user
    sudo chown user /lxc-shared/user
    mkdir /lxc-shared/user/sshfsmnt

    sshfs -o allow_root user@remote: /lxc-shared/user/sshfsmnt

    (container in question has fstab line bind-mounting /lxc-shared/user to shared & /shared exists in container rootfs)

    Now, when I run

    lxc-unshare -s MOUNT — sh -c “mount –make-rslave /lxc-shared && \
    set -x; exec lxc-start -l DEBUG -d -C -n lxc-container”

    The /shared/sshfsmnt in the container is empty.

    I presume this is the point when the sshfs disappears in container context:

    lxc-start 1409759852.050 DEBUG lxc_conf – umounted ‘/lxc_putold/lxc-shared/user/sshfsmnt’

    I just wonder why; this line was there before:

    lxc-start 1409759851.975 DEBUG lxc_conf – mounted ‘/lxc-shared/user/sshfsmnt’ on ‘/usr/lib/x86_64-linux-gnu/lxc/shared/sshfsmnt’, type ‘none’

    — is this something related to kernel handling of fuse filesystems; sshfs (with debug output) did
    not show anything to happen there.

    I tried all kinds of tricks there, but nothing helped…

    This is on ubuntu 14.04.1 server, lxc 1.0.5-0ubuntu0.1 (apparmor prohibited lxc-start making those mounts slave, but hacking around that did not help…)

    If I fusermount -u the fs in the host and remount, then the fs is propagated to the container.

    One hacky way to get sshfs initially seen in the container is to add the full path to
    it to container fstab…or as i have it:

    lxc.mount.entry = /lxc-shared/user shared none bind 0 0
    lxc.mount.entry = /lxc-shared/user/sshfsmnt shared/sshfsmnt none bind 0 0

    Now, if e.g. the sshfs connection is broken, the mount can be restored to the container(s)
    by first umounting /shared/sshfs in the container(s) and then re-running sshfs in the host.

    But it would be nice if using just the one would make things just work.

    Tomi

    Note to self: remember to test sudo mount -t tmpfs none /lxc-shared …

    • s3hh says:

      Hi,

      to test whether this is fuse-specific you could of course re-test with a simple root-created bind mount in place of the sshfs mount.

      However if I’m reading correctly, there’s nothing odd going on here. You called adding a lxc.mount.entry to the container configuration a ‘hacky way’ to get it seen, but that’s precisely how it is meant to be done.

      • Tomi Ollila says:

        Hi,

        I finally got back to this:

        Instead of

        sshfs -o allow_root user@remote: /lxc-shared/user/sshfsmnt

        I did

        sudo mount –bind /usr/lib /lxc-shared/user/sshfsmnt

        … and the results are the same; the mount is not visible in the container.

        In the ‘hacky way’ I meant I had to make those 2 lxc.mount.entry above
        (instead just the latter).

        But, as I’ve survived half a year w/o this I probably have to put this to even
        lower-priority task; this would be interesting but there is just not enough time.

      • s3hh says:

        Uh, hm, just to be sure, were you using ‘rbind’ instead of ‘bind’ in the lxc.mount.entry for /lxcshared? Otherwise indeed sub-mounts would not show up in the container.

  15. Dmitri says:

    I want the mounted folder inside the container (with CIFS) to be visible on the host.
    So, let say I mount inside the container a shared folder from server on local network, as:

    $ mount -t cifs //192.168.1.xxx/shared -o username=’name’ /mnt/my_folder

    and I want that mounted folder “my_folder” to be visible on the host as /var/lib/lxc/my_container/roootfs/mnt/my_folder, so I can access from host the mounted files inside container!
    Please help me!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s