Whither cgmanager

A few years ago we started the cgmanager project to address two issues:

  1. Simplify code in callers.
  2. Support safe delegation of cgroups to nested container managers.

Historically, advice on where and how to mount cgroups was varied. As a result, programs which wanted to manipulate cgroups had quite a bit of work to do to find cgroup mountpoints for specific controllers and calculate the full paths to use. Cgmanager simplified this code by doing that work for callers. With the ‘cgm’ command, it was also greatly simplified for users. Today, with the advent of the cgroup-lite package, now the cgroupfs-mount package, and systemd, all of which agree on mounting each controller separately under /sys/fs/cgroup/$controller, container manager code can be greatly simplified. This can be evidenced by comparing the older ‘cgfs’ cgroup driver in lxc to the newer ‘cgfsng’ cgroup driver which benefits from assuming the simpler layout (falling back to the more complicated driver if needed).

The other core motivating factor for cgmanager, safer support of cgroup delegation, is now – at last! – deprecated by the availability of cgroup namespaces.

A few lessons?

When starting actual coding for cgmanager, one open question was what kind of communication it should support. We decided on using a dbus interface, implemented using libnih-dbus. I had strong reservations about that. In retrospect, it worked better at the time than I had expected, but had some severe problems. First, performance was pretty horrific. Every dbus connection required 4 round trips to get started. For the simple upstart based systems of the time this was ok, and we kept it from becoming worse by having a cgmanager proxy in a multiply-nested container talk straight to the host cgmanager. However, as Ubuntu switched to systemd, which made very heavy use of cgroups, we started seeing huge performance impacts. It also failed to satisfy the requirements of google folks who otherwise may have been more interested.

Secondly, as Ubuntu switched from systemd to upstart, and upstart – and libnih – became unsupported, this started affecting cgmanager.

So, if cgmanager were still needed today, I would strongly consider rewriting it to have a simple interface over a unix socket. However, as described above, cgmanager has happily become unnecessary. A different sort of cgroup manager may in fact be needed – a modern day ‘libcgroup’ for simple administration of service resources on systems without systemd. However, that is not cgmanager’s role.

So, with that, the cgmanager project is effectively closed. It will continue to be supported by the lxc project (us) on legacy systems. But on the whole, migrating systems to a kernel supporting cgroup namespaces (or at least lxcfs-provided cgroup filesystems) is highly recommended.

PS

By the same token, the cgroup part of lxcfs is also effectively deprecated. I will probably move it under a deprecation build flag soon. lxcfs will continue to be used for procfs virtualization, and likely expanded to support some /sys information.

Posted in Uncategorized | Leave a comment

Two worthwhile books

This past summer, we went to the local bookstore and picked up a copy of Jason R. Briggs’ “Python Programming for Kids” (No Starch Press). I had been looking for a fun kids’ programming book for a while, and decided on this one. My then 8 year old basically worked through the book alone, slowly following the recipe toward building the game which is the books’ climax. I had wanted to do a detailed review of this book, but it recently had an unfortunate encounter with a whole raw chicken, so that may not happen. Suffice it to say I think it’s a great book for a child to work through.

(On a side note, there is a free program called ‘laby’, where kids program a robot ant to escape a labyrinth in several languages, which was great fun for the kids. Heck – it’s fun for adults.)

More recently, I was sent a review copy of ‘The Car Hacker’s Handbook” (also No Starch Press) by Craig Smith. I was excited about this one, as the subject matter is both fascinating and disturbing. New cars have some great features, but the fact that I can (for instance) click a link on a google map to send directions to a car is a bit disconcerting. So, as Chris Evans says in the prologue,

“We’re all safer when the systems we depend upon are inspectable, auditable, and documented – and this definitely includes cars.”

This book starts by teaching about threat models. It goes into great detail describing various in-car networks, as these are a gateway to inspecting, modifying, and perhaps subverting the vehicle’s systems. It gives details about tools to retrieve diagnostic info, modify the ECU programming, and listen in on TPMS systems. It goes over cracked keyless entry systems. It (I think rightly) defers details about disassembling existing programs to other texts, but shows how to write a weaponized exploit. Overall, a wonderful – and motivating – book.

Posted in Uncategorized | 2 Comments

Edbrowse

Earlier this year I was looking for minimalist browsers not based on webkit. While looking through the list at suckless.org, I ran across a new to me project called edbrowse, originally intended for use by blind users.

Edbrowse is a line based editor (like ed) and browser – and more. If you’ve used ed before, it will serve you well in using edbrowse as an editor. However, edbrowse is much more. Edbrowse has multiple buffers. This alone can be a nice feature. It becomes particularly useful when browsing the web. For instance, when looking through a set of search results, you can follow a set of links, placing each in its own buffer. Edbrowse supports flexible macros. I have macros to quickly search duckduckgo and lxr, as well as to get and delete readitlater/pocket lists. Edbrowse also interacts with databases,and functions as a mail client, though I’ve not used these features.

The user’s guide is very informative and a nice read.

There are a few issues. First, at the moment it doesn’t seem to work will in particular with github. The current author of edbrowse also has another project, cligh, which provides a nice command line interface to github – which I also use regularly.

One other issue is that while edbrowse has multiple buffers, loading a page is synchronous. I’ve not found a way to say “b http://slowsite.org and switch to buffer 3 while slowsite loads”. On the whole though, pages load very quickly and it has only been an issue when on very bad network links.

And, of course, for many people the steep learning curve and line based interface will be a problem. Personally, I find it can be very useful in helping to reduce the distractions on the rest of the page.

I’ve not replaced my use of vi with edbrowse. I’ve not even replaced my use of other browsers. But I do use edbrowse several times per day, both as an editor and as a browser. And, yeah, it’s kind of fun.

Posted in Uncategorized | Leave a comment

Using lxd snapshots

Lxd with zfs is fast. Very fast.

Yesterday I was testing a package upgrade problem. Purging and re-installing the package was not good enough to reproduce it. So,

1. create a base container

lxc launch ubuntu:xenial lv1

That takes about 2 seconds on my laptop. Next, customize as needed. Due to the point we’re at in the development cycle, I needed to

2. update it to yakkety,

cat > 01proxy << EOF
Acquire::http::Proxy "http://10.0.3.1:3142&quot;;
EOF
lxc file push 01proxy lv1/etc/apt/apt.conf.d/
lxc exec lv1 — apt update
lxc exec lv1 — apt -y upgrade
lxc exec lv1 — sed -i 's/xenial/yakkety/g' /etc/apt/sources.list
lxc exec lv1 — apt update
lxc exec lv1 — apt -y upgrade
(And install a few more packages)

lxc stop lv1 –force

3. snapshot that

lxc snapshot lv1

This created snap0 by default, and took less than 0.3 seconds.

lxc info shows the snapshot:

Name: lv1
Architecture: x86_64
Created: 2016/05/06 21:26 UTC
Status: Stopped
Type: persistent
Profiles: default
Snapshots:
snap0 (taken at 2016/05/06 23:28 UTC) (stateless)
snap1 (taken at 2016/05/06 23:45 UTC) (stateless)

Next, every time I wanted to do a test, i created a new container based on the snapshot, and used it

lxc copy lv1/snap0 lv2

which takes less than 0.4 seconds.

In the end I just scripted:

#!/bin/bash
lxc delete lv3 –force
lxc copy lv2/snap0 lv3
for f in *.deb; do lxc file push $f lv3/root/; done
lxc start lv3
lxc exec lv3 bash

So now the iterative cycle consisted of:

vi preinst postinst
tar zcf control.tar.gz control md5sums preinst
ar cr new.deb debian-binary control.tar.gz data.tar.xz
./script

In this way, each iteration was fast enough to not get distracted and forget
what i was doing between cycles. Nice.

Posted in Uncategorized | Tagged , , | Leave a comment

LXD Image Updates

LXD is sweet. To create a ubuntu xenial container, just do

lxc launch ubuntu:xenial x1

The remote xenial image will be automatically fetched on the first use, and cached for future uses. Nifty. Hmm… but what then?

There are three server settings which guide how images are treated after their initial use. And setting these can make the difference between frequent higher bandwidth use and more uptodate images.

The discussion will be eased if we define a term: a ‘cached’ image is one which is created implicitly as part of a container creation. I.e. the first time that you do

lxc launch ubuntu:trusty t1

lxd will fetch the ubuntu:trusty image, implicitly creating a cached copy on the target remote. In contrast, you can explicitly create a non-cached image using

lxc image copy ubuntu:wily local: –alias wily

As you might expect, these two images are treated a bit differently. The first is cached for a while to save on future bandwidth, but it’s a second class citizen compared to the second image, which the user explicitly requested.

With that in mind, as I said earlier, there are three configuration settings which guide image behavior.

First there is the cached image auto update setting,

lxc config set images.auto_update_cached true

Any cached images which are created while this is set to true will be auto-updated. Note that setting this to true after creating a container will not change that pre-existing cached image’s auto update setting.

For explicitly created images, you can instead specify auto-update manually using

lxc image copy ubuntu:wily local: –alias wily –auto-update

The second setting is remote_cache_expiry:

lxc config set images.remote_cache_expiry 1

This specifies, in days, how long an unused cached image is kept in the cache before it is flushed. If this value is not specified, then it defaults to 10 days. Note that explicitly copied images are not expired.

Finally, auto_update_interval specifies how many hours should pass between lxd’s checks for whether a cached image should be updated. Setting this value to 0 disables the checks. So if you find yourself temporarily on a slow, expensive, 3G router, you may want to

lxc config set images.auto_update_interval 0

to save yourself from accidentally downloading a few hundred megs or more of updated image contents. On the other hand, when back at the datacenter, you might want to

lxc config set images.auto_update_interval 1

to keep as in sync as possible.

Posted in Uncategorized | 5 Comments

Docker in LXD

Since the very early days of upstream Linux containers – around 2006 – we’ve been distinguishing between ‘application’ and ‘system’ containers. (The definition of application containers has changed a bit, and their use case has changed a *lot*, but the general gist remains the same).

A few years ago I would get regular – daily! – queries by lots of people asking what I thought of Docker. Some asked because, as one of the early people involved in kernel container functionality, I’d be interested. Others did so because I had been working with http://linuxcontainers.org/lxc, a particular container administration suite, and thought I’d feel competitive. However, as we’ve said for a long time, Docker is a great tool for application containers and application container purposes. From a LXC/LXD perspective, we’re looking at different use cases. One of those is hosting containers in which to run Docker:)

And, in Ubuntu 16.04, you can easily do so. (The Docker patches to enable this are on their way upstream.) To run Docker inside a container, the container must have a few properties. These are conferred by the ‘docker’ profile. The docker profile does not include a network interface, so you’ll want to create a container with both the default and docker profiles:

lxc launch ubuntu-daily:xenial docker1 -p default -p docker

Now, enter the container and install the docker.io package:

lxc exec docker1 — apt update
lxc exec docker1 — apt install docker.io
lxc exec docker1 — docker pull ubuntu
lxc exec docker1 — docker run -it ubuntu bash

et voila, a docker container is running inside your lxd container. By itself this may seem like a novelty. However, when you start deploying the lxd hosts with openstack nova-lxd plugin or juju-lxd, the possibilities are endless.

Posted in Uncategorized | Leave a comment

PSA: nested lxc containers

lxc has long supported nesting containers. There’s a lot of (historically accurate) documentation out there saying to use the line

lxc.aa_profile = lxc-container-default-with-nesting

to enable that. Sadly, a somewhat new kernel restriction has recently required a bit more work. To support that, the new way to support nesting in lxc is to use the configuration line:

lxc.include = /usr/share/lxc/config/nesting.conf

That configuration file includes the old aa_profile line. If you have your own custom nesting profile, you can follow the above lxc.include line with your lxc.aa_profile line, i.e.

lxc.include = /usr/share/lxc/config/nesting.conf
lxc.aa_profile = my-custom-nesting-profile

If you’re using lxd, this of course does not affect you. You can continue to use the ‘security.nesting: true’ config property as always.

Posted in Uncategorized | Leave a comment