Adventures in Building Containerd

The other day, I was sitting with some of the awesome folks at Docker in the hopes of getting my feet wet with docker internals. Anil Madhavapeddy suggested that I start by building my own containerd and successfully running a container there, the idea being to determine if the README (of containerd) was good enough. I ran this all in AWS on a clean install of the latest Ubuntu AMI. All of the below commands were run as the default user (ubuntu).

To begin with, I figured I would need Go and Git. I downloaded the latest version of Go from the website to ensure I had the latest version from 1.6 (which was the best choice at the time of writing this, YMMV).


Then I installed make, gcc, git:

sudo apt-get install make git

Now I was ready to check out the sources.

Important: If you are working on a forked version of containerd, ensure that in $GOPATH your code is still in, as the Makefile assumes the source will be found there.

mkdir -p $GOPATH/src/
 cd $GOPATH/src/
 git clone
 cd containerd
 sudo bin/containerd

If things work, you should see something like this (but with ansi colors!):

ubuntu@ip-172-30-0-65:~/go/$ sudo bin/containerd
 WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096

The next step is to run an OCI bundle (which is the same concept as a docker image, just using the new open standard). I used redis, as recommended in the containerd README. I needed to prepare the filesystem (that will be used inside of the container) as follows:

sudo mkdir -p /containers/redis/rootfs
 sudo docker pull redis
 sudo docker create --name tempredis redis
 sudo docker export tempredis | tar -C redis/rootfs -xf -
 sudo docker rm tempredis

To generate the config used for the OCI bundle, I used the runc binary compiled from the runc github repository. It is also possible to download a binary directly, but given that I’ve just built containerd from scratch and have the toolchain on hand, I built from source:

mkdir -p $GOPATH/src/
 cd $GOPATH/src/
 git clone
 cd runc

At this point I discovered that I also needed the libseccomp C headers for the new seccomp stuff:

apt-get install libseccomp-dev

And now I could finally build runc:


If all goes well (and thankfully it did), I’ll have an executable runc. At this point, I went back to /containers to prepare the OCI bundle config:

cd /containers
 $GOPATH/src/ spec

That will give a spec, which is the config needed to run the container. However, if I try to run this with ctr, I’ll still get an error:

ubuntu@ip-172-30-0-65:/containers/redis$ sudo $GOPATH/src/ containers start redis /containers/redis
 [ctr] rpc error: code = 2 desc = containerd-shim not installed on system

At this point I discovered that if containerd-shim and runc (good thing we got runc to make the spec file!) aren’t present in $PATH, I need to tell containerd where they are. So kill containerd, and relaunch as follows:

sudo ./containerd --shim $GOPATH/src/ --runtime 

And then try again:

sudo $GOPATH/src/ containers start redis /containers/redis
 sudo $GOPATH/src/ containers list
 redis /containers/redis running init


Since it’s actually running sh (as I didn’t change the config yet), I can’t actually do anything with it yet (since there’s currently no ctr attach functionality).

To actually interact, I could relaunch and attach, or create a second process and attach:

sudo $GOPATH/src/ containers exec --id redis -a -t --pid 1 --cwd / -u 0 -g 0
 # id
 uid=0(root) gid=0(root) groups=0(root)
 # exit

To see that it really works, I ask to become the “redis” user:

sudo $GOPATH/src/ containers exec --id redis -a -t --pid 1 --cwd / -u 999 -g 999 /bin/sh
 $ id
 uid=999(redis) gid=999(redis) groups=999(redis)
 $ exit

To kill it, I have to (rather forcefully) terminate init:

sudo $GOPATH/src/ containers kill redis --pid init --signal 9

ubuntu@ip-172-30-0-65:/containers/redis$ sudo $GOPATH/src/ containers

To actually run redis inside, I want to change some information in the config.json:

Specifically, process.args needs to change from ["sh"] to ["redis-server", "--bind", ""] and process.user from {} to {"uid":999,"gid":999}

Then launch again:

sudo $GOPATH/src/ containers start redis /containers/redis

This successfully gives us a container running redis. I still didn’t touch any networking that may be needed to expose redis, nor did I play with the new seccomp functionality (which is in the latest version), but this was a good starting point (and to be honest, everyone was leaving and the cleaning crew kicked me out of the room).

Let's put these tips to good use

Monetize with ironSource