Blog
Adventures in Building Containerd

The other day, I was sitting with some of the awesome folks at Docker in the hopes of getting my feet wet with docker internals. Anil Madhavapeddy suggested that I start by building my own containerd and successfully running a container there, the idea being to determine if the README (of containerd) was good enough. I ran this all in AWS on a clean install of the latest Ubuntu AMI. All of the below commands were run as the default user (ubuntu).

To begin with, I figured I would need Go and Git. I downloaded the latest version of Go from the website to ensure I had the latest version from 1.6 (which was the best choice at the time of writing this, YMMV).

wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz

Then I installed make, gcc, git:

sudo apt-get install make git

Now I was ready to check out the sources.

Important: If you are working on a forked version of containerd, ensure that in $GOPATH your code is still in github.com/docker/containerd, as the Makefile assumes the source will be found there.

mkdir -p $GOPATH/src/github.com
 cd $GOPATH/src/github.com
 git clone https://github.com/docker/containerd
 cd containerd
 make
 sudo bin/containerd

If things work, you should see something like this (but with ansi colors!):

ubuntu@ip-172-30-0-65:~/go/github.com/docker/containerd$ sudo bin/containerd
 WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096

The next step is to run an OCI bundle (which is the same concept as a docker image, just using the new open standard). I used redis, as recommended in the containerd README. I needed to prepare the filesystem (that will be used inside of the container) as follows:

sudo mkdir -p /containers/redis/rootfs
 sudo docker pull redis
 sudo docker create --name tempredis redis
 sudo docker export tempredis | tar -C redis/rootfs -xf -
 sudo docker rm tempredis

To generate the config used for the OCI bundle, I used the runc binary compiled from the runc github repository. It is also possible to download a binary directly, but given that I’ve just built containerd from scratch and have the toolchain on hand, I built from source:

mkdir -p $GOPATH/src/github.com/opencontainers
 cd $GOPATH/src/github.com/opencontainers
 git clone https://github.com/opencontainers/runc
 cd runc

At this point I discovered that I also needed the libseccomp C headers for the new seccomp stuff:

apt-get install libseccomp-dev

And now I could finally build runc:

make

If all goes well (and thankfully it did), I’ll have an executable runc. At this point, I went back to /containers to prepare the OCI bundle config:

cd /containers
 $GOPATH/src/github.com/opencontainers/runc/runc spec

That will give a spec, which is the config needed to run the container. However, if I try to run this with ctr, I’ll still get an error:

ubuntu@ip-172-30-0-65:/containers/redis$ sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers start redis /containers/redis
 [ctr] rpc error: code = 2 desc = containerd-shim not installed on system

At this point I discovered that if containerd-shim and runc (good thing we got runc to make the spec file!) aren’t present in $PATH, I need to tell containerd where they are. So kill containerd, and relaunch as follows:

sudo ./containerd --shim $GOPATH/src/github.com/docker/containerd/bin/containerd-shim --runtime 
/home/ubuntu/godev/src/github.com/opencontainers/runc/runc

And then try again:

sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers start redis /containers/redis
 sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers list
 ID PATH STATUS PROCESSES
 redis /containers/redis running init

Success!

Since it’s actually running sh (as I didn’t change the config yet), I can’t actually do anything with it yet (since there’s currently no ctr attach functionality).

To actually interact, I could relaunch and attach, or create a second process and attach:

sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers exec --id redis -a -t --pid 1 --cwd / -u 0 -g 0
 # id
 uid=0(root) gid=0(root) groups=0(root)
 # exit

To see that it really works, I ask to become the “redis” user:

sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers exec --id redis -a -t --pid 1 --cwd / -u 999 -g 999 /bin/sh
 $ id
 uid=999(redis) gid=999(redis) groups=999(redis)
 $ exit

To kill it, I have to (rather forcefully) terminate init:

sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers kill redis --pid init --signal 9

ubuntu@ip-172-30-0-65:/containers/redis$ sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers
ID PATH STATUS PROCESSES

To actually run redis inside, I want to change some information in the config.json:

Specifically, process.args needs to change from ["sh"] to ["redis-server", "--bind", "0.0.0.0"] and process.user from {} to {"uid":999,"gid":999}

Then launch again:

sudo $GOPATH/src/github.com/docker/containerd/bin/ctr containers start redis /containers/redis

This successfully gives us a container running redis. I still didn’t touch any networking that may be needed to expose redis, nor did I play with the new seccomp functionality (which is in the latest version), but this was a good starting point (and to be honest, everyone was leaving and the cleaning crew kicked me out of the room).

Let's put these tips to good use

Grow your game with ironSource