Harry Marr

CodeCube: Docker-powered Runnable Gists

Whether you’re soliciting programming advice on a mailing list, providing technical support over IRC, or saving a useful snippet of code for later, GitHub’s Gist is an indispensable tool.

I’ve been writing more and more code in Go recently, and something struck me about the Go community: they don’t use Gists. When Go developers share code, they do so almost exclusively through a tool called the Go Playground. The key advantage of the Go Playground over GitHub Gists is that you can run the code snippets and see the output live in your browser. This is extremely powerful. I’ve seen countless Gists that have pages of output dumped at the end of the file in code comments. Reproducing the output is often extremely difficult, as it can be hugely dependent on the environment the code was run in. The Go Playground fixes all this. Front-end developers have a similar (but even more powerful) tool in JSFiddle.

Remote Code Execution as a Service

So what would it take to build a Go Playground that works for every language? The obvious issue is that the entire service would effectively be one big remote code execution vulnerability. To run such a service, there are a few desirable properties:

There are a number of ways of achieving each of these objectives (virtual machines, chroot, cgroups, etc), but Docker emerged as a pretty great solution that covers each of the points mentioned. Built on LXC and cgroups, it enables creation and teardown of relatively secure sandboxed environments in a fraction of a second. Each time a code snippet is run, a docker container can be created, started, used to run the untrusted code, then killed and destroyed. It’s incredible that all this can happen in a matter of milliseconds.

Making it happen with Docker

My implementation of this is up and running at codecube.io. Here’s a breakdown of how the system works:

The first step is to build a Docker image for the container that runs the code snippets. It includes everything necessary to run code in any of the supported languages (initially Python, Ruby, Go, and C). Here’s the corresponding Dockerfile:

FROM base
MAINTAINER Harry Marr <harry [at] hmarr.com>

RUN apt-get update
RUN apt-get install -y build-essential python ruby golang-go

ADD entrypoint.sh entrypoint.sh
ADD run-code.sh run-code.sh

ENTRYPOINT ["/bin/bash", "entrypoint.sh"]

Note there are two shell scripts added to the image. entrypoint.sh is a script that sets up an unprivileged user to run the code with, and run-code.sh detects the code’s language, and builds and runs it accordingly.

The server that accepts the code examples from the web, and orchestrates the Docker containers was written in Go. Go turned out to be a pretty good choice for this, as much of the server relied on concurrency (tailing logs to the browser, waiting for containers to die so cleanup could happen), which Go makes joyfully simple.

Resource Limiting

This is where the fun really begins. How do you stop someone from consuming 100% of the host’s CPU? What happens when someone runs while (1) { malloc(1024); }?

CPU

This is easy. Docker enables the allocation of CPU shares when starting a container. Note that this is a relative weighting, that affects how the processes in the container are scheduled. This functionality is provided by cgroups under the hood.

Memory

As with CPU, Docker exposes cgroups functionality that solves this problem. You simply specify the maximum amount of memory a container can use (in bytes) when you start the container.

Network

Fine-grained control is possible using tc, but for the initial version I opted to turn networking off. This is something I’d like to improve in a future version.

Disk

Limiting IO is currently an unsolved problem. There has been talk about using blkio to limit IOPS, however there are issues with this approach.

Disk quotas are also not easy - Docker doesn’t do much to help with this right now. Some people have worked around this problem by mounting a size-limited loopback device, and storing variable data exclusively in the mounted directory. This works well if you’re running a database or image-hosting service. But I wanted to apply the quota to the entire filesystem, as users have free reign over the filesystem. In theory you could place each container on its own loopback device, but that would require a lot of moving parts.

I solved the quota problem as follows:

This approach exposed something that was not entirely clear at first: it’s important to delete docker containers after using them, as by default they stay around forever. Every time you use docker run, a new container will be created, and remains on disk even after it has finished running. During early development, I found that the quotas weren’t being cleared after a code example ran. Updating the code to delete the containers as soon as they stopped resolved this issue.

Update: the newly-released Docker v0.6.4 adds a -rm flag to docker run, which auto-deletes containers once they finish running.

The End Result

CodeCube is up and running at codecube.io. It’s open source and available on GitHub.

Before pushing this live, I also added persistence. When you visit the application, you are redirected to a randomly-generated URL. Each time the code is run, it’s saved to Redis under the key that is present in the URL path. However, there’s still no proper authentication, so any snippet can currently be edited by anyone.

Things to Improve

The full code is up on GitHub - contributions are very welcome! If you’ve got any questions or suggestions, tweet me at @harrymarr.