Integrating ZooKeeper into a Docker environment can get tricky, if you consider clustering support and use Exhibitor for supervision. In this post, I’m documenting my own process with this stack, a customized Netflix image and a single boot2docker virtual machine as the hosting environment.
Update: The process gets significantly easier, when using
docker-machine for virtual host management. Check it out here.
Running a single ZooKeeper instance
Let’s start by running a standalone Exhibitor container, using the NetflixOSS image from Docker Hub:
The image is set up to spawn a ZooKeeper instance and an Exhibitor process for supervision, which should be sufficient for most development purposes and for getting the feel of the service. Once the container is up, you can head to the Control Panel to confirm the status and explore the data.
To access the panel with
boot2docker on Mac OS X:
To launch the ZooKeeper CLI in the running container:
Note 2: You should remove the container once you’re finished, since we’re going to bind those ports in the next phase as well. Use
docker rm -f exhibitor to do this in one step.
Running a ZooKeeper ensemble
Let’s add a high availability component by setting up an ensemble of nodes. As suggested by the ZooKeeper Administrator’s Guide, it will consist of five nodes, which will allow for two of them to fail, and still keep the service operational (a majority is required; check the guide for more details).
For demonstration purposes, we’re going to set them up on a single Docker instance, which will be running on the boot2docker virtual machine. This should never take place in production of course, but it should be enough for getting a high-level picture of the environment.
Preparing network interfaces on the Docker host machine
To emulate multiple Docker nodes on Mac OS X, we’re going to add some additional IP addresses to the boot2docker virtual machine. That way, we can associate each node in the ZooKeeper ensemble with a unique IP and provide a clear hostname mapping later.
Let’s start by finding out the network interface and the IP address, which are used by the virtual machine:
As you can see, my boot2docker VM is talking over eth1/192.168.59.103. Let’s add some additional IPs to this interface:
Once you run the script, make sure that you can access the VM using all 5 IPs.
Preparing a custom Docker image for Exhibitor
An Exhibitor installation can use one of the several config types, which will imply the storage solution for sharing configuration within the cluster. For example, you could store your cluster config in an Amazon S3 bucket, in an external ZooKeeper server or using a shared property file on a dedicated filesystem.
In any case, you’ll need some sort of customization for the Exhibitor container, since the base image isn’t production ready for the moment. For these reasons, we’re going to modify the base Docker image and setup a shared property file, which will handle the distributed config management.
Start by fetching the base Netflix template from GitHub:
Dockerfile and add a
--fsconfigdir switch as shown below:
You can replace
/exhibitor/shared with any reasonable location, as long as it stays consistent with the volume switch in
Build a new image using the modified
Create a base config file for Exhibitor
Copy the property file from the Git repository to a shared directory on the
boot2docker virtual machine:
Spawning Exhibitor instances using Docker
Finally, we can start spawning our ensemble:
Once the script is finished and all instances are up and running, you can proceed with the clustering setup. Open the Control Panel on
… and set the
Config -> Ensemble -> Servers section to the following value:
Commit -> All at once and wait. Once the config gets updated, you should see a complete list of servers with a green status icon next to each item:
Keep in mind, that the initial startup may take some time. If you’re impatient or something goes wrong, use the
logs command to see what’s going on inside a container (e.g.
docker logs -f exhibitor1).
You can use the following Scala template to check: connectivity, replication, node failures and possibly other cases, that may require a standalone client.
Just clone the repository and launch
sbt ~run in the root directory of the project.
Please note, that if a
znode is created using the
EPHEMERAL mode, then it will be available for the active session only. You may want to change this, depending on your test case requirements. Check Curator and ZooKeeper docs for more information regarding ephemeral nodes and the client implementation itself.
This is my first technical post, so thanks for getting this far :-) If you have any remarks or suggestions, then please feel free to send me an e-mail or leave a comment below.