Four years ago, I started a github project to share my Drupal + CiviCRM container hosting solutions/ideas. It’s called “Simuliidae”, because that’s the family name of the species we know as black flies.
What’s Happened
Like so many open source projects, it was ambitious. I claimed four goals. Four years later, here’s my evaluation of those goals:
1. A simple way for evaluators to launch their own local Drupal/CiviCRM installation.
This was my priority and relatively successful, at least for a while. I’d say it was successful as a technology demonstration, but of limited value for a potential CiviCRM evaluator because of weak support for Docker on Windows and Mac projects. It was also a lot of work to keep it up to date, and I didn’t.
2. A standard for generating testing and development copies of production sites.
This was not particularly successful as a goal. Although I used some of the ideas here with my hosting services, I never moved beyond an idea stage for trying to come up with a standard that anyone else might be willing to adopt. With the change in how CiviCRM got installed with Drupal 8, it became a more complicated goal.
3. A basis for an automated Drupal/CiviCRM demonstration site generator.
My first version of this was tested at a CiviCRM sprint, with mixed results. I wasn’t convinced I had the right model and have been thinking about this ever since without coming up with a new method. I was particularly concerned with the privacy, security and scalability issues related to self-service site creation.
4. A basis for a multi-site host.
Using containers for my hosting service was the original impetus for exploring Docker, and this project was my attempt at open sourcing what I had learned after I started using Docker in production.
On the other hand, the demands and details of running a hosting service are difficult to capture in a coherent project, so I wasn’t using this project directly, but instead trying to push my code and practices out to the project as they evolved and got more convincing.
So that leads me to the meat of this blog post.
Why Now
I’m in the midst of setting up a couple of new servers and that’s an opportunity to evolve how I set them up. Of course, with the continuous changes to the security landscape of the Internet and the tools available, change to machines and their resources and costs, changes to Linux and their distributions, and changes to Docker itself, every new machine requires some reconsideration of how things are set up.
So this time, I decided it was time to merge some of my new ideas back into the Simuliidae project and see if I can use it directly on my servers - i.e. test and force goal number four. A version of “eating your dog food”.
What’s New
There are two big areas of change in v2: building, and composing. Which of course, is pretty much everything, until I include my server setup stuff.
In my initial project, I had combined the build and compose pieces, since that’s the usual way (or was at the time) when using the standard docker-compose tool. But because I wanted to base my images on the standard Drupal Docker image, it became a complicated process to maintain the slightly altered Drupal base image, followed by adding new stuff for the Simuliidae images. Trying to keep everything up to date with security releases was pretty onerous, and with the advent of Drupal 8, it required a whole new strategy which was rapidly evolving.
For v2, I’ve split out the build and compose parts into separate directories, which makes things more maintainable and makes more sense anyway because you can use these images in different ways and needn’t be dependent on their building.
A New Build Strategy
A couple of years ago, I forked the project to explore the (at that time) new Docker build thing called “multistage build”. My goal was to try and simplify the build in a way that allowed me to only maintain one Docker file per variant instead of the multiple builds with dependencies of the v1 design.
I used that design to generate some useful images that I used in production, but now I had the opposite problem of requiring complex Dockerfiles that were difficult to manage over time.
For v2, I finally paid attention to how the Drupal docker file is maintained, and I’ve implemented a compromise, using the git “submodule” feature in a new “reference” directory, to make both Dockerfile management and image management easier.
Here’s my readme for how it now works: https://github.com/BlackflySolutions/Simuliidae/blob/master/build/README.md
My holy grail is that I can set up automation so that when security and minor releases come out, new versions of my existing images are automatically generated within 24 hrs.
One of the results of this change is a more sophisticated and complicated image naming convention, which I don’t love, but I accept as the price of automation.
Wordpress?!
I’m not now, and never have been, a fan of including CiviCRM in a wordpress install. Partly, that’s just my bias, I know Drupal a lot better. But what I do know of Wordpress has convinced me that it’s not a wise idea from the point of view of maintainability, privacy and security, i.e. for anything but a small install with minimal component use.
On the other hand, a few clients use Wordpress for their main site and I host their CiviCRM on Drupal as a subdomain, and I’m okay with that kind of setup. About a year ago, a client asked if I could host their Wordpress, and I decided it was worth seeing if my Docker setup would work with Wordpress. As well as potentially being useful, it was also an opportunity to include a discipline in my code that would abstract away any Drupal specificity.
It turned out to be relatively straightforward, and I’m now hosting a number of wordpress sites, using this project: https://github.com/BlackflySolutions/Ceratopogonidae which is a relatively naive copy/hack version of Simuliidae. “Ceratopogonidae” is a family of flies commonly known as no-see-ums, or biting midges.
But of course, I don’t really want to be maintaining similar bits of code in two projects, and to really force the CMS-agnostic discipline, I’ve restructured Simuliidae to include wordpress.
I’m still not actually using Wordpress with CiviCRM. I’m really only interested in the version without CiviCRM that this project supports.
Composing
The “compose” part of using Docker images is sometimes referred to as “orchestration”, but these are confusing terms. The real scope of this part of the project is: “how can I use these images for a working site”?
In fact, that question is a little too broad in scope as well. The Docker Way of containers is to split your application’s “processes” into separate containers that all work together in a “stack”. And the Docker way of defining which images are involved and how they are connected together is via the docker “compose” file, which is a nice example of a declarative document that captures a kind of “infrastructure in code”. And of course it uses YAML syntax.
A CiviCRM install has a web server and a database, at minimum, and the standard demonstration Docker compose file has two images, with the webserver exposed outside the mini-network created by the Dockerfile.
But that’s not a great way to set up a civicrm install in production, particularly if you want scale and performance. So with v1, I developed a way to generate more interesting and useful compose files, as described in here:
https://civicrm.org/blog/alandixon/simuliidae-open-source-design-hosting-containers
For v2, I ran into the challenge that one of the features of the docker-compose tool has been dropped in the past year. For now, I’m using an older version that still works, but I’ll have to update this sometime in the next year.
But also for v2, I’ve added some improvements to that script and will be adding some additional yml files that I use in production. For example:
Imapd.yml - Adds a little imapd server that mounts the postfix virtual mailbox directory from the host - I use this to get civimail bounce messages.
Vredis.yml - IMHO, all drupal website should use redis! This adds a simple redis server, you can configure how much memory it uses.
Deploy.yml - a collection of settings used for deploying a site - things like memory and cpu limits, and passing in of labels and environment variables.
Sftp.yml - a small sftp container that allows designers or coders limited access. I use this for example to give access to install modules or extensions, or to customize themes.
Next
After my new servers launch using this project, I’ll turn to the idea of setting up an automated test site generator.