Skip to main content

Varnish saves the day, in unexpectedly awesome ways.

Four and half years ago, I wrote a blog post about Varnish, a 'front-end proxy' for webservers. My best description of it then was as a protective bubble, analogous to how it's namesake is used to protect furniture. I've been using it happily ever since.

How Americans Voted poster
But last week, I got to really put Varnish through a test when the picture here, posted by Fair Vote Canada (one of my clients), went viral on Facebook. And Varnish saved the server and the client in ways I didn't even expect.

1. Throughput

Varnish prides itself on efficiently delivering http requests. As the picture went viral, the number of requests was up to about 1000 per minute, which Varnish had no trouble delivering - the load was still below 1, and I saw only a small increase in memory and disk usage. Of course, delivering a single file is exactly what Varnish does best.

2. Emergency!

Unfortunately, Varnish was not able to solve a more fundamental limitation, which was the 100Mb/s network connection. Because the poster was big (760Kb), the network usage, which is usually somewhere in the 2-5 Mb/s range, went up to 100Mb/s, and even a bit beyond. That meant the site (and others sharing that network connection) started suffering slow connections, and I got a few inquiries about whether the server had 'crashed'.

At that stage, I had no idea what was actually going on, just that requests for this one file was about to cause the site as a whole to stop responding. I could see that the referrer was almost exclusively facebook, I also noticed that the poster on it's own wasn't really helping their cause, and the client also had no idea that it was happening - they had uploaded the poster to facebook, so it shouldn't be requesting it from their site.

Fortunately, because the limitation was in the outgoing network, there was a simple solution - stop sending the poster out. With a few lines in my varnish VCL, the server was now responding with a simple 'permission denied', and within a few seconds, everything settled down.

In fact, the requests kept coming in, at ever higher numbers, for the rest of the day, but Varnish was able to deflect them without any serious blip in the performance of the server.

3. And Better

The next day, after some more diagnostics, we discovered that the viral effect had actually come from someone else's facebook post who shared the poster as it had gone out in an email. Although the poster on it's own wasn't going to help the cause of PR directly, we didn't really want to stem whatever people were getting out of it, so I uploaded the poster to an Amazon S3 bucket, (an industrial file service) and modified my varnish vcl to now give a redirect to the amazon copy instead of a permission denied.

Now the poster could go safely viral.

4. And Best

After some more discussion, Fair Vote noted it would be better if people ended up on the facebook campaign url here  rather than just the poster. So I updated the varnish vcl so that if the poster request comes from a facebook referrer, then it redirects them instead to the above url.

Four days later now, it seems like it's worked - the poster is still pretty viral, and even the requests for the original url is still going strong (3.4 million requests in the 48 hours ending at 3 am this morning).

Without Varnish, my server would have crashed and been unable to get back up, even now. Instead, the poster is still being shared, the rest of the site is still working, and the facebook share is even more effective than it would have been.

Popular posts from this blog

Orchestrating Drupal + CiviCRM containers into a working site: describing the challenge

In my previous posts, I've provided my rationale for making use of Docker and the microservices model for a boutique-sized Drupal + CiviCRM hosting service. I've also described how to build and maintain images that could be used for the web server (micro) service part of such a service.

The other essential microservice for a Drupal + CiviCRM website is a database, and fortunately, that's reasonably standard. Here's a project that minimally tweaks the canonical Mariadb container by adding some small configuration bits: https://github.com/BlackflySolutions/mariadb

That leaves us now with the problem of "orchestration", i.e. how would you launch a collection of such containers that would serve a bunch of Drupal + CiviCRM sites. More interestingly, can we serve them in the real world, over time, in a way that is sustainable? i.e. handle code updates, OS updates, backups, monitoring, etc? Not to mention the various crons that need to run, and how about things like…

Building and maintaining Drupal + CiviCRM application containers

In my previous two posts, I provided some background into why I decided on using containers for a boutique Drupal + CiviCRM hosting platform, and why Docker and its micro-services approach is a good choice for building and maintaining containers.

Although I promised to talk about orchestration, that was getting ahead of the story - first I'm going to look at the challenge of keeping your application containers up-to-date with OS and application-level updates. There's a fair amount of work in that, but the tooling is mature and there is lots of good documentation.

A great place to start is to visit the official Drupal docker hub page. From there, you can pull a working Drupal code container, and it gets re-built frequently with all the OS and Drupal-code updates, so you just refresh your containers whenever you want (i.e. whenever a security release comes out, or more often to stay up-to-date).

A nice thing about that project is that it demonstrates a technique for maintaining …

The Tyee: Bricolage and Drupal Integration

The Tyee is a site I've been involved with since 2006 when I wrote the first, 4.7 version of a Drupal module to integrate Drupal content into a static site that was being generated from bricolage. About a year ago, I met with Dawn Buie and Phillip Smith and we mapped out a number of ways to improve the Drupal integration on the site, including upgrading the Drupal to version 5 from 4.7. Various parts of that grand plan have been slowly incorporated into the site, but as of next week, there'll be a big leap forward that coincides with a new design [implemented in Bricolage by David Wheeler who wrote and maintains Bricolage] as well as a new Drupal release of the Bricolage integration module.PlansApplication integration is tricky, and my first time round had quite a few issues. Here's a list of the improvements in the latest version:File space separation. Before, Drupal was installed in the apache document root, which is where bricolage was publishing it's content. This …