You are here

Varnish and Drupal 7

This article explains how to install Varnish reverse-caching in front of Drupal 7 sites to hence performance. The configuration assumes that you have a stand-alone Varnish server, which sends requests to different back-ends (Drupal servers) base on the site requested. The configuration can however easily be modified to run on the same server with one site, if required.

The most important thing about running an reverse-cache (Varnish) is to have enough memory, as caching images and page can take op a lot of memory. I would recommend 1 Gb of memory for a site with around 6000 visitors a day. This guides configurations assumes that you have a Debian 6 server with 1 Gb of dedicated memory to Varnish.

Server configuration

I have previous written about setting up servers in the article about Aegir hosting, which can be found here. If you are planing on having a dedicated Varnish server, the basic steps are the same as for the Aegir host. You should also configure a firewall on the server, if it's dedicated to Varnish, which can be found here.

Varnish installation

Installing Varnish on a Debian server is described at http://www.varnish-cache.org/installation/debian, but the steps is also listed below.

~$ wget http://repo.varnish-cache.org/debian/GPG-key.txt
~$ apt-key add GPG-key.txt
~$ echo "deb http://repo.varnish-cache.org/debian/ $(lsb_release -s -c) varnish-3.0" >> /etc/apt/sources.list.d/varnish.list
~$ apt-get update
~$ apt-get install varnish

Varnish configuration

The configuration used in this articles is based on this article at lullabot. I have made some changes in relation to that article to match the set up that I uses. I recommend reading the lullabot article for more in-depth information about the Varnish Configuration Language (vcl) options and the documentation at https://www.varnish-cache.org.

Start by edit the default configuration at /etc/default/varnish, you should notice that Varnish only handles http request not https. There are several solutions to this problem, such as setting up nginx to proxy https requests.

DAEMON_OPTS="-a :80\
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -u varnish -g varnish \
             -S /etc/varnish/secret \
             -p thread_pool_add_delay=2 \
             -p thread_pools=4 \
             -p thread_pool_min=2 \
             -p thread_pool_max=4000 \
             -p session_linger=50 \
             -p sess_workspace=262144 \
             -s malloc,768M"

This configures Varnish to listen on port 80 (http) for requests and use 768 Mb of memory for caching, which is not much but will match a server with 1 Gb of available memory. The pool and thread numbers are based on a server with 4 CPU's (for more information see the lullabot article mentioned above).

Varnish Configuration Language

Varnish has it own configuration language which is used to tune Varnish to the type of site that you are running, so that you can gain a higher performance boots. The configuration file used as default is located at /etc/varnish/default.vcl. I use the configuration shown below.

backend www { .host = "192.168.176.184"; .port = "80"; }
backend www2 { .host = "192.168.176.185"; .port = "80"; }

# Respond to incoming requests.
sub vcl_recv {
  if (req.http.host ~ "(?i)^(www.)?test1.dk$") {
    set req.backend = www;
  }
  else if (req.http.host ~ "(?i)^(www.)?test2.dk$") {
    set req.backend = www2;
  }

  # Make sure that the client ip is forward to the client.
  if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
      set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    }
    else {
      set req.http.X-Forwarded-For = client.ip;
    }
  }

  # Do not cache these paths.
  if (req.url ~ "^/status.php$" ||
    req.url ~ "^/update.php$" ||
    req.url ~ "^/ooyala/ping$" ||
    req.url ~ "^/admin/build/features" ||
    req.url ~ "^/info/.$" ||
    req.url ~ "^/flag/.
$" ||
    req.url ~ "^./ajax/.$" ||
    req.url ~ "^./ahah/.$") {
    return (pass);
  }

  # Pipe these paths directly to Apache for streaming.
  if (req.url ~ "^/admin/content/backup_migrate/export") {
    return (pipe);
  }

  # Allow the backend to serve up stale content if it is responding slowly.
  set req.grace = 6h;

  # Use anonymous, cached pages if all backends are down.
  if (!req.backend.healthy) {
    unset req.http.Cookie;
  }

  # Always cache the following file types for all users.
  if (req.url ~ "(?i).(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[\w\d=.-]+)?$") {
    unset req.http.Cookie;
  }

  # Remove all cookies that Drupal doesn't need to know about. ANY remaining
  # cookie will cause the request to pass-through to Apache. For the most part
  # we always set the NO_CACHE cookie after any POST request, disabling the
  # Varnish cache temporarily. The session cookie allows all authenticated users
  # to pass through as long as they're logged in.
  if (req.http.Cookie) {
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
      # If there are no remaining cookies, remove the cookie header. If there
      # aren't any cookie headers, Varnish's default behavior will be to cache
      # the page.
      unset req.http.Cookie;
    }
    else {
      # If there are any cookies left (a session or NO_CACHE cookie), do not
      # cache the page. Pass it on to Apache directly.
      return (pass);
    }
  }
  # Handle compression correctly. Different browsers send different
  # "Accept-Encoding" headers, even though they mostly all support the same
  # compression mechanisms. By consolidating these compression headers into
  # a consistent format, we can reduce the size of the cache and get more hits.
  # @see: http:// varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.http.Accept-Encoding ~ "gzip") {
      # If the browser supports it, we'll use gzip.
      set req.http.Accept-Encoding = "gzip";
    }
    else if (req.http.Accept-Encoding ~ "deflate") {
      # Next, try deflate if it is supported.
      set req.http.Accept-Encoding = "deflate";
    }
    else {
      # Unknown algorithm. Remove it and send unencoded.
      unset req.http.Accept-Encoding;
    }
  } 
  return (lookup);
}

# Code determining what to do when serving items from the Apache servers.
sub vcl_fetch {
  # Allow items to be stale if needed.
  set beresp.grace = 6h;
}

TODO:

  • Add support for probe, to check server health.
  • Add default error page when back-end is down.
  • Add explanation of include pr. site/back-end server configuration.

Drupal configuration

We need to tell Drupal that it's behind a proxy/caching server or it will not set the correct headers and Varnish will not cache pages. You should add the lines below to your settings.php.

// Tell Drupal it's behind a proxy.
$conf['reverse_proxy'] = TRUE;

// Tell Drupal what addresses the proxy server(s) use.
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

// Bypass Drupal bootstrap for anonymous users so that Drupal sets max-age < 0.
$conf['page_cache_invoke_hooks'] = FALSE;

// Make sure that page cache is enabled.
$conf['cache'] = 1;
$conf['cache_lifetime'] = 0;
$conf['page_cache_maximum_age'] = 21600;

Varnish module

It's possible to make Drupal communicate with Varnish and automatically invalidate cache entries based on cache clears in Drupal. It is also possible to get statistical information form Varnish. Start be downloading and enable the Varnish module (http://drupal.org/project/varnish).

~$ drush dl varnish
~$ drush en varnish

If you have Varnish running on the same server as Drupal you can simply go to admin/config/development/varnish and enable the communication. The page requires to known the Varnish control key, which can be found in /etc/varnish/secret. If Varnish is running on another server a SSH tunnel create with autossh can be used to enable the communication, as shown below.

~$ autossh -fN <varnish server> -L 6082:localhost:6082

You can tell Drupal to use varnish as page cache, which will ensure that varnish purges page from the cache when Drupal update pages. You can do this be adding this to the settings.php file.

// Add Varnish as the page cache handler.
$conf['cache_backends'] = array('sites/all/modules/varnish/varnish.cache.inc');
$conf['cache_class_cache_page'] = 'VarnishCache';

Helpful Varnish commands

Varnish comes with a range of different utility programs, that can help keeping an eye on the performance of Varnish and help resolve caching issues.

Varnish administration

At some point you will have to flush the cache and the commando below will do that. You can change "." to match the path that you which to flush from the cache,

~$ varnishadm -T 127.0.0.1:6082 url.purge "."

Varnish history

The Varnish history command shows the distribution of the last N requests processing. Hits are marked with a pipe character ("|"), and misses are marked with a hash character ("#").

~$ varnishhist

Varnish top

The command below shows a list over the top most requests sent to the back-end, hence the URLs that are not in the cache.

~$ varnishtop -b -i TxURL

Comments

how to increase the compression level of deflate or gzip like. htaccess in varnish?

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.