Drupal performance tricks

Most people hosting Drupal sites experiences that their site becomes less responsive as the number of users raises and that their server load becomes very high. The first cure to levitate the issue is to buy more memory which may levitate the issue for now, but as the number of users raises once more that more memory may not be the right solution. This collection of posts will try to give some pointers to where you could start to get more out of your hardware by tweaking your configuration.

My standard setup runs on Debian Linux with PHP 5.3/5.4, MySQL 5.5.x and Varnish 3.x or newer. The web-server it self is either Apache or nginx based on the setup requirements. This series of articles will look into this stack and the extensions available to boots performance (APC, MemCached, PHP and Drupal tweaks).

You should be aware that many of the tricks described in this article is based on work by others that I have found on the Internet while learning to boost my own servers and so almost every section will have references back to these sources. So this is largely a compilation based on my experiences in hosting and developing Drupal sites over the years.

These posts will cover the following topics:

All scripts, code and most of the configuration files used in these posts can be found here http://github.com/cableman/configuration.

Part I: Linux Debian

Personally I prefers to use Debian Linux when building web-servers, but you could as well use any other flavor of Linux. If you want to use Debian 7 you should first test that your site runs with PHP 5.4 as that is the default version in Debian 7 (it can be downgraded).

Historically the stable releases of Debian comes without the newest version of the software required, so to remedy this I use the Dotdeb repository.

Lets get off the ground by adding the Dotdeb repository to our package manager and upgrade our system.

~$ echo "deb http://packages.dotdeb.org wheezy all" >> /etc/apt/sources.list
~$ echo "deb-src http://packages.dotdeb.org wheezy all" >> /etc/apt/sources.list
~$ wget http://www.dotdeb.org/dotdeb.gpg
~$ cat dotdeb.gpg | sudo apt-key add - && rm dotdeb.gpg
~$ apt-get update
~$ apt-get upgrade

If you not already have installed the basic packages needed to run a PHP based web-server this is the time to do so.

~$ apt-get install apache2-mpm-worker php5 php5-cli php5-gd php5-mysql php5-sqlite mysql-server php-pear php5-dev

Read more

If you want a more detailed walk through see my previous article Drupal 7 and Aegir server setup. It assumes that you are installing an Aegir based stack, if you do not which to run Aegir simply skip the Aegir parts of the article.

References

Part II: Apache

One of the challenges with Apache is to find the right number of threads to match your workload and at the same time not use up the servers memory. If you server is configured with to many threads it may start to use swap space to keep up with requests and you should see a drastically drop in response times. But before looking at threads you will need to enable/disable basic modules to ensure better performances.

Apache modules

I assume that you have activated mod_rewrite to get clean URLs running, but have you enabled mod_expires? This module handles HTTP expire headers which is used to set browser cache-control headers. Drupal's default htaccess file will make sure that static content (images, CSS, etc.) have a default two week expire header, which will give a performance boots by utilizing browser cache.

To enable the module run these commands on the server.

~$ a2enmod expires
~$ /etc/init.d/apache2 reload

Another module that might be worth a close look is mod_pagespeed, which is developed by Google. I have not my self used it yet, but thinks it's worth mentioning (See Drupal documentation for more information about this module).

You should look through the enabled modules and disable all modules not required by your site as they will system resources from each Apache process. Look in /etc/apache2/mods-enabled and at the Apache module list to figure out what you don't need to have enabled.

Here the authentication and index modules are disabled as I don't use them on my own servers.

~$ a2dismod auth_basic mod_authn_file autoindex

Threads

Configuring the number of clients and threads that Apache is allowed to spawn will determine the maximum number of simultaneous requests that the server can handle and the amount of memory used by Apache.

MaxClients

This setting determines the maximum number of child processes that can be spawned at any time. If there are more requests then processes to handle them the requests will be queued (maybe even timeout) and if the number of processes is to large there may not be sufficient memory on the system and swapping will kill your performance.

The magical number is the amount of available memory divided with the average size of your Apache child processes. Remember that we will need memory for other parts of the system as well, so you may need to come back and recalculate this number later on.

MaxClients = Memory / Child size

To find the average memory used by your Apache child processes this awk script below can be useful. This will only make yield a useable result on a running server which is handling requests. If you don't have a running server you should be safe guessing about 60-70 Mb, but this is very dependent on the number of Drupal modules and which modules you are using.

Simply create a file called memory.awk with the script below.

#/usr/bin/env awk
{
    count[NR] = $0;
    if (min == "") {
        min = max = $0
    };
    if ($0 > max) {
        max = $0
    };
    if ($0 < min) {
        min = $0
    };
}
END {
    print "Processes: ", length(count);
    if (NR % 2) {
        print "Median: ", count[(NR + 1) / 2], "\r";
    } else {
        print "Median: ", (count[(NR / 2)] + count[(NR / 2) + 1]) / 2.0, "\r";
    }
    print "Max:", max, "\r";
    print "Min:", min;
}

Run this command to feed the awk script with information about the currently running Apache processes.

~$ ps -o rss -C apache2 --sort:rss --no-headers | awk -f memory.awk

You should get a result like this, where the memory used in Kb is shown. You may need to run the script more than once to get a better picture of the usages.

Processes: 152
Median: 64976
Max: 78684
Min: 7440

If I have 2 Gb of memory available to run the Apache processes that would yield 31 MaxClients with a process size of 64.9 Mb as discovered above.

2048 / 65 = 31.5

Note: If you need more than 256 MaxClients you have to increase the ServerLimit setting as well, it defaults to 256 connections. Again remember to have enough free memory for the other optimisations and caches mentioned in this article, so return and adjust MaxClients.

Max Requests Per Child and Keep Alive

The MaxRequestsPerChild setting sets a limit no the number of request a child process can handle before it dies. It can be set to a value of a few thousands to prevent memory leaks on busy servers.

KeepAlive should be turned on so that each TCP connection can be used to send more than one request and thereby remove the overhead of establishing new connections for each request. The KeepAliveTimeout should be set to 2-4 seconds so the server does not wait to long for the client to send another request and thereby ties up resources.

Htaccess files

One of the expensive parts of running Apache is the lookup for htaccess files in the file system. So moving those into your Apache configuration and disable by setting allow override to none can also improve performance.

PHP-FPM (Not completed)

You should start by installing PHP-FPM as described here.....

~$ apt-get install libapache2-mod-fastcgi
~$ a2enmod actions fastcgi alias
~$ /etc/init.d/apache2 restart
<IfModule mod_fastcgi.c>
  AddHandler php5-fcgi .php
  Action php5-fcgi /php5-fcgi
  Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
  FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -host 127.0.0.1:9000 -pass-header Authorization
</IfModule>

Read more

For more information about getting Apache to run faster you should really read Configuring Apache for Maximum Performance from 2006, it still has many valid points.

References

Part III: Nginx (SSL end-point)

I use Nginx as a proxy to handle SSL/HTTPS request and change them into HTTP to run sites in HTTPS only but stile utilize varnish etc. First you have to setup Nginx and modify your Drupal installation, so it knows that it should generate links as HTTPS even though the request to it was HTTP.

Nginx

server {
  listen 443 ssl;

  server_name example.com;

  ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
  ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

  add_header Strict-Transport-Security max-age=15768000;

  location / {
    # Pass the request on to Varnish.
    proxy_pass http://example.localhost:6081;

    client_max_body_size 20M;

    # Pass a bunch of headers to the downstream server, so they'll know what's going on.
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # Most web apps can be configured to read this header and understand that the current session is actually HTTPS.
    proxy_set_header X-Forwarded-Proto https;

    # We expect the downsteam servers to redirect to the right hostname, so don't do any rewrites here.
    proxy_redirect off;

    # We get a lot of traffic, so we'll need a lot buffers.
    proxy_buffers 256 4k;
  }
}

# Redirect all standard site URLs to the secure version.
server {
  listen 80;
  server_name example.com;
  return 301 https://example.com$request_uri;
}

Varnish

As you turn secure communication into in-secure on the server (or your local network) you should update varnish to only trust defined proxies. This is done by adding an acl with the upstream proxies in your VCL.

# List of upstream proxies we trust to set X-Forwarded-For correctly.
acl upstream_proxy {
  "127.0.0.1";
  "172.21.0.0"/24;
}

You should update the VCL to include the ACL first in the vcl_recv function.

# Make sure that the client ip is forward to the client.
  if (req.restarts == 0) {
    if (client.ip ~ upstream_proxy && req.http.x-forwarded-for) {
      set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    }
    else {
      set req.http.X-Forwarded-For = client.ip;
    }
  }

Apache

Apache should not have the SSL module enabled, so it will not set the "X-Forwarded-Proto" header from the SSL proxy and Drupal will not be able to detect that it's behind a SSL Proxy. So you have to set the HTTPS flag in your vhost configuration file as shown below.

 SetEnvIf X-Forwarded-Proto https HTTPS=on

Drupal

In Drupal you should download and enable the sslproxy module and configure it, as show below in setttings.php (or "admin/config/sslproxy").

$conf['sslproxy_var'] = 'X-FORWARDED-PROTO';
$conf['sslproxy_var_value'] = 'https';

Part IV: PHP-FPM

Work in progress... sorry..

~$ apt-get install php5-fpm
~$ /etc/php5/pool.d/www.conf

References

Part V: MySQL

Database performance optimization is one of the harder things to explain as configuring MySQL right is no easy task and the values given here are all dependent on your web application and the amount of memory available. So the values given here are more guidelines then anything else and there is a whole book (High Performance MySQL) just about MySQL performance. I known that was not what you wanted to hear, so I'll try to explain the basics in relation to Drupal and want I have experienced as giving a performance boots.

When it comes to database performance one of the biggest bottlenecks is disk I/O which can be levitated by adjusting buffer sizes and memory usage. Getting them right can have a huge impact on performance, but getting them wrong can have a huge negative effect.The default configuration that comes with MySQL on most installations is meant to be all-around and is not optimized for high performance web hosting.

I'm assuming that you are using the InnoDB storage engine for all your tables as that seams to be the best fit for Drupal. If not you can use the Percona Toolkit to convert them by using the following command.

~$ apt-get install percona-toolkit
~$ pt-find DATABASENAME --ask-pass --user root --engine MyISAM --exce "ALTER TABLE %D.%N ENGINE=INNODB" --print

MySQL has many memory related variables that can be adjusted, but some are engine related and has no effect on InnoDB and should be ignored (See Jenny Chen's Weblog) for an in-depth overview. The following lists the settings that I usually changes.

MySQL's configuration file is located in /etc/mysql/my.cnf and will start by updating these values. Not all options may exist in your current configuration file in which case you should just add them to the file.

~$ nano -w /etc/mysql/my.cnf
query_cache_type = 1 
query_cache_size = 64M

table_open_cache = 1024

tmp_table_size = 16M
max_heap_table_size = 16M

join_buffer_size = 512K
read_buffer_size = 512K
sort_buffer_size = 1M

The first two lines enables the query cache and gives it 64 Mb to work with. This caches stores previous executed queries and speeds to selects in the database by using the result of already executed queries that yield the same results.

The table cache is used to store open tables, so the database don't have to access the disk each time a table is need to querying. The value can be is calculated by multiplying the number of most frequently used tables by the average number of concurrent connections (_ max_connections_ option, which defaults to 151 connections). You can run mysqladmin status to see how many tables you have open. For more information see How MySQL Opens and Closes Tables.

The table definition cache stores the table definitions (.frm files) and should match the number of tables you have.

This sets the maximum size of in-memory temporary tables and the limit it determined as the as the minimum of tmp_table_size and max_heap_table_size, hence we need to increase both. You should look server status variables Created_tmp_disk_tables, Created_tmp_files and Created_tmp_tables to see if you need to increase these values. See the section below about status variables for more information.

These buffers are used to speed up different query types and they are allocated for each connection and should not be larger then 1Mb as the full amount is always allocated.

Now lets look at the variables that are directly related to InnoDB. I normally create a new configuration file for these.

~$ nano -w /etc/mysql/conf.d/innodb.cnf
[mysqld]
innodb_buffer_pool_size=1G
innodb_flush_method=O_DIRECT
innodb_additional_mem_pool_size=20M
innodb_flush_log_at_trx_commit=0
innodb_thread_concurrency=8

Choosing the correct size for your InnoDB buffer pool size might be one of the most important performance optimizations that you can do with your MySQL configuration. It's used to store cached version of the data stored in your InnoDB database tables and with lots of readers can reduces the disk overhead. As a basic rule you should give it the size of your database table space plus 10%, but as it can be hard to guess the further size of your database you can always give it as much as you can spare. The extra 10% is to store insert buffers, hash indexes and locks.

The additional pool size is used to store internal dictionaries and structural data about your database and 20 Mb should be fin for most Drupal installations.

The flush log at a transaction commit value can be changed to 0, but with the side effect that any MySQL processes that crashes may not have saved the last second of transactions. I most cases this will be a fine trade-off for a little less IO.

By setting the flush method to O_DERECT you by-pass the operation systems disk cache and ensures that there is no double buffering. MySQL will handle the cache when writing files to disk. There is some talk around the net that this can have a negative effect on some system setups but I have never experienced any problems on the relative simple setups that I have used.

This setting will limit the number of active threads that InnoDB can use at once, which may be use full to protect yourself against thread thrashing.

MySQL status variables

To obtain more information about the currently running MySQL server and its current configuration variables these commands can be used.

~$ mysql -u root -p -e 'SHOW ENGINE InnoDB STATUS;'
~$ mysql -u root -p -e 'SHOW variables;'
~$ mysql -u root -p -e 'SHOW GLOBAL STATUS;'

Slow query log

You can use the Percona toolkit to analyse the slow queries log and get advises on how to make the queries execute more efficient. You can of cause use the same trick to analyse all your queries to optimize your application.

~$ pt-query-advisor /path/to/slow-query.log

Moving temporary tables into memory

If you have problems with slow queries do to large joins from views or other modules a performance boots can be gained by moving MySQL temporary tables into a RAM disk. See this blog post: Reduce your server's resource usage by moving MySQL temporary directory to tmpfs from 2bits.

Tuning primer

The following script can be used to analyse the current configuration for problems. The database should have been running for at least 48 hours before the scripts output is reliable. Download the last version of MySQL tuning primer.

~$ apt-get install bc
~$ wget https://launchpad.net/mysql-tuning-primer/trunk/1.6-r1/+download/tuning-primer.sh
~$ chmod u+x tuning-primer.sh
~$ ./tuning-primer.sh

Read more

You should really consider reading the High Performance MySQL book as mentioned in the beginning of this section.

References

Part VI: PHP

I assume that you are using PHP 5.3 or 5.4, which gives a significant performance boost compared to older versions. This section will look at memory usage, realpath cache, logging and op-code caching (APC) to boots performance.

Memory

Drupal contains many lines of code and you may have installed a large number of modules and if PHP runs out-of-memory your users will get the WSOD. So you may need to adjust the memory size based on the modules and you sites audience. I have seen users upload very large images that would make image styles crash do to insufficient memory during image resizing (See Find and Image Magick). So edit your php.ini, which on most installations is located in /etc/php5/apache2/php.ini.

Find the memory_limit line and change the amount that you need to run your site (most will suffices with 196Mb).

memory_limit = 196M

Realpath

In PHP 5.3 real path cache have been introduced which caches expensive file/directory lookups, thereby boosting performance. Realpath cache is maintained as a per thread cache and is cleared when the thread is killed.

Drupal uses a lot of files with relative paths (require_once/include_once) and it's here the realpath cache comes into play. On most systems the default size is 16K, which is to small for Drupal or other large PHP projects. Change the value in php.ini to something like the ones below to take better advantages of the cache.

realpath_cache_size = 256k
realpath_cache_ttl = 600

You can use the performance report module to get a look into the usage of realpath cache for your Drupal site.

Logging

In a production environment you should setup PHP to send log message to the systems sys-logger and not display errors to the end user (you have of cause used the E_STRICT setting during development).

Find and edit the lines below in php.ini.

error_reporting = E_ALL & ~E_DEPRECATED
display_errors = Off
display_startup_errors = Off
log_errors = On
error_log = php_errors.log
error_log = syslog

APC

The most important trick in getting PHP to run faster is using an opcode cache that stores the scripts as compiled byte code. This will give you a large performance boost as you skip the step of interpreting the PHP script every time its executed. Alternative PHP Cache (APC) stores the compiled byte code in memory ready for execution.

First you need to install APC, which can be done using the dotdeb sources or via pecl. I use dotdeb through apt-get so it will be updated together with the rest of the system.

~$ apt-get install php5-apc

The following settings configures APC to use a shared memory segment of 32Mb with a time to live (TTL) set to 4 hours and to not check the disk for new versions. The apc.stat tells APC not to check the disk for new versions of the script, which gives a yet a performance boots, but it also requires you to restart Apache to re-read the script if changed (or flush the APC cache from inside PHP). For more information see Optimizing APC for Drupal it's from 2010 but still applies.

apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=32M
apc.ttl=7200
apc.user_ttl=14400
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1
apc.cache_by_default=1
apc.stat=1

APC comes with a script that when accessed on you web-server shows the memory usage of APC and you should use it to optimize APC. If you give APC to little memory it will flush the cached files and you will have the opposite effect thereby getting a performance penalty. So use the script to give you cache the right amount of memory in relation to your site.

Other benefits that you may notice is that your Apache threads memory footprint gets smaller when using APC, so revisit the Apache section to adjust you threads and memory usages.

You can use the performance report module to get a look into the usage of APC from inside your Drupal site.

Part VII: Varnish

I front of your web-server you can place a revers proxy also known as an application cache to take the load of the whole server by serving content directly from memory and out the network card. Varnish is a very fast reveres proxy that is able to scale to a very high number of request per second.

Varnish can also be used as a load balancer to divide the request out over a number of backends, but here we only use varnish with a single default backend. I have earlier written about Varnish and Drupal 7 where more than one backend is defined.

First step is to configure your Apache to listen to another port (8080) and configure Varnish to listen to port 80 to capture all HTTP requests. Varnish's basic configuration is located in /etc/default/varnish and the following lines should be useable for most configurations. It allocations 512Mb of memory to varnish, if you have more memory to give it, it will like that.

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/drupal.vcl \
             -u varnish -g varnish \
             -S /etc/varnish/secret \
             -p thread_pool_add_delay=2 \
             -p thread_pools=4 \
             -p thread_pool_min=2 \
             -p thread_pool_max=4000 \
             -p session_linger=50 \
             -p sess_workspace=262144 \
             -s malloc,512m"

Varnish it self is configure in the VCL (varnish configuration language), which looks a lot like C code in syntax. The configuration is compiled into binary code when varnish is started to make varnish even faster. The code below should give you a good starting point for caching you Drupal site and should be place in /etc/varnish/drupal.vcl as defined by the configuration above.

# Define backend(s).
backend default {
  .host = "127.0.0.1";
  .port = "8080";
  .probe = {
    .timeout = 2s;
    .interval = 30s;
    .window = 10;
    .threshold = 2;
    .request =
      "GET /status.php HTTP/1.1"
      "Host: www.example.com"
      "Connection: close"
      "Accept-Encoding: text/html";
  }
}

# Respond to incoming requests.
sub vcl_recv {
  
  # Make sure that the client ip is forward to the client.
  if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
      set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    }
    else {
      set req.http.X-Forwarded-For = client.ip;
    }
  }

  # Do not cache these paths.
  if (req.url ~ "^/status.php$" ||
    req.url ~ "^/update.php$" ||
    req.url ~ "^/admin/build/features" ||
    req.url ~ "^/info/.$" ||
    req.url ~ "^/flag/.
$" ||
    req.url ~ "^./ajax/.$" ||
    req.url ~ "^./ahah/.$") {
    return (pass);
  }

  # Pipe these paths directly to Apache for streaming.
  if (req.url ~ "^/admin/content/backup_migrate/export") {
    return (pipe);
  }

  # Deal with GET and HEAD requests only, everything else gets through
  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  # Allow the backend to serve up stale content if it is responding slowly.
  set req.grace = 6h;

  # Use anonymous, cached pages if all backends are down.
  if (!req.backend.healthy) {
    unset req.http.Cookie;
  }

  # Always cache the following file types for all users.
  if (req.url ~ "(?i).(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[\w\d=.-]+)?$") {
    unset req.http.Cookie;
  }

  # Remove all cookies that Drupal doesn't need to know about. ANY remaining
  # cookie will cause the request to pass-through to Apache. For the most part
  # we always set the NO_CACHE cookie after any POST request, disabling the
  # Varnish cache temporarily. The session cookie allows all authenticated users
  # to pass through as long as they're logged in.
  if (req.http.Cookie) {
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
      # If there are no remaining cookies, remove the cookie header. If there
      # aren't any cookie headers, Varnish's default behaviour will be to cache
      # the page.
      unset req.http.Cookie;
    }
    else {
      # If there are any cookies left (a session or NO_CACHE cookie), do not
      # cache the page. Pass it on to Apache directly.
      return (pass);
    }
  }
  # Handle compression correctly. By consolidating compression headers into
  # a consistent format, we can reduce the size of the cache and get more hits.
  # @see: http:// varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.http.Accept-Encoding ~ "gzip") {
      # If the browser supports it, we'll use gzip.
      set req.http.Accept-Encoding = "gzip";
    }
    else if (req.http.Accept-Encoding ~ "deflate") {
      # Next, try deflate if it is supported.
      set req.http.Accept-Encoding = "deflate";
    }
    else {
      # Unknown algorithm. Remove it and send unencoded.
      unset req.http.Accept-Encoding;
    }
  }

  # If we get to here lookup the cache.
  return (lookup);
}

# Code determining what to do when serving items from the Apache servers.
sub vcl_fetch {
  # Varnish determined the object was not cacheable
  if (beresp.ttl <= 0s) {
    set beresp.http.X-Cacheable = "NO: Not Cacheable";

  # You don't wish to cache content for logged in users
  } elsif (req.http.Cookie ~ "(UserID|_session)") {
    set beresp.http.X-Cacheable = "NO: Got Session";
    return(hit_for_pass);
 
  # You are respecting the Cache-Control=private header from the backend
  } elsif (beresp.http.Cache-Control ~ "private") {
    set beresp.http.X-Cacheable = "NO: Cache-Control=private";
    return(hit_for_pass);
 
  # Varnish determined the object was cacheable
  } else {
    set beresp.http.X-Cacheable = "YES";
  }

  # Allow items to be stale if needed.
  set beresp.grace = 6h;

}

sub vcl_deliver {
  // Debugging code that should be removed in production.
  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }

  # Unset drupal cache and PHP info
  set resp.http.X-Powered-By = "Awesomeness and Open source";
  unset resp.http.X-Drupal-Cache;
}

Status script

As you may have noticed the definition of the backend above reference to a status.php file on the server. It's used to check if the backend is healthy by checking the database and Memcache operations. The script is shown below and can easily be extended to check for more things.

<?php

register_shutdown_function('status_shutdown');
function status_shutdown() {
  exit();
}

// Drupal bootstrap.
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_DATABASE);

// Build up our list of errors.
$errors = array();

// Check that the main database is active.
$uid = db_query('SELECT uid FROM {users} WHERE uid = 1')->fetchField();
if (!$uid) {
  $errors[] = 'Master database not responding.';
}

// Check that all memcache instances are running on this server.
if (isset($conf['cache_default_class']) && $conf['cache_default_class'] == 'MemCacheDrupal') {
  foreach ($conf['memcache_servers'] as $address => $bin) {
    list($ip, $port) = explode(':', $address);
    $memcache = new Memcache();
    if (!$memcache->addServer($ip, $port, false)) {
      $errors[] = 'Memcache bin <em>' . $bin . '</em> at address ' . $address . ' is not available.';
    }
    else {
      if (!$memcache->set('status_string', 'A simple test string')) {
        $errors[] = 'Memcache bin <em>' . $bin . '</em> at address ' . $address . ' is not available.';
      }
    }
  }
}

// Check that the files directory is operating properly.
if ($test = tempnam(variable_get('file_directory_path', conf_path() .'/files'), 'status_check_')) {
  if (!unlink($test)) {
    $errors[] = 'Could not delete newly create files in the files directory.';
  }
}
else {
  $errors[] = 'Could not create temporary file in the files directory.';
}

// Print all errors.
if ($errors) {
  $errors[] = 'Errors on this server will cause it to be removed from the load balancer.';
  header('HTTP/1.1 500 Internal Server Error');
  print implode("<br />\n", $errors);
}
else {
  // Split up this message, to prevent the remote chance of monitoring software
  // reading the source code if mod_php fails and then matching the string.
  print 'CONGRATULATIONS' . ' 200';
}

// Exit immediately, note the shutdown function registered at the top of the file.
exit();

Part VIII: Drupal

When it comes to Drupal most of the easy performance gains comes from making the right settings in settings.php and enable the right modules to support the cache technologies installed on the server.

Drupal Caching

Drupal comes with build in caching support for pages, blocks and a other content implemented in contribute modules. See the code section for information about using caches in your own code.

The following lines can be added to settings.php to enable all Drupal core caching.

$conf['cache'] = 1;
$conf['block_cache'] = 1;
$conf['preprocess_css'] = 1;
$conf['preprocess_js'] = 1;

When using forms in Drupal they are cached in the database and should not be moved into volatile storage as this could corrupt the forms if cache is cleared. So to ensure that this do not happen add the following line to settings.php.

$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

Logging

Drupal normally logs watchdog messages to the database, which can introduce an extra load on the database. So sending these to the servers system logging system and not the database can have a huge impact on performance.

First step is to enable the sys-logger and disable the database logger.

~$ drush en syslog -y
~$ drush dis dblog -y

You can configure the syslog by adding this line to settings.php thereby setting the identity used be the server to hit the right log file. You should change SITENAME in the following paragraphs to you sites name.

$conf['syslog_identity'] = 'SITENAME';

Next you have to tell your syslog where to log message from Drupal by adding a file to /etc/rsyslog.d/SITENAME.conf,

$template DRUPAL,"<%PRI%> %TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag% -- %msg%\n"
:syslogtag, contains, "SITENAME" /var/log/drupal/SITENAME.log;DRUPAL

Lastly restart the sys-logger to read the new configuration.

~$ /etc/init.d/rsyslog restart

Memcache(d)

Memcache is a key/value store that lives in memory only and can be used to move temporary data from the database server (cache tables) and direct into memory. This gives a significant performance boots and gives the database server more power to do want it does best.

To enable Drupal to use Memcache for its cache tables you simply install memcached and the PHP integration extension.

~$ apt-get install memcached php5-memcache

To configure memcached edit the file /etc/memcache.conf where the important options are -m (memory) and -l (list). Its important that you protect memcache, by link it to the local interface (127.0.0.1) because memcache have no data protection, so every one can read its contents.

Next download the Memcache module from Drupal.org but you should not enable the module. It contains a new cache backend for Drupal, which you configure via the sites settings.php.

// Memcache configuration.
$conf += array(
  'memcache_extension' => 'Memcache',
  'show_memcache_statistics' => 0,
  'memcache_persistent' => TRUE,
  'memcache_stampede_protection' => TRUE,
  'memcache_stampede_semaphore' => 15,
  'memcache_stampede_wait_time' => 5,
  'memcache_stampede_wait_limit' => 3,
  'memcache_key_prefix' => basename(realpath(conf_path())),
);

// Include cache backends.
include_once('./includes/cache.inc');
include_once('./sites/all/modules/contrib/memcache/memcache.inc');

$conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';

// Configure cache servers.
$conf['memcache_servers'] = array(
  '127.0.0.1:11211' => 'default',
);
$conf['memcache_bins'] = array(
  'cache' => 'default',
);

Memcached profiling

Memcache may not be performing optimal and you can profile and investigate its performance using tcpdump and the Percona toolkits mk-query-digest program to analyse the communication to your installation of memcache.

See Memcached server profiling with mk-query-digest article for more information about profiling memcache. There are also information available in the documentation at Percona just search for memcache on the page.

Used memory

The PHP script below can be used to find the current usage of memory and thereby help finding the right amount of memory that you should allocate to memcache.

<?php
// Print sizes in a pretty format.
function bsize($size) {
  foreach (array('', 'K', 'M', 'G') as $k) {
    if ($size < 1024) {
      break;
    }
    $size /= 1024;
  }
  return sprintf("%5.1f %sBytes", $size, $k);
}

// Connect to memcache.
$memcache = new Memcache();
$memcache->addServer('localhost', 11211, false);

// Get stats.
$stats = $memcache->getExtendedStats();
$stats = $stats['localhost:11211'];

// Print information about memcache usage.
echo 'Size: ', bsize((real)$stats['limit_maxbytes']), "\n";
echo 'Used: ', bsize((real)$stats['bytes']), "\n";
echo 'Free: ', bsize($stats['limit_maxbytes'] - $stats['bytes']), "\n";
echo 'Read: ', bsize((real)$stats['bytes_read']), "\n";
echo 'Written: ', bsize((real)$stats['bytes_written']), "\n";

echo "\n",'Used percent',"\n";
$used = ((real)$stats['bytes']/(real)$stats['limit_maxbytes']) * 100;
echo 'Used: ', sprintf('%01.2f', $used), "%\n";
echo 'Free: ', sprintf('%01.2f', 100-$used), "%\n";

You can also use the performance report module to look at the memcache usage from inside Drupal.

Another solution to look into the performance of Memcached is the PHPMemcacheAdmin, which gives you a graphical administration interface (like phpMyAdmin for MySQL).

Varnish (Not completed)

// Varnish.
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
$conf['page_cache_invoke_hooks'] = FALSE;
$conf['varnish_flush_cron'] = 1;
$conf['varnish_version'] = 3;
$conf['varnish_control_key'] = 'FOUND IN VARNISH SECRET FILE';

APC

The fastest cache that you can use in relation to PHP have to by the Op-code cache, so this would be a good place to store the bootstrap cache table. We could in theory use APC for all cache tables but we would ran into problems because APC is really bad at flushing/removing entries from its cache. So it would get fragmented and thereby trashing the performance.

$conf['cache_backends'][] = 'sites/all/modules/contrib/apc/drupal_apc_cache.inc';
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';

Entity cache

Drupal 7 uses the concept of entities under the hood, so caching the load of these will import performance (specially in combination with memcached). So simply download the module and activate it. That's it.

The final settings.php

<?php
// Drupal cache.
$conf['cache'] = 1;
$conf['block_cache'] = 1;
$conf['preprocess_css'] = 1;
$conf['preprocess_js'] = 1;

// Forms cache table.
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

// Memcache configuration.
$conf += array(
  'memcache_extension' => 'Memcache',
  'show_memcache_statistics' => 0,
  'memcache_persistent' => TRUE,
  'memcache_stampede_protection' => TRUE,
  'memcache_stampede_semaphore' => 15,
  'memcache_stampede_wait_time' => 5,
  'memcache_stampede_wait_limit' => 3,
  'memcache_key_prefix' => basename(realpath(conf_path())),
);

include_once('./includes/cache.inc');
include_once('./sites/all/modules/contrib/memcache/memcache.inc');

$conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_servers'] = array(
  '127.0.0.1:11211' => 'default',
);
$conf['memcache_bins'] = array(
  'cache' => 'default',
);

// APC.
$conf['cache_backends'][] = 'sites/all/modules/contrib/apc/drupal_apc_cache.inc';
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';

// Logging.
$conf['syslog_identity'] = 'SITENAME';

// Varnish.
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
$conf['page_cache_invoke_hooks'] = FALSE;
$conf['varnish_flush_cron'] = 1;
$conf['varnish_version'] = 3;
$conf['varnish_control_key'] = 'FOUND IN VARNISH SECRET FILE';

$conf['cache_lifetime'] = 0;
$conf['page_cache_maximum_age'] = 21600;

References