Drupal server performance can be tricky especially with high traffic sites. If you are upgrading from an existing platform or migrating to a new server you can use the Apache access logs to generate close to real workloads, using Seige, on the new server and make adjustments before going live.

Note: The script have now been updated to use PHP 5.4 and SQLite3 (13/05/14).

Analyzing Apache access logs

Siege uses a file with URLs that it should "hit" and to generate this file based on Apache access logs you can use this PHP script that I have created. It's some what CPU and IO hungry, so if you are using it give it some time to execute. I have used it on several GB of Apache logs and used to test several sites. The script uses a local SQLite database to store analyzed log files, so URLs can be generated form different conditions to optimize your load tests.

You can download the script from GitHub here: Access2Siege

To analyze an Apache access file the following command can be used, but be aware that this match the logging format I had for my Apache. So you may need to change the pattern (-p) and time format (-t) parameters. See the documentation in the top of the script for more information about its usage.

~$ ./access2siege.php -i access.log -p "0,3,6,8" -t "d/m/y:H:i:s"

After you have loaded all your access information into the SQLite database, the script can provide statistical information about the content in the database. For more information about the possibilities see the documentation inside the script.

~$ ./access2siege.php -c urls
~$ ./access2siege.php -c urls -x '/user
~$ ./access2siege.php -c ips

The first shows the number of URLs in the database. The next shows the number of URLs that starts with "/user" and the last counts the number of IPs in the database. You can use this information to make decisions about output generating later.

Large access logs

It takes time to analyze large access logs, so you may want to split the logs into smaller parts and only analyze some of theme. The following commands splits a log file into 25 Mb pieces and analyzes the ones containing the names in the parentheses.

~$ split -b 25m access.log.3 access_3_logs_
~$ for fh in access_3_logs_*(a|b|c|h|j|k|t|q|w|g|p|r); do ./access2siege.php -i $(fh) -p "1,4,7,9" -t "d/m/y:H:i:s"; done

Outputting URLs

The following command will generate Siege URL files each containing 1.000.000 URLs, which can be used to "hit" the server http://stg.site.

~$ ./access2siege.php -o <site>_{n}.txt -d http://stg.<site> -l 1000000

Installation

It's time to install Siege either from your package system or from source code, as shown below.

~$ wget http://www.joedog.org/pub/siege/siege-latest.tar.gz
~$ tar -zxvf siege-latest.tar.gz
~$ cd siege-2.70
~$ ./configure --prefix=/opt/siege-2.70
~$ make
~$ sudo make install
~$ sudo ln -s /opt/siege-2.70/bin/* /usr/local/bin/

Installation on Mac

If you are you are using a Mac, you can use home brew to install Siege, as shown below.

~$ brew install siege

Running Siege

If you simply want to test that Siege is correctly installed or just run a simple load test against a single URL use this command. It will execute 10 concurrently threads and stop after 4 minutes in verbose mode, so you can see what its during.

~$ siege -c 10 -t 4M -v http://linuxdev.localhost

The result of the test above could look like this, where you should look at the transaction rate and throughout. Do not forget to look at failed transactions, as they may occur if the server stops responding when the load gets to high.

To run Siege with a URL file simply add the -f and the file you want to use and it will "hit" those URLs.

~$ siege -c 100 -d 3 -t 60m --log=siege.log -f access-urls-1.log

Logged in

It's possible to run Siege as a logged in user by using a configuration file with the following content (remember to modify the credentials).

logfile = siege.log
verbose = false
fullurl = true
show-logfile = true
logging = true
logfile = siege.log
protocol = HTTP/1.1
chunked = true
cache = false
connection = close
concurrent = 25
time = 60M
file = access-urls-1.log
delay = 3
timeout = 30
internet = false
user-agent = Siege benchmark test
accept-encoding = gzip
spinner = true
login-url = https://<site>/user POST name=admin&pass=12345&form_id=user_login&op=Log+in

Running Siege with the configuration file.

~$ siege -R siege_login_rc

Get Apache style log from Varnish

You can get logging information from Varnish in the same format as Apache uses by running this command on your Varnish server. Use ctrl+c to stop logging to the file.

~$ varnishncsa -w varnish.log

Adjustments parameters

This is a list of things that may improve performance, but be aware that it may also have the opposite effect and decrease performance. Remember to run Siege before adjusting any of you configuration parameters, so you have a reference point for your performance improvements. You should also run Siege under the same conditions and with the same parameters every time to make the results comparable.

• MySQL innodb (buffers)
• Apache threads
• MySQL engine
• Disk performance
• APC configuration
• PHP configuration
• Reverse caching (Varnish)