What can a threaded web server do for a memory hungry application ?

You are welcome to extract excerpt of this article, but do not copy it entirely on your blog/web site.
You are only allowed to use this post content in an printed article or a newsletter by clearly exposing the source URL of this article.

First of all, Hello to all my readers.
It's been some time I haven't updated my blog, but I have not been lazy.

I've been polishing my web page screenshots service and I was tweaking my servers to extract as much performances as I could. My script goes well by itself, the database performs good too, but one thing was bugging me down: the memory usage...

A bit of self promotion here:
This web shooting service I'm talking about is located at http://www.web-screenshots.com and I'm currently looking for beta testers, as there is nothing like a real stress test.
If you are interested to participate to this beta-test (access for all the functions of the service for free), please send me an email at thierry@webalis.com

Memory, you never have too much...

As my service will use the Gecko library to render the pages to be screenshoted, it require a nifty 26M of resident memory (17M of shared though). In itself, it's not that much, but when you line up 10 of those, plus the Apache processes (30M of resident with the PHP module and the others) it can sum quite fast.
Add to this the Xfree server back-end to render the page themselves, and it's a lot that's needed for just 1 screenshot.
Now, I do have 4Gb of ram available, but it's not because I have plenty of them that I should waste it, ain't it ?

Where can I gain some ?

The first obvious answer that popped in my mind was: an the Apache web server!
I use Apache with PHP only as a front-end for my service. It get the request, check if it's valid, and send it to a python application I wrote which will grab the page, screenshot it, try to optimize the rendering, and then gives the hand back to PHP to send the screenshot back to the requester.

In his default configuration, Apache use a "pre-fork" Mpm.
Mpm stands for "Multi-Processing Module". His work is to receive every requests, and answer them.

The pre-fork model is "per processes" based, meaning that each request will span a new Apache process if one already existing is not available.
On my setup, each processes would eat around 30Mb of ram.

Not a big deal, would you say !?
I don't agree... Mind you, a 1x1 PX image is a request, a css file is a request, an external js file is a request; and they all use 30Mb of ram just to transfer a bit of text...
Without realizing it, you can flood you server if you (or your web host) are not attentive to this.

First solution: a static content server

It's the solution the most frequently used, and the most frequently used server for that is lighttpd.
It's a fast web server with a very limited footprint.

I only have 2 problems with it:

1. No virtual hosts support
2. No PHP as a module.

For the first one, it could be avoided easily, by setting up a virtual interface and asking for a second IP to my data center, but still, I found it annoying.

The second point, on the other side, is more annoying.
PHP can only be embed via FastCGI.
Having PHP as CGI would mean that the request to be processed by PHP would have to be directed to a /cgi-bin directory, which I simply cannot afford to do now.
Almost all of the pages of the site are 100% pure PHP classes, and smarty is used to the HTML output. This kind of change is too drastic to be implemented now, it would mean rewriting the front-end.

But....
Reading about FastCgi and PHP performances, I stumbled on an interesting note:

Instead of using operating system environment variables and pipes, the FastCGI protocol multiplexes the environment information, standard input, output, and error over a single full-duplex connection. This allows FastCGI programs to run on remote machines, using TCP connections between the Web server and the FastCGI application.

What tickled me here was that FastCGI is "multiplexing"...
It means, in a simplified form, that it will instantiate one time an application, and reuse the same for multiple requests.

What it means too, in my case, is that this application (the PHP engine, in this case) would not be embedded in each Apache process, but limited to 10/15 occurrences outside the Apache process, thus diminishing the memory load.

But the main problem stayed. As it would force me to run PHP as CGI, it would require to rewrite the front-end. But it gave me an alternate road to follow; what if I could do that with Apache directly?

Apache MPM modules: the Worker

As you know now, the pre-fork MPM means that each request create a new fully featured Apache process.
On the other hand, the worker module is a threaded MPM. At the opposite of the pre-fork model, the worker model launch only a few process, and then will use Threads into those process to answer to the requests.

From the worker documentation page:

This Multi-Processing Module implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with less system resources than a process-based server. Yet it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.
...
A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread which listens for connections and passes them to a server thread for processing when they arrive.

The big difference to catch is that a thread is not a process, a thread is inside a process.
This means that several request will all use the same process, but this process will "multiplex" itself with threads to answer each requests.

This is great, exactly what I was looking for !
But, as always, there is a drawback: your web server is threaded, which means that modules (be they PHP or Apache) who are not "thread safe" might cause havoc on your system...
In fact, even the PHP guys are advising you not to do this:
http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2

PHP is glue. It is the glue used to build cool web applications by sticking dozens of 3rd-party libraries together and making it all appear as one coherent entity through an intuitive and easy to learn language interface. The flexibility and power of PHP relies on the stability and robustness of the underlying platform. It needs a working OS, a working web server and working 3rd-party libraries to glue together. When any of these stop working PHP needs ways to identify the problems and fix them quickly. When you make the underlying framework more complex by not having completely separate execution threads, completely separate memory segments and a strong sandbox for each request to play in, feet of clay are introduced into PHP's system.

If you feel you have to use a threaded MPM, look at a FastCGI configuration where PHP is running in its own memory space.

And finally, this warning against using a threaded MPM is not as strong for Windows systems because most libraries on that platform tend to be threadsafe.

I felt fairly confident that I would not run into problems, as I don't use external libraries, and well, I just had to try.

What this means too is that you have to choose your Apache and PHP modules with caution, and for many of them, you might need to recompile them with the correct options to enable the thread safe support.

I operate a server running Gentoo Linux, and in my case I had to rebuild Apache, mod_php, sqlite and postgresql. This was simply carried with 1 command, but different Linux distributions might be more problematic.

Apache and PHP are installed with threaded support, and what are the results?

Well, I have to admit that I'm impressed with them. Being mostly idle now, the server have just 2 Apache process. 1 being the master weight 48M in resident, and the other one is 28Mo.

During my self conducted load tests, I used a simple Jmeter web test plan with both requests to cached and not cached web pages.
During the test, I had 21 concurrent requests running, 6 of them being for not cached content and 15 for cached content.

Basically, the cached request went to the speed of 1.7 request per second, for the myspace home page on a jpeg, 250x187 px with it's quality set to 75%.
The "live" capture was done on the Google.com home page, with the same picture settings, and reached a darn good 59.5 requests per minutes. That's almost 1 screenshot per seconds without using the caching mechanism !
During that test, I had 3 Apache process, totalizing less than 100Mo of ram on my server and my server load reached 3.5 and stabilized after that with intensive network I/O and db access, but nothing too scary.

As the screenshots are fetched live, I was thinking that I could end up with a lot of "bad" screenshots, as the request would pile up, but it seems that it stay relatively low.
My last test ran for 2502 requests (1002 not cached, and 1500 who where cached).
I had a 0.07% of error on the cached myspace page, and a 1.8% on the live Google web page, which makes a global error rate of 0.76% for 2500 requests.

Considering that most of the request should be done for cached screenshots, I'm pretty satisfied of those results. The errors where all that the screenshots had a 0 byte size, and was discarded. As those are automatically rejected from the cache system, it does looks fine to me.

I was not expecting so much out of the box !!!

Links:

• http://doughboy.wordpress.com/2008/02/13/apache-worker-and-php/ Brian Moon explain how switching to a Worker based MPM helped to run dealnews.com smoothly.
  
• http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2 What the PHP manual has to say about multi threaded support.

---

Did you enjoy this post? Why not leave a comment below and continue the conversation, or subscribe to my feed and get articles like this delivered automatically each day to your feed reader.

A License To K{ode|ill} | Powered by WordPress | ElegantBlue is a custom theme by Nathan Rice 31765 pages viewed, 24 today
20878 visits, 22 today
 Powered by FireStats