The performance test of 6 leading frameworks

The performance test of 6 leading frameworks

As I already wrote in the previous posts, several months ago I chose framework for make a site. Most important for me were simple using of him, stability and performance.
For absence of the concrete developed information about speed comparison of frameworks in the Internet, (I have found only one testing, but it turned out naked enough. It was limited to three frameworks RubyOnRails, Django, Symfony and didn’t contain absolutely any detail), I conducted the detail analysis for decision of the 6 best frameworks in all these parameters.

src="http://digg.com/api/diggthis.php?u=http://www.digg.com/programming/The_permance_test_of_6_leading_web_frameworks" frameborder="0" width="55" scrolling="no" height="82">

<script src="http://media.alrond.com/prototype.js" type="text/javascript"></script> <script src="http://media.alrond.com/scriptaculous.js?load=effects" type="text/javascript"></script> <script src="http://media.alrond.com/effects.js" type="text/javascript"></script> <script src="http://media.alrond.com/lightbox.js" type="text/javascript"></script> So, the purpose of testing was the decision of work speed of frameworks and their comparisons with each other on speed of pages generating and on maximum quantity of inquiries by the given configuration.
I have chosen the test model, at which the controller rendered the appointed template and generated the answer. As a result turned out the HTML-file "Hello World!" (below there are codes for everyone framework). The Database in the test is not used, because it itself limits the speed important.

I did not plan to compare functionality of frameworks, community and work with database. It was important to me to determine work speed of MVC-framework, by as small as possible influence of external factors.
Therefore I used sheaf Nginx + FCGI through IP:Port

The following frameworks have been tested:

• CodeIgniter on PHP
• Catalyst on Perl
• Django on Python
• Django on Python with Psyco
• RubyOnRails on Ruby
• Symfony on PHP
• TurboGears on Python

Conducted the test on following hardware and software:

CPU: AMD OpteronT Processor 146 (2 GHz)
Memory: 2 GB
OS: Debian 3.1 (Linux 2.6.14)
Web-Server: nginx/0.5.5

Versions of frameworks and programming languages:

• CodeIgniter 1.5.1
• Catalyst 5.7006 Rev.5996
• Django Rev.4254 (28Dec.2006)
• RoR 1.1.6
• RoR 1.2.1
• Symfony 1.0beta2 SVN-Rev 3122
• Turbogears 1.0b3 Rev.2323
  
• Python 2.4.4
• Python Psyco 1.5.2
• Flup Rev.2303
• Ruby 1.8.5-p12
• mongrel 0.3.13.4
• PHP 5.2.0 (cgi-fcgi)
• perl, v5.8.4
• CPAN ver 1.8802

Programs for test:

Siege 2.65 http://www.joedog.org/JoeDog/Siege
Http_load 12.03.2006 http://www.acme.com/software/http_load/
ab 2.0.41-dev Rev: 1.141

Technique of testing:

• Measurement of memory (ps aux: VSZ "virtual set size" and RSS "resident set size").
• Testing Apache Benchmark (2 times in succession)

  ab -c 5 -n 1000 http://project.com/

• Measurement of memory and of used processor-time (ps aux: VSZ, RSS, %CPU, %MEM).
• Restart of Framework.
• Testing Apache Benchmark (2 times in succession)

  ab -c 100 -n 10000 http://project.com/

• Measurement of memory and of used processor-time.
• Restart of Framework.
• Measurement with http_load

  http_load -rate 10 -seconds 5 project.com

• Measurement of memory and of used processor-time.
• Restart of Framework.
• Test Siege with 50 concurrent users during one minute.

  siege -d1 -t1M -c50 project.com

• Test Siege with 200 concurrent users during one minute.

  siege -d1 -t1M -c200 project.com

• Test Siege with 300 concurrent users during one minute.

  siege -d1 -t1M -c300 project.com

• Measurement of memory and of used processor-time.

Comments to a technique:

For the beginning I have used only Nginx with the elementary config, at which on any inquiry the dot transparent gif-file in the 43b size comes, generated by a server.
This test determined an approximate maximal capacity of the server. In front I shall tell that a stock of its capacity is 10 times more then production capacity of the fastest framework.
All frameworks were started as FastCGI 127.0.0.1:PORT. If framework specified development- and production- modes, I worked with production.
In nginx the identical config is used for all frameworks.
New start of framework I conducted to clear statistics among tests different programs.

My subjective additions on every framework by installation and start:

1. CodeIgniter
   It is easy and quick to organize it. I haven’t any problems with it. For start I used spawn-php.sh with five processes.
   There is pretty bright framework for PHP.
2. Catalyst
   For the beginning it is necessary to install CPAN and a couple of modules for FCGI. Installation looked a little involved.
   Start isn’t simple too. For the beginning it is necessary to start Framework

   ./CatInABox/start.sh

   Then the project itself

   ./script/world_fastcgi.pl -l 127.0.0.1:9500 -n 5

3. Django
   It is easy to install from repository. Projects are created easy too. I started with two methods prefork and threaded. But the python doesn’t work so well with treads.
   

   python manage.py runfcgi method=threaded host=127.0.0.1 port=8801
   python manage.py runfcgi method=prefork host=127.0.0.1 port=8801

   I also tested the framework with the acceleration module of psyco. In manage.py put:

   from django.core.management import execute_manager
   import psyco
   psyco.full()

4. RubyOnRails
   Following the instruction from a site it is easy to install, but it turned out difficult to start it. Though on the Internet I saw different configurations for sheaf Nginx + FastCGI, I failed to start it together with nginx.
   However with lighttpd it all started normally. Therefore I have taken advantage of the "official" recommendation to start it through a server mongrel.
   Owing to Alexey Kovyrin it is possible to estimate a multiplication factor.
   Nginx+FastCGI is faster approximately in 1.29 times than Nginx + Mongrel.
   I started 5 servers mongrel:

   mongrel_rails start -d -e production --port 8501 --pid /tmp/rb1.pid
   mongrel_rails start -d -e production --port 8502 --pid /tmp/rb2.pid
   mongrel_rails start -d -e production --port 8503 --pid /tmp/rb3.pid
   mongrel_rails start -d -e production --port 8504 --pid /tmp/rb4.pid
   mongrel_rails start -d -e production --port 8505 --pid /tmp/rb5.pid
   

   In a few days after test realization a new version RoR 1.2.1 came out.
   Of course I has also tested it with all parameters, has only renewed framework, checked all specifications. Results are shocking a little: factor productivity halved in comparison with previous version in all tests!
5. Symfony
   It wasn’t complicated to install it, but then was worn out with the project itself. There is difficult enough. For start I also used spawn-php.sh with five processes.
6. TurboGears
   By installation were appearing some problems, which were solved with some efforts.
   For start in threaded mode I used a script, I only changed

   from fcgi import WSGIServer
   	to
   from flup.server.fcgi  import WSGIServer
   
   WSGIServer(application=wsgiApp).run()                                                                                                                                                               
   	to
   WSGIServer(application=wsgiApp, bindAddress=
   
   	('127.0.0.1', 8900)).run()

Configuration files of web-server:

• „чистый“ nginx
• nginx + fcgi
• nginx + mongrel

Project codes for every framework:

• CodeIgniter
• Catalyst
• Django
• RubyOnRails
• Symfony
• TurboGears

Start script of PHP-FCGI:

• spawn-php.sh

Web-server results for comparison:

• Nginx

Results for frameworks each taken separately:

• CodeIgniter
• Catalyst
• Django prefork
• Django threaded
• Django prefork with Psyco
• Django threaded with Psyco
• RubyOnRails 1.1.6
• RubyOnRails 1.2.1
• Symfony
• TurboGears
• TurboGears with Psyco (error is taken out)

Results:

1. Test Apache Benchmark
   • Table of inquiry working speed
     
     * 1) in second time framework is down
     * 2) a multiplication factor according to Kovyrin = 1.29. For conversion of performance from mongrel into fastcgi.
     
   • Diagram of results. Test „ab c 5 n 1000“.
     
   • Diagram of results. Test „ab c 100 n 10000“.
     
   • Memory expense before and after tests
     * 3) The python is made so, that it restarts the processes at strong load in prefork-mode. Therefore it is impossible to determine real load of processor and maximal use of memory.
     
   • VSZ "virtual set size"
     
   • RSS "resident set size"
     
   • CPU loads
     
2. Test http_load
   • Table with results
     
   • Diagram of time for connect
     
   • Diagram with time of first-response
     
3. Siege test
   • General table with results
   • Transaction diagram
   • Response time
   • Longest transaction
   • Memory expense after tests
   • Processor load

Conclusions:

Results are visible according to tables and diagrams.
Some short additions:
Django has the least use of the processor.
Catalyst has surprised. By excessive test it overloads the processor. But losses of inquiries were not observed.
RoR 1.2.1 also loads the processor under big load.
It was interesting by TurboGears, which has shown low time use CPU in test "ab", but in the siege-test had the worst result.
Catalyst has appeared the greatest memory use. By RoR this more likely happens because of start through mongrel.
PHP-frameworks borrow many resources CPU even in a quiet condition.

Average connect time is approximate equal for all frameworks. The first response time differs strongly from each other. Django is exception, having the least “average time”, and the highest “maximal time”. Python-frameworks have proved good oneself, and RoR has disappointed.

During the siege-test by CodeIgniter, Symfony, RoR 1.2.1 and 1.1.6 losses were observed at a large number of "concurrent users".
RoR is fast by the small load, but it sharply loses productivity at a plenty of users.

Psyco module accelerates Django on 15-30 %, but it will be worth the grown memory use VSZ on 80 % in a prefork-mode and on 400 % (!!!) in threaded-mode. Use of RSS memory increases in 2-2.5 times.
The Prefork-mode takes away more memory, but for it the system comes out, which stably works on extreme loads, and the smaller use of the processor resources.
In a threaded-mode Django hanged under the big load and did not answer on inquiries.
The same happened with TurboGears too. This is due to the fact that python works badly in a treaded-mode.

Comments from Ivan Sagalaev:

"- FastCGI - server is started in a threaded-mode. It is known, that the Python works in treads essentially slow through GIL (Global Interpreter Lock), because of which all treads wait each other: the interpreter can be used of only one of these. Therefore in Unix at the opportunity servers with python-code should be started in a prefork-mode – it will be faster. By the way, in Windows this situation is reverse, because there creation of process is essentially slower than in Unix. And as far as I know, Python however hangs there less.
- About Psyco. Django-developers have tested it, and have found something interesting. On 32-bit systems it really gives an increase in productivity. And on 64 bit - on the contrary productivity sinks. It happens, because the processor switches under Psyco to a 32-bit mode and does not use all advantages of one’s own architecture. If it is so, than this fact, that Django worked more quickly under Psyco, very likely means, that Linux and Python don’t use 64 bits system (in spite of work on Overtone). That’s right?”

Yes, it is 32 bit system.

Distribution of places according to this test:

1. Django has won over the nearest competitors with the approximate triple superiority.
2. Second and third places have divided TurboGears and RoR 1.1.6, because they are equally fast, but behave differently at different loads, overtaking each other.
3. .
4. Catalyst. To be honest, I expected a good deal more from Perl-framework.
5. CodeIgniter. Just as expected PHP-frameworks have appeared the slowest. But CodeIgniter is to recommend those who wish to program only on PHP, and also to have handy system.
6. Results of RoR 1.2.1 shocked “a little”: falling of productivity in 2-4 times in comparison with 1.1.6 version. The response-time in http_load is also more in 2 times and excessively high CPU-load under high stream - all this attests to possible error in the new version.
7. Symfony has got last place. It is very complicated and slow framework. A difference with Django is up to 35 (!!!) times.

P.S. Because of the addition to Django of the module Psyco in test I have conducted at many requests a short test for PHP with eAccelerator-0.9.5.
I have used 2 times in succession the following commands:

ab -c 5 -n 1000 http://ci.test.com/
ab -c 100 -n 10000 http://ci.test.com/

As a result CodeIgniter works 6,5 times faster, gives out to 600 resquests / sek.
The difference with Django became to 2 times.