Ir al contenido principal

Simple server load test with cron and ab (Linux)

Load testing "refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as web servers."

I've found many articles online explaining how ApacheBench lets you "load test" with a single command from a Linux terminal, but is this a realistic load test? A single execution of ab is a very limited simulation of what actually happens when multiple users try to access your web application. A server may perform well if it has to work hard for a 30 seconds (possible execution time of an ab command), but what happens when 20000 extra requests hit your web app after it's already been stressed for hours?


Apache HTTP server benchmarking tool (ApacheBench) is a simple yet great tool which was "designed to give you an impression of how your current Apache installation performs." It can be leveraged to load test any web server setup, but we need to think for a minute about what exactly we're simulating. Here are a few examples:

  1. An average of 1000 request per minute by 30 different users reach a web server, with spikes of up to 5000 request by 100 users every hour or so.
  2. We expect 15000 requests every five min (by 50-100 different users), which doubles from 7 to 10pm on weekdays.
  3. Up to 10 other systems access my REST API, with 500,000 up to a million requests per hour each.

This is where cron comes into play. Cron is a time-based job scheduler in Linux, this means you can use it to program commands to execute at specific times in the background, including recurrent runs of the same command (for example on minute 15 of every hour). Like ab, it's a pretty simple tool which you can access with the crontab -e command in Linux, which opens your preferred editor (typically nano) for you to enter single-line 6-field expressions in the CRON format – which may vary slightly among Linux distributions: m h dom mon dow command (minutes, hours, day of month, month, day of week, command). Going back to the 3 examples:

  1. We need 2 entries in crontab:
    * * * * * ab -k -c 30 -n 1000 http://server-hostname/ # every minute
    0 * * * * ab -k -c 70 -n 4000 http://server-hostname/ # every hour
  2. Now we may need 3 entries:
    /5 0-19 * * * ab -k -c `shuf -i 50-100 -n 1` -n 1000 http://webapp-hostname/ # every 5 min in the initial normal hours (12am to 7pm)
    /5 19-22 * * * ab -c `shuf -i 50-100 -n 1` -n 4000 http://webapp-hostname/path/ # every 5 min in "rush hours" (7-10pm)
    /5 22,23 * * * ab -k -c `shuf -i 50-100 -n 1` -n 1000 http://webapp-hostname/path/ # every 5 min in the remaining normal hours (10pm and 22pm)
  3. A single entry will do here:
    30 * * * * ab -c 10 -n `shuf -i 50000-1000000 -n 1` http://api-hostname/get?query # every hour (on minute :30)
Notes:
I use -k in some of the ab commands to web applications, this uses HTTP keep-alive and is meant to simulate returning individual users.
The shuf on Linux command generates a random number within a given range (-i) and increment (-n).

These are simple use cases which begin to approximate complete load tests but which do not take into account certain factors such as multiple URL paths or POST requests. Also, in order to see the output of the ab commands executed by cron, we need to add log files to the mix. I'll leave that for you to figure out but here's a tip, on example #3:

30 * * * * ab -c 10 -n `shuf -i 50000-1000000 -n 1` http://api-hostname/get?query >> /home/myuser/my/api/tests/load/cronab.log

ab's output is a report that looks like this:

Concurrency Level:      35
Time taken for tests:   38.304 seconds
Complete requests:      220000
Failed requests:        0
Keep-Alive requests:    217820
Total transferred:      70609100 bytes
HTML transferred:       18480000 bytes
Requests per second:    5743.58 [#/sec] (mean)
Time per request:       6.094 [ms] (mean)
Time per request:       0.174 [ms] (mean, across all concurrent requests)
Transfer rate:          1800.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       5
Processing:     0    6   1.4      6      25
Waiting:        0    6   1.4      6      25
Total:          0    6   1.4      6      25

...

Final tip: How to determine infrastructural limits?

Every server's capacity is different. While reports from trial-and-error executions of ab can give you an idea of where a web application's infrastructure (servers) start to falter (mean response times go up exponentially), the very best way is by having a visual APM such as Amazon CloudWatch, in AWS. Monitoring graphs of different metrics over time –e.g. requests handled, errors, dropped connections, CPU utilization, memory, or swap usage– once we have left ab run on cron for hours or even days lets you better adjust the number of requests and concurrency for future ab commands. Try to find that breaking point!

Thanks for reading (=

Comentarios

Entradas populares de este blog

sqlalchemy ProgrammingError can't adapt type numpy.float64 - Python troubleshooting

Are you getting a

sqlalchemy.exc.ProgrammingError: (ProgrammingError) can't adapt type 'numpy.float64'
???

It's because psycopg (psycopg2) - which is used inherently by SQL Alchemy but if you're just dealing directly with alchemy you might not know this - has no idea how to make the numpy.core.numerictypes.float64 (complete type path: another thing they don't tell you) a string for SQL querying.

Simple solution, add:

# solution by Jorge Orpinel found at rehalcon.blogspotcom
import numpy
from psycopg2.extensions import register_adapter, AsIs
def addapt_numpy_float64(numpy_float64):
return AsIs(numpy_float64)
register_adapter(numpy.float64, addapt_numpy_float64)
somewhere in the program.

For a more complete reference of the problem see http://initd.org/psycopg/docs/advanced.html#adapting-new-python-types-to-sql-syntax

Dockerize your Django App (for local development on macOS)

I'm writing this guide on how to containerize an existing (simple) Python Django App into Docker for local development since this is how I learned to to develop with Docker, seeing that the existing django images and guides seem to focus on new projects.

For more complete (production-level) stack guides you can refer to Real Python's Django Development with Docker Compose and Machine or transposedmessenger's Deploying Cookiecutter-Django with Docker-Compose.

Pre-requisitesAn existing Django app which you can run locally (directly or in Virtualenv). We will run the local dev server with manage.py runserver.A requirements.txt file with the app dependencies, as is standard for Python projects; including MySQL-python.Working local MySQL server and existing database. (This guide could easily be adapted for other SQL engines such as Postgres.)Install Docker. You can see Docker as a virtual machine running Linux on top of your OS ("the host"), which in turn can run con…

Mapa del Metrobus de la Ciudad de México

Update: Por fin el Gobierno hizo su chamba luego de años de que mi mapa fuera el único en Google Maps. Hoy (05-11-2013) me percaté que en www.metrobus.df.gob.mx/mapa.html (solo con "www"...) se puede apreciar ya este mapa (zoom centrado en centro de la ciudad):

Ver Plano de Sistema de Corredores de Transporte Público de Pasajeros del D.F., Metrobús en un mapa ampliado


Para quien prefiera los iconos que yo utilicé, aquí dejo la última versión que actualicé (hasta línea 4 y proyecto de línea 5): En mi post "Mapa del Metrobus en Google Maps" publiqué el mapa del metrobus en Abril dl 2009. Ahora que hay más estaciones y más líneas, es necsaria esta actualización.
Incluye íneas 1 a 4 (actual a Junio 13 del 2012). La descripción de cada estación incluye correspondencias con otras líneas de MB y con el meto.
Además quiciera es incluir el ten ligero, las rutas/paradas del turibus, y algunas rutas de RTP - y tal vez el ten suburbano.
* En la vista satelital aun no aparecen m…