HTTP Blog

There are quite a few web server software daemons that can be used for almost any content hosting, but for high performance and fast static content hosting the choice is not that big.

Zeus (commercial)
Lighttpd (free)
Nginx (free)
Thttpd (free)
Cherokee (free)

We have tested quite a few http server daemons listed above (except Zeus). If you are looking for basic hosting, here is some great Linux web hosts.

My top server software would be:

Nginx and Cherokee. Easy to use, easy configuration, lightweight and very reliable. I do not pay for the software I can get for free, sorry folks that develop Zeus

OK, if you want to redirect user to a different page or even a site, you can easily do it using mod_rewrite rule set:

RewriteCond %{HTTP_USER_AGENT} “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)” [NC]
RewriteRule ^(.*) http://yoursite.com/page.html [R=301,L]

Will redirect all users with Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) to http://yoursite.com/page.html

If your web site has been slammed with multiple downloads from one IP causing the web site to slow load due to CPU, memory or network stack, it’s time for an action.

For apache users there is an easy workaround - you need mod_limitipconn module and you are set.

Apache 1.3 and version 2 has a bit different code, thus you will need to download the correct mod_limitipconn module depending on the apache version you are running.

Install this module as usual (DSO or static compile) and setup a Location definition in httpd.conf file.

MaxConnPerIP 3

Will limit 3 connections per IP. All other connections above the previous definition (3) will simply be dropped….

Enjoy !

thttpd providers excellent and easy to set-up bandwidth throttling based on files, directories, file types or default web server bandwidth speed.

If you want to cap the default bandwidth of your thttpd server to 512Kbps you will need to add the following configuration line to your thttpd config file:

** 51200If exe download files are consuming too much of your upstream, consider limiting download speed for all executable files via:

**.exe 102400

The above configuration line limits all .exe file downloads to max of 1Mbps per second at any time.

You can also limit bandwidth speed on some specific directory, for example, if you are providing a trial file downloads and want to offer better speed for your paid members on the same server. Create a folder named /trial/ and set-up thttpd config file to include

trial/**  25600

The above setting will limit all files located under /trial/  account to have a maximum download speed of 256Kbps.

At this time, there are quite a few modules available that will stop traffic leech and limit simultaneous connections, bandwidth based on files, virtual host and so on.

mod_bw
Apache 2 (also available for Windows platform):

http://www.ivn.cl/apache/

mod_cband (Apache2):

http://cband.linux.pl/

mod_bandwidth

http://www.cohprog.com/v3/bandwidth/doc-en.html#

mod_security is a powerful attack shield that can block http requests based on pre-defined rules. It’s a nice shield against known software exploits (e.g. non-patched forums, scripts etc)…

A non-aggresive rule set:
# Turn the filtering engine On or Off
SecFilterEngine On

# Change Server: string
SecServerSignature “Apache”
# This setting should be set to On only if the Web site is
# using the Unicode encoding. Otherwise it may interfere with
# the normal Web site operation.
SecFilterCheckUnicodeEncoding Off

# The audit engine works independently and
# can be turned On of Off on the per-server or
# on the per-directory basis. “On” will log everything,
# “DynamicOrRelevant” will log dynamic requests or violations,
# and “RelevantOnly” will only log policy violations
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog logs/audit_log

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# Action to take by default
SecFilterDefaultAction “deny,log,status:403″

## ## ## ## ## ## ## ## ## ##
## ## ## ## ## ## ## ## ## ##

# Require HTTP_USER_AGENT and HTTP_HOST in all requests
# SecFilterSelective “HTTP_USER_AGENT|HTTP_HOST” “^$”

# Require Content-Length to be provided with
# every POST request
SecFilterSelective REQUEST_METHOD “^POST$” chain
SecFilterSelective HTTP_Content-Length “^$”

# Don’t accept transfer encodings we know we don’t handle
# (and you don’t need it anyway)
SecFilterSelective HTTP_Transfer-Encoding “!^$”

# Protecting from XSS attacks through the PHP session cookie
SecFilterSelective ARG_PHPSESSID “!^[0-9a-z]*$”
SecFilterSelective COOKIE_PHPSESSID “!^[0-9a-z]*$”

SecFilter “viewtopic\.php\?” chain
SecFilter “chr\(([0-9]{1,3})\)” “deny,log”

# Block various methods of downloading files to a server
SecFilterSelective THE_REQUEST “wget ”
SecFilterSelective THE_REQUEST “lynx ”
SecFilterSelective THE_REQUEST “scp ”
SecFilterSelective THE_REQUEST “ftp ”
SecFilterSelective THE_REQUEST “cvs ”
SecFilterSelective THE_REQUEST “rcp ”
SecFilterSelective THE_REQUEST “curl ”
SecFilterSelective THE_REQUEST “telnet ”
SecFilterSelective THE_REQUEST “ssh ”
SecFilterSelective THE_REQUEST “echo ”
SecFilterSelective THE_REQUEST “links -dump ”
SecFilterSelective THE_REQUEST “links -dump-charset ”
SecFilterSelective THE_REQUEST “links -dump-width ”
SecFilterSelective THE_REQUEST “links http:// ”
SecFilterSelective THE_REQUEST “links ftp:// ”
SecFilterSelective THE_REQUEST “links -source ”
SecFilterSelective THE_REQUEST “mkdir ”
SecFilterSelective THE_REQUEST “cd /tmp ”
SecFilterSelective THE_REQUEST “cd /var/tmp ”
SecFilterSelective THE_REQUEST “cd /etc/httpd/proxy ”
SecFilterSelective THE_REQUEST “/config.php?v=1&DIR ”
SecFilterSelective THE_REQUEST “&highlight=%2527%252E ”
SecFilterSelective THE_REQUEST “changedir=%2Ftmp%2F.php ”
SecFilterSelective THE_REQUEST “arta\.zip ”
SecFilterSelective THE_REQUEST “cmd=cd\x20/var ”
SecFilterSelective THE_REQUEST “HCL_path=http ”
SecFilterSelective THE_REQUEST “clamav-partial ”
SecFilterSelective THE_REQUEST “vi\.recover ”
SecFilterSelective THE_REQUEST “netenberg ”
SecFilterSelective THE_REQUEST “psybnc ”
SecFilterSelective THE_REQUEST “fantastico_de_luxe ”

SecFilter “bcc:”
SecFilter “bcc\x3a”
SecFilter “cc:”
SecFilter “cc\x3a”
SecFilter “bcc:|Bcc:|BCC:” chain
SecFilter “[A-Z0-9._%-]+@[A-Z0-9._%-]+\.[A-Z]{2,4}\,\x20[A-Z0-9._%-]+@[A-Z0-9._%-]+\.[A-Z]{2,4}”
SecFilterSelective POST_PAYLOAD “Bcc:”
SecFilterSelective POST_PAYLOAD “Bcc:\x20″
SecFilterSelective POST_PAYLOAD “cc:”
SecFilterSelective POST_PAYLOAD “cc:\x20″
SecFilterSelective POST_PAYLOAD “bcc:”
SecFilterSelective POST_PAYLOAD “bcc:\x20″
SecFilterSelective POST_PAYLOAD “bcc: ”
SecFilterSelective THE_REQUEST “Bcc:”
SecFilterSelective THE_REQUEST “Bcc:\x20″
SecFilterSelective THE_REQUEST “cc:”
SecFilterSelective THE_REQUEST “cc:\x20″
SecFilterSelective THE_REQUEST “bcc:”
SecFilterSelective THE_REQUEST “bcc:\x20″
SecFilterSelective THE_REQUEST “bcc: ”
# WEB-PHP phpbb quick-reply.php arbitrary command attempt
SecFilterSelective THE_REQUEST “/quick-reply\.php” chain
SecFilter “phpbb_root_path=”

If you are getting attacked or simply burning too much traffic (your upstream provider says so)… it’s perhaps the best time to check out  how much of traffic are you really bursting?

There is a software (freeware of course) that perfectly counts/measures your server traffic usage so go ahead and use it.

http://dast.nlanr.net/Projects/Iperf/

We will soon perform some benchmarking with thttpd and large file serving, but I am confident that it will be one of the best.

Why? Because it’s FAST and I mean it…:

• it doesn’t fork
• perfect memory management
• small runtime
• implements http 1.1 protocol with minimum requirements
• secure and robust

You can easily check a web server header using a simple telnet utility.

telnet www.yahoo.com 80
HEAD / HTTP/1.0

and hit Enter two times.

You will get a header response with the HTTP status code….

HTTP/1.1 200 OK
Date: Mon, 31 Jul 2006 05:47:22 GMT
P3P: policyref=”http://p3p.yahoo.com/w3c/p3p.xml”, CP=”CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV”
Cache-Control: private
Vary: User-Agent
Set-Cookie: FPB=7cum1i2o812cr6ba; expires=Thu, 01 Jun 2006 19:00:00 GMT; path=/; domain=www.yahoo.com
Connection: close
Content-Type: text/html

Connection closed by foreign host.

Ok here is the deal if you do not want to switch to other web server software (e.g. Cherokee, thttpd, roxen, boa, lighttpd) and want the best performance for serving static files that do not often change the best solution is to use mod_mmap_static module. This module is not compiled in by default and you will need to recompile apache to enable it:

–enable-module=mmap_static

Don’t forget if any of your web files change - you will need to restart Apache web server…. it’s up to you now - use it or not

In httpd.conf there is a directive MaxClients that is usually defined ~ 150. This value allows to serve more simultaneous requests and process queue more faster.

Please note that you will need to recompile apache httpd (you must edit the HARD_SERVER_LIMIT entry in httpd.h and recompile) in order to set it higher that 256. Remember that each daemon requires more memory and an average figure for each process is 4…8MB. For example, you can set-up MaxClients value to ~250 if you have a 1GB of RAM.

Enjoy!

For best performance you need to strip down your apache binary - the less modules, the less memory used the better speed / performance you will get. If you are loading modules via DSO you can easily remove module from the loading list by commenting out the LoadModule option. If you have a module linked statically you will need to recompile apache http daemon.
By default 1.3 has compiled in the following modules:

Compiled-in modules:
http_core.c
mod_env.c
mod_log_config.c
mod_mime.c
mod_negotiation.c
mod_status.c
mod_include.c
mod_autoindex.c
mod_dir.c
mod_cgi.c
mod_asis.c
mod_imap.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_access.c
mod_auth.c
mod_setenvif.c

you will need to compile apache and remove modules that are not needed, for example: mod_auth.c mod_imap.c using the following syntax:

./configure –disable-module=auth –disable-module=imap

I suggest you disable everything, except

mod_mime
mod_dir
mod_log_config
You can of course remove log config module if no logging is needed. If your site requires any specific modules or the ones listed above - you will need to enable them at the compilation time. It’s try and see option, for those, who are not 100% sure what modules are really needed.

Apache 2.0.* has compiled in the following modules by default:

Compiled in modules:
core.c
mod_access.c
mod_auth.c
mod_include.c
mod_log_config.c
mod_env.c
mod_setenvif.c
prefork.c
http_core.c
mod_mime.c
mod_status.c
mod_autoindex.c
mod_asis.c
mod_cgi.c
mod_negotiation.c
mod_dir.c
mod_imap.c
mod_actions.c
mod_userdir.c
mod_alias.c
mod_so.c

The most critical ones are again,  mod_mime;mod_dir; less mod_log_config.

Please note than DSO support is activated/enabled by default and you can turn it off by removing this module at the compilation time.

Enjoy!

If you have mod_status support compiled in your apache (either statically or using DSO) and activated in httpd.conf file with ExtendedStatus On you are probably affecting your site and server performance because every request to apache (e.g. web site hit) will generate multiple queries to gettimeofday or times function/system call depending on your Operating System.Make sure it’s turned off for the best performance:

ExtendedStatus off

in your httpd.conf file

The option MaxKeepAliveRequests specifies the number of requests allowed per connection when the KeepAlive on has been set. When the value of this option is set to 0 then unlimited requests are allowed on the server.

For better server performance, it’s recommended to allow unlimited requests or you can always define it with a high value, for example, 10,000.

Boosting apache performance in high load environments.

I suggest decreasing timeout value and keepalivetimeout, as well as other values listed and described below.

Original timeout has been set to 300 by default. I suggest decreasing it to 120 (2 minutes) so all connections will timeout after 2 minutes. period.

Timeout 120
By default, keepalive is turned on for apache daemon. This is good, but there are some cases that it should be turned off as there is no gain. Usually this happens when you are serving medium to large files with a lot concurrent connections. Play around and see what works the best. Please pay attention to keepalive timeout as well (we will shortly post sysctl tweaks for network stack as well).

KeepAlive On
Maxkeepaliverequests is the maximum number of http requests over one persistent connection. If you are serving a lot of small files, increasing this value will boost overal performance. If the persistent connectivity is closed and browser requests a new request, a new connection will open, thus, slowing down overal performance a bit. This value however doesn’t apply if you have keepalive’s turned off.

MaxKeepAliveRequests 1000

Keepalivetimeout is the value in seconds to wait for the next request over the same connection for the same client. If you are serving a lot of small files, increasing this value will help. In another case, if you have thousands of concurrent connections, decreasing this value will boost the performance.

KeepAliveTimeout 15

For high load servers, please test KeepAliveTimeout with 2…5 seconds and see how does it affect you.

I hope this helps you. Enjoy

DC