Scenario I encountered recently which has me stumped..

• httpd at max showing many backed-up requests
• Incoming traffic is far below normal and pages load incredibly slowly (2-3min per request) -- understandable since most clients connecting are seeing timeouts and slow page loads -- I'm measuring traffic at eth level, so it's not just apache traffic but all traffic to the machine.
• outgoing traffic is also far below normal
• machine is near idle, far below normal on cpu and memory usage
• Plain text pages load just as slow as mod_perl pages
• Restarting httpd doesn't help.. server goes right up to maxing out on requests
• Cycling the box doesn't help.
• Http servers downstream are zippy and mostly idle, only receiving few requests from the main (the problematic one) server
• Everything else on the machine is fine.. no other issues that can be found.. log files are at their typical size, hard drive has plenty of room
• Firewall is logging absolutely nothing unusual
• No errors in any log files (systems and httpd)
• Scan of access_log shows nothing out of ordinary (other than fewer requests than normal due to slowness)
• The same slowness occurs when loading pages locally on the machine itself.
• Nameserver isn't the issue either.. nslookup to the two servers configured were zippy.

Now after couple hours of tearing my hair out, I shut the site down at which point it was only serving a very small text file with a "technical difficulties" msg for any request. This file was taking just as long to load as the pages when site was active. After 15 minutes or so of shutdown, the problem went away. Turned the site back on.. been on for hours, problem has not returned.

I'm suspecting some sort of httpd attack that threw apache in for some internal loop.. but then again, cpus were nearly idle, so it wasn't processing hard.. just spawning processes at the max and not returning pages.

Anyone have any ideas?

There's this handy little thing (that Jeremy pointed out last night, but I already knew..)..

apachectl configtest

It may save headaches at 4am when httpd won't restart after logrotate.. (yes, I know, mod_log_sql.. I know.. I use it for myself).

Was doing this on multiple servers today, might as well get a blog entry out of it.
Note: this requires mod_so to be installed.. w/o that you'll need to do a lot more work.
To find out if you have mod_so, use /usr/local/apache/bin/httpd -l

1. Download mod_throttle
   http://www.snert.com/Software/mod_throttle/mod_throttle312.tgz


2. upack & unzip the tgz file
   tar -xzvf mod_throttle312.tgz


3. Edit mod_throttle.c Change: (this is for a freebsd system)
   
   #define USE_POSIX_SERIALIZATION
   #undef USE_SYSTEM_V_SERIALIZATION


4. Edit Makefile
   
   Important stuff:
   1. APXS=/usr/local/apache/bin/apxs
   2. APACHECTL=/usr/local/apache/bin/apachectl


5. make install (that's not a mistake, don't need to do make first).
   You'll need to do this as root or using sudo since it edits the httpd.conf file.


6. Edit httpd.conf to set a throttle policy
   
   #
   # Throttle policy limits connections to 100k/s per IP address
   # Keeps a history of 1000 ip addresses
   #
   
   <IfModule mod_throttle.c>
      ThrottleClientIP 1000
      ThrottlePolicy Speed 100K 1s
      <Location /ts>
         SetHandler throttle-status
      </Location>
   </IfModule>
   


7. Restart apache
   
   /usr/local/apache/bin/apachectl restart


8. You're now throttling the entire server to 100k/s per IP.

I set a throttle policy of maximum 50 requests per ip per second. The idea is to stop harvesters.. not regular users. Hopefully 50 should be a good balance.. if anyone is seeing problems accessing this site, please let me know?. The idea is to limit site usage to harvesters and scripts, not regular users..

mod_throttle has incredibly dense documenation.. had to read it several times to understand what needed to be done.. and I'm not all that slow.

Here's my config:

In the Server configuration section of apache:

<IfModule mod_throttle.c>
   ThrottleClientIP 100
</IfModule>

Which essentially says: Throttle per client IP, keep a history of 100. 100 should be more than enough for my site, since my average requests per hour is 80-90. Too long of a history and it'll be a burden on system resources.

In the virtual host configuration section:

<IfModule mod_throttle.c>
   ThrottlePolicy Request 50 1
</IfModule>

In plain English: limit requests to 50 per IP address per second. I had it initially set to 20 but found that to be too low.. a very busy page (like MT's config panel) wouldn't load completely.. that's bad.

Now since this is my first time configuring mod_throttle and documentation is rather dense and information online very limited I may find out I'm doing something incredibly stupid here.. I guess I'll have to wait and see :)

Installed mod_throttle today.

The installation instructions are pretty easy to follow.. they don't mention editing the makefile to specify locations of apxs and apachectl if it's no in your path.. but that's pretty self-explanatory anyway.. Should never try to compile something without at least reading the makefile.

I know I'm probably just being paranoid, but it annoys me to see someone use wget on my entire site and suck up all my bandwidth for 30 minutes to download all my files. It has to make connecting for others miserable. I don't even want to think of what nefarious reasons anyone could have to want a copy of my site. (Read? Like anyone reads this..).

So far it's just the basic config that produces the stats (the url isn't a correct one for this site, but it gives you an idea of how this works). No throttling yet.. but soon to follow as I come up with a policy that makes sense and isn't a burden on regular users.

<IfModule mod_throttle.c>
   ThrottlePolicy none
   <Location /throttle-status>
      SetHandler throttle-status
   </Location>
</IfModule>

As I noted couple days ago I am now running Movable Type using mod_perl. It runs nicely.. apparently it can run better when you follow the actual instructions!

Kasia reading and following documentation? Hey, stranger things have happened.. What's next.. commenting my code?

I just got mod_perl running and my Movable Type is now executing under mod_perl.

It's a *lot* faster.. 3 times at least.

Here's the apache config I put in..

<IfModule mod_perl.c>
       <Location /mtype>
            SetHandler perl-script
            PerlHandler Apache::PerlRun
            Options +ExecCGI
        </Location>
</IfModule>

Now this isn't the best way to do this.. since it involes moving all static pages (images, docs, stylesheets) out of the mtype location.. I'm looking into doing it in a more elegant way.

Huge difference in speed!

This is too cool.. can't wait to start writing code to do something with this..

mysql> select time_stamp, remote_host from access_log limit 1;
+------------+---------------+
| time_stamp | remote_host   |
+------------+---------------+
| 1030934140 | 12.243.203.xx |
+------------+---------------+

The things you learn when working on other peoples boxes...

1. Make sure there is only one version of perl installed (remove that /usr/bin/perl5.01)


2. If you want mod_so (you most likely do) don't even attempt to compile mod_perl into apache, it will seg-fault when running


3. If your perl is compiled for large file support, disable it in mod_perl, (PERL_USELARGEFILES=0 when running perl Makefile.PL) otherwise it won't work when installed with apxs - it will seg-fault.


4. If you want large file support, you need to compile mod_perl into apache, and that isn't very happy with mod_so
.

I'm tired :)