After monitoring the comments that Akismet blocked very carefully, I can report that I’ve had no false positives for nearly a week. I’m not quite sure what changed to fix things.
With a dynamic system like Akismet, things will change over time. That is the nature of the beast. I don’t know whether Matt and the crew tweaked something, or whether a concerted poisoning attempt stopped being effective, but I’m glad I can start trusting it again.
I suspect it was the former because the change back was very dramatic, though I’m sure Automattic would not want to admit to it.
Server Trouble
In the meantime, at around one this morning, my server went down, or rather my blog stopped working. After a quick investigation, I determined that the database server was complaining of too many connections. I checked and there were a large number of httpd processes running. Presumably each, or most had a database connection open. Static files were being served ok, but anything involving the database was failing.
I restarted the Apache and that seemed to cure it. I started checking through log files to see if I could determine the culprit, but found nothing suspicious. Fifteen minutes later the site was down again. I then spent the next two hours monitoring the situation. A quick script allowed me to watch the process count:
ps -ef | grep httpd | wc -l
It was growing quite rapidly from an initial 16 to over 100, though the site would start failing at about 80. In the end I gave up when the process count stayed stable for 20 minutes. Though when I checked after a few hours sleep, it had gone down again and was down for over 5 hours. I’m presuming it was an attack of some kind.
It has since gone down again, but the growth in number of processes seems to take a much longer time. I didn’t find anything obvious in the logs that I checked, but maybe it is one of the lesser sites which is being attacked. I will continue to investigate…
I’ve had similar problems on my server for quite a while. Sometimes I get these bursts of connections that would grind my server to a halt. I eventually modified my apache configuration to limit the maximum number of running servers, until I found a level that the box seems to be able to handle.
In the past week nothing has changed on the algorithim side, so it might just have been that the system finally learned to identify the strain of spam and ham you were getting after it got more data. Spam is a lot like a virus and as new strains pop up it can take some time (anywhere from 2 – 10 comments) for it to be identified and squashed.
Mike,
I’m working on a website and using your journalized winter theme, real cool! How do I set up the right column so I upload links from the admin screen in WordPress.
If I use the classic theme, I can upload links through the admin screen, but with your theme, they do not show up.
Thanks,
Mike
Hi Mike,
You need to use the latest version of the them which corrects this problem. Get it from http://zed1.com/journalized/themes/journalized-winter/
Mike
Mike, consider running a cron job such as
killall –HUP httpd
httpd
This will periodically disable your Web daemons for a few milliseconds and restart a single session. Choose frequency in accordance with server strain.
Hi Roy,
That’s a good idea but it would be much ‘cleaner’ to use the proper start up script to do something like that. In my case
/etc/init.d/httpd restart
But I’d still rather do that only having detected something wrong like too many connections
Mike
I assume that restarting httpd will send a signal to one or all of the existing daemons. Why not add a conditional statement (e.g. ps –ef | grep httpd | wc –l > 2). Nothing to lose here… if the state is morbid, killall –HUP httpd; /etc/init.d/httpd restart; (if that’s the path in your distribution)
Roy,
It is the ‘killall’ command that I regard as not clean. For instance, I sometime run more than one web server on my machine. I wouldn’t want to kill all of them and only re-start one.
The restart option for the apache start-up script will first stop the process it started before starting another.
Thus my cron script might look like
if [ `ps -ef | grep httpd | wc -l` -gt "25" ]
then /etc/init.d restart
fi
(I haven’t tried this, so the syntax might be wrong)
Mike
hey. i was wondering what the deal with your photo gallery is. i’d really like one of my own and i love the format of yours. i just don’t know where to get started.
help me?
Hi Maddy,
I use Gallery, from http://gallery.menalto.com/
It’s a good package and pretty easy to set up once you have read the instructions.
Mike
it sayes to update your server
wene i go on the web site it seyes to update my server so im updateing it
Indeed, I got the same attack at my weblog last week. I had to scramble and tone a lot of system services down, since the attacker hit my httpd, and it was configured with a maxclients setting too high, causing a load of over 266.