web6 Emergency Reboot

[Jeff]

Hi,

web6 is being rebooted at this time. I will update this thread when it is back up, which should be shortly. I apologize for any inconvenience.

[Jeff]

The server is now back online.

[Jeff]

Hi,

If you've been a customer of Myriad Network for any length of time, you know that we take downtime of any server very seriously, and always notify our customers of why any downtime occurred when possible.

I would like to explain now the reason for the downtime earlier (which was less than 10 minutes total).


1. We use modsec (http://www.modsecurity.org) on all of our servers. modsec enables us to perform some webserver firewalling security functions, to include keeping spammers from abusing any web based contact forms, amongst other things. The modsec software was installed via cPanel, which, to install, consists of checking a box that says:

Install and Keep Updated

This software was not being updated automatically by cPanel as it should have been. Below are the versions before and after the upgrade:

Before the upgrade:

Installed Version: 1.8.7-1.00RC1
Version: 1.9.1-1.7

After the upgrade:

Installed Version: 1.9.1-1.7
Version: 1.9.1-1.7

Some people have been complaining that modsec is consuming massive amounts of memory, crashing their servers (http://www.gotroot.com/tiki-view_forum_thread.php?forumId=35&comments_parentId=658). The issue earlier was that the server got critically low on memory, and, although it was still online, became unresponsive to web requests and most other system activity.


Now, we don't use unnecessarily large rulesets for modsec that contain over 18,000 lines of filtering rules (http://www.gotroot.com/tiki-index.php?page=mod_security+rules), but prefer to keep our rulesets trimmed down, blocking only what's necessary. This means that, for every web request, modsec only has to parse a minimal amount of rules before deciding to either accept or deny the traffic, vice 18,000 rules per web request.

However, even with the minimalist ruleset we were using, there were still instances of httpd running that were consuming unnecessarily large amounts of system memory. This activity has not been noticed since modsec was upgraded on the server after the reboot.

This would also explain the last web6 reboot thread that was posted. At the time there was no information to post about why it needed to be rebooted, because there was simply no information for us to work with in the system logs, as there was this time. We spent a great deal of time examining and reexamining various system logs the last time this happened to no avail. Furthermore, we received no notifications from the server before it became unresponsive, as the server likely did not have very much time to handle much of anything before it went into an unresponsive state.

If you have any questions or comments, please feel free to post them in this thread. Thanks.

[cvos]

how do we know if were on web6 or not?

[tomp]

Hi Cayley,

Shoot me an email and I will take a look for you. I don't think you have any accounts on that server. The easiest way to see what server you're on is by telnetting to the IP address of your website on port 25.

telnet www.example.com 25

The banner that will be displayed will identify the server you're on.

Tom