Jeff
11-17-2005, 01:19 AM
This guide was written based on a real world example of a VDS customer whos server was being used to send spam as recently as this morning. The point of this guide is to inform all of our VDS customers of how they can spot abuses of their web forms by spammers, and how to take action. I should emphasize that this example is in no way intended to make light of our customer. This is an issue that many are not familiar with, and exists in numerous web forms. In fact, the particular application that was abused on this customer's server is a very popular open source program used by many people these days, and it is growing fast. We want to show you what to look for, how to fix it, and how to contact us for assistance if necessary. To not take action against this "bug" is to allow AOL and other networks to block your host from sending email to their customers. If you decide not to take action when you notice this issue, and part of your business involves sending email to AOL customers, you WILL be blocked by AOL when enough of their customers file spam complaints. It does not take very long either. This issue is more prevalent than many realize, so if you are running a web based contact form of any type that allows someone to fill out their name, email address, comments etc, and send them somewhere via email, you should take action NOW. If you suspect your server is being used to send spam, you can always contact us at any time. We will be more than happy to assist, as we've done for several others so far.
We received a ticket from a customer who stated they received the following email asking if it should be of concern to them:
-------- Original Message --------
Subject: Mail delivery failed: returning message to sender
Date: Wed, 16 Nov 2005 13:03:19 +0100
From: Mail Delivery System <Mailer-Daemon@theCustomersHostname.com>
To: nobody@theCustomersHostname.com
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
mparidca@ol.com
unrouteable mail domain "ol.com"
try2keepsmilin@aol.com
unrouteable mail domain "aol.com"
snglfnjc@aol.com
unrouteable mail domain "aol.com"
nsuck8351@aol.com
unrouteable mail domain "aol.com"
scavenger62@aol.com
unrouteable mail domain "aol.com"
trinidadfelixsr@aol.com
unrouteable mail domain "aol.com"
shirleymnj@aol.com
unrouteable mail domain "aol.com"
nbhallati@aol.com
unrouteable mail domain "aol.com"
rozie912@aol.com
unrouteable mail domain "aol.com"
preardon1635@aol.com
unrouteable mail domain "aol.com"
veronicas@aol.com
unrouteable mail domain "aol.com"
steve2annie@aol.com
unrouteable mail domain "aol.com"
srfconsult@aol.com
unrouteable mail domain "aol.com"
myoh58a@aol.com
unrouteable mail domain "aol.com"
mmitch1628@aol.com
unrouteable mail domain "aol.com"
mummyemy@aol.com
unrouteable mail domain "aol.com"
ruffryder729899@aol.com
unrouteable mail domain "aol.com"
meesterclean@aol.com
unrouteable mail domain "aol.com"
trakete@aol.com
unrouteable mail domain "aol.com"
thezaku@aol.com
unrouteable mail domain "aol.com"
macncheese417@aol.com
unrouteable mail domain "aol.com"
rcdsfernunihagen@aol.com
unrouteable mail domain "aol.com"
nap3k@aol.com
unrouteable mail domain "aol.com"
troyclog@aol.com
unrouteable mail domain "aol.com"
poopooplatta@aol.com
unrouteable mail domain "aol.com"
sunsign6@aol.com
unrouteable mail domain "aol.com"
paulyoung1@aol.com
unrouteable mail domain "aol.com"
syd614@aol.com
unrouteable mail domain "aol.com"
mautry040371@aol.com
unrouteable mail domain "aol.com"
mertins94@aol.com
unrouteable mail domain "aol.com"
papichulo3215@aol.com
unrouteable mail domain "aol.com"
renek24@aol.com
unrouteable mail domain "aol.com"
rayyjayy3132@aol.com
unrouteable mail domain "aol.com"
motpmot@aol.com
unrouteable mail domain "aol.com"
clsvz@aol.com
unrouteable mail domain "aol.com"
nvgrama@aol.com
unrouteable mail domain "aol.com"
seraylucky@aol.com
unrouteable mail domain "aol.com"
mbabe7397@aol.com
unrouteable mail domain "aol.com"
myluv415@aol.com
unrouteable mail domain "aol.com"
philippa15@aol.com
unrouteable mail domain "aol.com"
theeldo@aol.com
unrouteable mail domain "aol.com"
nuskafka3@aol.com
unrouteable mail domain "aol.com"
okensheild@aol.com
unrouteable mail domain "aol.com"
rokakkugouji89@aol.com
unrouteable mail domain "aol.com"
tmoetoc@aol.com
unrouteable mail domain "aol.com"
newmie61@aol.com
unrouteable mail domain "aol.com"
simondmason@aol.com
unrouteable mail domain "aol.com"
murphyr44@aol.com
unrouteable mail domain "aol.com"
mmff69@aol.com
unrouteable mail domain "aol.com"
mistu69man@netscape.net
unrouteable mail domain "netscape.net"
rosem411@aol.com
unrouteable mail domain "aol.com"
slavernsmith@aol.com
unrouteable mail domain "aol.com"
sunflowerfairy11@aol.com
unrouteable mail domain "aol.com"
shmendel87@aol.com
unrouteable mail domain "aol.com"
rickynzx@aol.com
unrouteable mail domain "aol.com"
srctdill@aol.com
unrouteable mail domain "aol.com"
policysfccc@aol.com
unrouteable mail domain "aol.com"
nitemoods2@aol.com
unrouteable mail domain "aol.com"
pegibundy@aol.com
unrouteable mail domain "aol.com"
tammydnvll@aol.com
unrouteable mail domain "aol.com"
nitedweler@aol.com
unrouteable mail domain "aol.com"
miebd@aol.com
unrouteable mail domain "aol.com"
partyzippl@aol.com
unrouteable mail domain "aol.com"
teruakikawamoto@aol.com
unrouteable mail domain "aol.com"
suitan75@aol.com
unrouteable mail domain "aol.com"
toothfairy3600@aol.com
unrouteable mail domain "aol.com"
tillerjt2@aol.com
unrouteable mail domain "aol.com"
msnb73@aol.com
unrouteable mail domain "aol.com"
ml1107@aol.com
unrouteable mail domain "aol.com"
tallbjc@aol.com
unrouteable mail domain "aol.com"
popat56@aol.com
unrouteable mail domain "aol.com"
qtnnyc33@aol.com
unrouteable mail domain "aol.com"
oursmarina@aol.com
unrouteable mail domain "aol.com"
pattihotz@aol.com
unrouteable mail domain "aol.com"
stacyvanm@aol.com
unrouteable mail domain "aol.com"
traceyporpora@aol.com
unrouteable mail domain "aol.com"
wmmoscow@aol.com
unrouteable mail domain "aol.com"
poecmbpoe@wmconnect.com
unrouteable mail domain "wmconnect.com"
markellins@aol.com
unrouteable mail domain "aol.com"
mystikal1455@aol.com
unrouteable mail domain "aol.com"
rd60impala@aol.com
unrouteable mail domain "aol.com"
ownmry@aol.com
unrouteable mail domain "aol.com"
selectautows@aol.com
unrouteable mail domain "aol.com"
macstang77@aol.com
unrouteable mail domain "aol.com"
smile12692@aol.com
unrouteable mail domain "aol.com"
thebigbuzzard59@aol.com
unrouteable mail domain "aol.com"
tmogood@aol.com
unrouteable mail domain "aol.com"
mushroomhead27x@aol.com
unrouteable mail domain "aol.com"
shelleyj420@aol.com
unrouteable mail domain "aol.com"
ralfschafers@aol.com
unrouteable mail domain "aol.com"
------ This is a copy of the message, including all the headers. ------
Return-path: <nobody@theCustomersHostname.com>
Received: from nobody by theCustomersHostname.com with local (Exim 4.52)
id 1EcAIh-00065O-4w; Wed, 16 Nov 2005 00:33:19 +0100
To: "Domain" <email@domainHostingTheWebForm.com>
Subject: blah blah spam
From: "nisowen
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: If you missed the tech boom, don't despair... here is your chance
Message-Id: <E1EcAIh-00065O-4w@theCustomersHostname.com>
Date: Wed, 16 Nov 2005 00:33:19 +0100
SYMBOL: UPDA
Current price: .47
Projected Short Term Growth: 1.00+
Rating: 10 out of 10
UPDATE VERY HOT SECTOR FOR INVESTMENT
Universal Property Development & Acquisition Benchmark Coverage To Be Initiated By Investrend Research
NEW YORK--(BUSINESS WIRE)--Nov. 14, 2005--(Investrend Research Syndicate) Universal Property Development & Acquistion Corp. (OTCBB: UPDA - News) has enrolled in the unique shareholder empowerment platform administered by Investrend Communications, Inc., a provider of financial intelligence programs. Benchmark research coverage will be initiated by an Investrend Research analyst to be assigned in the next few days.
blah blah spam
2824df41c0d4bc6a68647fee38bc3827
." <of6295@domainHostingTheWebForm.com>
MIME-Version: 1.0
X-Mailer: Very Popular Software Mailer
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
of6295@domainHostingTheWebForm.com
Q. What should we observe from this?
A. Everything:
1. The customer's hostname. This indicates that the email was sent from their host. This has been replaced with "theCustomersHostname.com".
2. The domain name of the website that appeared in the email. This has been replaced with "domainHostingTheWebForm.com".
3. The date and time the email was sent - "Date: Wed, 16 Nov 2005 13:03:19 +0100"
4. The username associated with the email. This shows as the user "nobody". If you are using suexec this should show up as the name of the user associated with the domain, making it simple to spot which account the activity originated from.
5. The X-Mailer: portion of the header, which has been replaced with "X-Mailer: Very Popular Software Mailer". The name of the software in this case could have helped to pinpoint the issue.
6. Anything and everything else in the headers. The headers tell the entire story. Once you have viewed just a few, you will see that they are not complicated to read through.
So we have a customer who received a spam email that we have determined originated from their server, and we have the username and software it orginated from. We also have something very important - the time the email was sent.
Our next step is to verify everything against the mail logs. I assume many of you already know where your mail logs are stored for your MTA, along with all of your other system and services logs.
Since this particular server is using Exim, we check /var/log/exim_mainlog for one of the @aol.com email addresses in the list near the top of this thread:
[/var/log]# tail -10000 exim_mainlog|grep try2keepsmilin
2005-11-16 00:37:20 1EcAIh-00065O-4w == try2keepsmilin@aol.com R=lookuphost T=remote_smtp defer (0): SMTP error from remote mail server after end of data: host mailin-02.mx.aol.com [64.12.138.185]: 421-: (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html\n421 SERVICE NOT AVAILABLE
Exim tip: for basic but useful information about your Exim log files, read up on section 48.5 of the Exim server specification, located here: http://exim.org/exim-html-4.50/doc/html/spec.html
Next we go to the link in the log output (http://postmaster.info.aol.com/errors/421dynt1.html) which tells us the following:
EXPLANATION:
The IP address you are sending from has been blocked due to AOL Member complaints and/or high volumes of e-mail.
This pretty much confirms what we already knew: this server has been successfully used to send out spam. Now AOL has blocked our customer because the developers of the software they are using didn't take the time to add a few very simple filtering methods to prevent this from happening. Luckily, correcting the issue is simple and requires very little knowledge of how web applications are put together. This will be explained later, but first we have to locate the buggy web form.
We drop into /usr/local/apache/domlogs, and grep for the time the email was sent, as well as the word "POST", since contact forms typically use the POST method of http to send the information:
bash-2.05b# grep 00:33:19 *|grep POST
domainHostingTheWebForm.com:163.14.180.18 - - [16/Nov/2005:00:33:19 +0100] "POST /contact.php?do=send&id=9089cc380c93011e73ea92b4720eb75d HTTP/1.0" 200 1469 "http://www.domainHostingTheWebForm.com/" "-"
We could have grepped the time from the logfile "domainHostingTheWebForm.com", but sometimes we get lazy. What can we take from this log? Well, for one, it matches the EXACT time the spam email was sent to the customer. Coincidence? Hardly - that is precisely what we were grepping for. Second, we note the name of the .php file (contact.php - changed along with the variable names). Here is our buggy contact form. In fact, we can even correlate the name of the software from the X-Mailer: header with the header from the php file.
Let's take a look at the part of the form that stores the user input information:
if (isset($HTTP_GET_VARS['do']) && ($HTTP_GET_VARS['do'] == 'send')) {
$name = ($HTTP_POST_VARS['name']);
$email_address = ($HTTP_POST_VARS['email']);
$comments = ($HTTP_POST_VARS['comments']);
I will not be going into the details of how PHP works. For that you should visit http://php.net and take a few hours, a day, whatever, to familiarize yourself with how basic web applications work. The important thing to note here is that the web form is like numerous others of all types (PHP, Perl, etc):
1. It has a textbox where someone can enter their name (this gets stored in $name),
2. a textbox for the website visitor's email address (this gets stored in $email_address),
3. and a textbox to enter comments (you guess it, this gets stored in the $comments variable).
At this point the website visitor (or spammer) has filled out the form, and now they are ready to send it. The code that handles the sending of the form data stored in the 3 variables above is here:
if ($email_address) {
mail(ADMIN, $comments, $name, $email_address);
"ADMIN" is already defined elsewhere with the email address of the person running this particular website. The user at that email address is sent the contents of the $comments, $name, and $email_address variables.
If you've read the "Spammers Looking for Vulnerable Web Forms" thread in our security forums, you may have a decent idea of what the issue is here. The issue is that there is no proper sanitizing of the variables. As such, anyone can add carriage returns and line feeds to the variables. For example, they can make this:
$comments = "Hi, please contact me tomorrow between 9AM - 5PM EDT";
into this:
$comments = "GET A FREE HOME LOAN/BUY OUR HERBAL MEDICINES CHEAP/BLAH BLAH SPAM%0ABcc:user1@aol.com,user2@aol.com,user3@aol.c om,whoever@wherever";
Did you see exactly where and how the injection is taking place? If not, read it again. The linefeed - %0A - was "injected" into the form, allowing the spammer to add a Bcc: field and send the comments from the form to whomever they choose. YOU get a copy because your email address is defined in the "ADMIN" portion of the application. THAT is why you will sometimes receive spam emails that have originated from your server and have been sent to a bunch of other people.
"That is really something else", you say to yourself. "But, how do I fix this? I don't know too much about PHP". The answer is simple: before the mail function is called, check the variables for bad values, such as %0A. If the bad value exists in the string, stop further execution of the script immediately.
if (eregi("%0A",$name)) { die(); }
if (eregi("%0A",$email_address)) { die(); }
if (eregi("%0A",$comments)) { die(); }
What the above code does, and it can probably be done much nicer looking and more logically (I will be the first to tell you that I am not a good programmer), is that if either of the 3 variables used by the website visitor to populate the form, to simply stop execution of the form (contact.php) immediately.
You're probably not Bcc'ing anyone in your forms either, so you should strip that too:
if (eregi("Bcc:",$name)) { die(); }
if (eregi("Bcc:",$email_address)) { die(); }
if (eregi("Bcc:",$comments)) { die(); }
In fact, let's go ahead and remove carriage returns - %0D
if (eregi("%0D",$name)) { die(); }
if (eregi("%0D",$email_address)) { die(); }
if (eregi("%0D",$comments)) { die(); }
as well as the string "multipart/":
if (eregi("multipart/",$name)) { die(); }
if (eregi("multipart/",$email_address)) { die(); }
if (eregi("multipart/",$comments)) { die(); }
That's it - you're done. If you want to test it, load up your form in your browser and enter any of those strings in the form, then submit the form. If everything goes as planned, the form should stop executing and you should be left staring at a blank white page.
Q. What if I mess something up?
A. You won't, but make a backup first just in case (cp contact.php contact.php.backup)
Q. I'm not entirely clear on where to put these changes, can you elaborate?
A. Add the filters above after the variables are populated with user input, and before the mail function is called.
Q. If I get stuck, or don't feel comfortable making the changes, can you do it for me?
A. Absolutely. Open a trouble ticket with our helpdesk, and provide as much information as possible. Always include your VEID when opening a ticket.
Q. Will this have any adverse affects on the visitors of my website trying to send me information via my form?
A. Unless they are spammers, no. I have yet to see any reason why anyone would ever need to legitimately add carriage returns, line feeds, or the strings "bcc:" or "multipart/" to any part of their name, email address, or a few lines of comments.
Comments and suggestions are welcome.
We received a ticket from a customer who stated they received the following email asking if it should be of concern to them:
-------- Original Message --------
Subject: Mail delivery failed: returning message to sender
Date: Wed, 16 Nov 2005 13:03:19 +0100
From: Mail Delivery System <Mailer-Daemon@theCustomersHostname.com>
To: nobody@theCustomersHostname.com
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
mparidca@ol.com
unrouteable mail domain "ol.com"
try2keepsmilin@aol.com
unrouteable mail domain "aol.com"
snglfnjc@aol.com
unrouteable mail domain "aol.com"
nsuck8351@aol.com
unrouteable mail domain "aol.com"
scavenger62@aol.com
unrouteable mail domain "aol.com"
trinidadfelixsr@aol.com
unrouteable mail domain "aol.com"
shirleymnj@aol.com
unrouteable mail domain "aol.com"
nbhallati@aol.com
unrouteable mail domain "aol.com"
rozie912@aol.com
unrouteable mail domain "aol.com"
preardon1635@aol.com
unrouteable mail domain "aol.com"
veronicas@aol.com
unrouteable mail domain "aol.com"
steve2annie@aol.com
unrouteable mail domain "aol.com"
srfconsult@aol.com
unrouteable mail domain "aol.com"
myoh58a@aol.com
unrouteable mail domain "aol.com"
mmitch1628@aol.com
unrouteable mail domain "aol.com"
mummyemy@aol.com
unrouteable mail domain "aol.com"
ruffryder729899@aol.com
unrouteable mail domain "aol.com"
meesterclean@aol.com
unrouteable mail domain "aol.com"
trakete@aol.com
unrouteable mail domain "aol.com"
thezaku@aol.com
unrouteable mail domain "aol.com"
macncheese417@aol.com
unrouteable mail domain "aol.com"
rcdsfernunihagen@aol.com
unrouteable mail domain "aol.com"
nap3k@aol.com
unrouteable mail domain "aol.com"
troyclog@aol.com
unrouteable mail domain "aol.com"
poopooplatta@aol.com
unrouteable mail domain "aol.com"
sunsign6@aol.com
unrouteable mail domain "aol.com"
paulyoung1@aol.com
unrouteable mail domain "aol.com"
syd614@aol.com
unrouteable mail domain "aol.com"
mautry040371@aol.com
unrouteable mail domain "aol.com"
mertins94@aol.com
unrouteable mail domain "aol.com"
papichulo3215@aol.com
unrouteable mail domain "aol.com"
renek24@aol.com
unrouteable mail domain "aol.com"
rayyjayy3132@aol.com
unrouteable mail domain "aol.com"
motpmot@aol.com
unrouteable mail domain "aol.com"
clsvz@aol.com
unrouteable mail domain "aol.com"
nvgrama@aol.com
unrouteable mail domain "aol.com"
seraylucky@aol.com
unrouteable mail domain "aol.com"
mbabe7397@aol.com
unrouteable mail domain "aol.com"
myluv415@aol.com
unrouteable mail domain "aol.com"
philippa15@aol.com
unrouteable mail domain "aol.com"
theeldo@aol.com
unrouteable mail domain "aol.com"
nuskafka3@aol.com
unrouteable mail domain "aol.com"
okensheild@aol.com
unrouteable mail domain "aol.com"
rokakkugouji89@aol.com
unrouteable mail domain "aol.com"
tmoetoc@aol.com
unrouteable mail domain "aol.com"
newmie61@aol.com
unrouteable mail domain "aol.com"
simondmason@aol.com
unrouteable mail domain "aol.com"
murphyr44@aol.com
unrouteable mail domain "aol.com"
mmff69@aol.com
unrouteable mail domain "aol.com"
mistu69man@netscape.net
unrouteable mail domain "netscape.net"
rosem411@aol.com
unrouteable mail domain "aol.com"
slavernsmith@aol.com
unrouteable mail domain "aol.com"
sunflowerfairy11@aol.com
unrouteable mail domain "aol.com"
shmendel87@aol.com
unrouteable mail domain "aol.com"
rickynzx@aol.com
unrouteable mail domain "aol.com"
srctdill@aol.com
unrouteable mail domain "aol.com"
policysfccc@aol.com
unrouteable mail domain "aol.com"
nitemoods2@aol.com
unrouteable mail domain "aol.com"
pegibundy@aol.com
unrouteable mail domain "aol.com"
tammydnvll@aol.com
unrouteable mail domain "aol.com"
nitedweler@aol.com
unrouteable mail domain "aol.com"
miebd@aol.com
unrouteable mail domain "aol.com"
partyzippl@aol.com
unrouteable mail domain "aol.com"
teruakikawamoto@aol.com
unrouteable mail domain "aol.com"
suitan75@aol.com
unrouteable mail domain "aol.com"
toothfairy3600@aol.com
unrouteable mail domain "aol.com"
tillerjt2@aol.com
unrouteable mail domain "aol.com"
msnb73@aol.com
unrouteable mail domain "aol.com"
ml1107@aol.com
unrouteable mail domain "aol.com"
tallbjc@aol.com
unrouteable mail domain "aol.com"
popat56@aol.com
unrouteable mail domain "aol.com"
qtnnyc33@aol.com
unrouteable mail domain "aol.com"
oursmarina@aol.com
unrouteable mail domain "aol.com"
pattihotz@aol.com
unrouteable mail domain "aol.com"
stacyvanm@aol.com
unrouteable mail domain "aol.com"
traceyporpora@aol.com
unrouteable mail domain "aol.com"
wmmoscow@aol.com
unrouteable mail domain "aol.com"
poecmbpoe@wmconnect.com
unrouteable mail domain "wmconnect.com"
markellins@aol.com
unrouteable mail domain "aol.com"
mystikal1455@aol.com
unrouteable mail domain "aol.com"
rd60impala@aol.com
unrouteable mail domain "aol.com"
ownmry@aol.com
unrouteable mail domain "aol.com"
selectautows@aol.com
unrouteable mail domain "aol.com"
macstang77@aol.com
unrouteable mail domain "aol.com"
smile12692@aol.com
unrouteable mail domain "aol.com"
thebigbuzzard59@aol.com
unrouteable mail domain "aol.com"
tmogood@aol.com
unrouteable mail domain "aol.com"
mushroomhead27x@aol.com
unrouteable mail domain "aol.com"
shelleyj420@aol.com
unrouteable mail domain "aol.com"
ralfschafers@aol.com
unrouteable mail domain "aol.com"
------ This is a copy of the message, including all the headers. ------
Return-path: <nobody@theCustomersHostname.com>
Received: from nobody by theCustomersHostname.com with local (Exim 4.52)
id 1EcAIh-00065O-4w; Wed, 16 Nov 2005 00:33:19 +0100
To: "Domain" <email@domainHostingTheWebForm.com>
Subject: blah blah spam
From: "nisowen
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: If you missed the tech boom, don't despair... here is your chance
Message-Id: <E1EcAIh-00065O-4w@theCustomersHostname.com>
Date: Wed, 16 Nov 2005 00:33:19 +0100
SYMBOL: UPDA
Current price: .47
Projected Short Term Growth: 1.00+
Rating: 10 out of 10
UPDATE VERY HOT SECTOR FOR INVESTMENT
Universal Property Development & Acquisition Benchmark Coverage To Be Initiated By Investrend Research
NEW YORK--(BUSINESS WIRE)--Nov. 14, 2005--(Investrend Research Syndicate) Universal Property Development & Acquistion Corp. (OTCBB: UPDA - News) has enrolled in the unique shareholder empowerment platform administered by Investrend Communications, Inc., a provider of financial intelligence programs. Benchmark research coverage will be initiated by an Investrend Research analyst to be assigned in the next few days.
blah blah spam
2824df41c0d4bc6a68647fee38bc3827
." <of6295@domainHostingTheWebForm.com>
MIME-Version: 1.0
X-Mailer: Very Popular Software Mailer
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
of6295@domainHostingTheWebForm.com
Q. What should we observe from this?
A. Everything:
1. The customer's hostname. This indicates that the email was sent from their host. This has been replaced with "theCustomersHostname.com".
2. The domain name of the website that appeared in the email. This has been replaced with "domainHostingTheWebForm.com".
3. The date and time the email was sent - "Date: Wed, 16 Nov 2005 13:03:19 +0100"
4. The username associated with the email. This shows as the user "nobody". If you are using suexec this should show up as the name of the user associated with the domain, making it simple to spot which account the activity originated from.
5. The X-Mailer: portion of the header, which has been replaced with "X-Mailer: Very Popular Software Mailer". The name of the software in this case could have helped to pinpoint the issue.
6. Anything and everything else in the headers. The headers tell the entire story. Once you have viewed just a few, you will see that they are not complicated to read through.
So we have a customer who received a spam email that we have determined originated from their server, and we have the username and software it orginated from. We also have something very important - the time the email was sent.
Our next step is to verify everything against the mail logs. I assume many of you already know where your mail logs are stored for your MTA, along with all of your other system and services logs.
Since this particular server is using Exim, we check /var/log/exim_mainlog for one of the @aol.com email addresses in the list near the top of this thread:
[/var/log]# tail -10000 exim_mainlog|grep try2keepsmilin
2005-11-16 00:37:20 1EcAIh-00065O-4w == try2keepsmilin@aol.com R=lookuphost T=remote_smtp defer (0): SMTP error from remote mail server after end of data: host mailin-02.mx.aol.com [64.12.138.185]: 421-: (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html\n421 SERVICE NOT AVAILABLE
Exim tip: for basic but useful information about your Exim log files, read up on section 48.5 of the Exim server specification, located here: http://exim.org/exim-html-4.50/doc/html/spec.html
Next we go to the link in the log output (http://postmaster.info.aol.com/errors/421dynt1.html) which tells us the following:
EXPLANATION:
The IP address you are sending from has been blocked due to AOL Member complaints and/or high volumes of e-mail.
This pretty much confirms what we already knew: this server has been successfully used to send out spam. Now AOL has blocked our customer because the developers of the software they are using didn't take the time to add a few very simple filtering methods to prevent this from happening. Luckily, correcting the issue is simple and requires very little knowledge of how web applications are put together. This will be explained later, but first we have to locate the buggy web form.
We drop into /usr/local/apache/domlogs, and grep for the time the email was sent, as well as the word "POST", since contact forms typically use the POST method of http to send the information:
bash-2.05b# grep 00:33:19 *|grep POST
domainHostingTheWebForm.com:163.14.180.18 - - [16/Nov/2005:00:33:19 +0100] "POST /contact.php?do=send&id=9089cc380c93011e73ea92b4720eb75d HTTP/1.0" 200 1469 "http://www.domainHostingTheWebForm.com/" "-"
We could have grepped the time from the logfile "domainHostingTheWebForm.com", but sometimes we get lazy. What can we take from this log? Well, for one, it matches the EXACT time the spam email was sent to the customer. Coincidence? Hardly - that is precisely what we were grepping for. Second, we note the name of the .php file (contact.php - changed along with the variable names). Here is our buggy contact form. In fact, we can even correlate the name of the software from the X-Mailer: header with the header from the php file.
Let's take a look at the part of the form that stores the user input information:
if (isset($HTTP_GET_VARS['do']) && ($HTTP_GET_VARS['do'] == 'send')) {
$name = ($HTTP_POST_VARS['name']);
$email_address = ($HTTP_POST_VARS['email']);
$comments = ($HTTP_POST_VARS['comments']);
I will not be going into the details of how PHP works. For that you should visit http://php.net and take a few hours, a day, whatever, to familiarize yourself with how basic web applications work. The important thing to note here is that the web form is like numerous others of all types (PHP, Perl, etc):
1. It has a textbox where someone can enter their name (this gets stored in $name),
2. a textbox for the website visitor's email address (this gets stored in $email_address),
3. and a textbox to enter comments (you guess it, this gets stored in the $comments variable).
At this point the website visitor (or spammer) has filled out the form, and now they are ready to send it. The code that handles the sending of the form data stored in the 3 variables above is here:
if ($email_address) {
mail(ADMIN, $comments, $name, $email_address);
"ADMIN" is already defined elsewhere with the email address of the person running this particular website. The user at that email address is sent the contents of the $comments, $name, and $email_address variables.
If you've read the "Spammers Looking for Vulnerable Web Forms" thread in our security forums, you may have a decent idea of what the issue is here. The issue is that there is no proper sanitizing of the variables. As such, anyone can add carriage returns and line feeds to the variables. For example, they can make this:
$comments = "Hi, please contact me tomorrow between 9AM - 5PM EDT";
into this:
$comments = "GET A FREE HOME LOAN/BUY OUR HERBAL MEDICINES CHEAP/BLAH BLAH SPAM%0ABcc:user1@aol.com,user2@aol.com,user3@aol.c om,whoever@wherever";
Did you see exactly where and how the injection is taking place? If not, read it again. The linefeed - %0A - was "injected" into the form, allowing the spammer to add a Bcc: field and send the comments from the form to whomever they choose. YOU get a copy because your email address is defined in the "ADMIN" portion of the application. THAT is why you will sometimes receive spam emails that have originated from your server and have been sent to a bunch of other people.
"That is really something else", you say to yourself. "But, how do I fix this? I don't know too much about PHP". The answer is simple: before the mail function is called, check the variables for bad values, such as %0A. If the bad value exists in the string, stop further execution of the script immediately.
if (eregi("%0A",$name)) { die(); }
if (eregi("%0A",$email_address)) { die(); }
if (eregi("%0A",$comments)) { die(); }
What the above code does, and it can probably be done much nicer looking and more logically (I will be the first to tell you that I am not a good programmer), is that if either of the 3 variables used by the website visitor to populate the form, to simply stop execution of the form (contact.php) immediately.
You're probably not Bcc'ing anyone in your forms either, so you should strip that too:
if (eregi("Bcc:",$name)) { die(); }
if (eregi("Bcc:",$email_address)) { die(); }
if (eregi("Bcc:",$comments)) { die(); }
In fact, let's go ahead and remove carriage returns - %0D
if (eregi("%0D",$name)) { die(); }
if (eregi("%0D",$email_address)) { die(); }
if (eregi("%0D",$comments)) { die(); }
as well as the string "multipart/":
if (eregi("multipart/",$name)) { die(); }
if (eregi("multipart/",$email_address)) { die(); }
if (eregi("multipart/",$comments)) { die(); }
That's it - you're done. If you want to test it, load up your form in your browser and enter any of those strings in the form, then submit the form. If everything goes as planned, the form should stop executing and you should be left staring at a blank white page.
Q. What if I mess something up?
A. You won't, but make a backup first just in case (cp contact.php contact.php.backup)
Q. I'm not entirely clear on where to put these changes, can you elaborate?
A. Add the filters above after the variables are populated with user input, and before the mail function is called.
Q. If I get stuck, or don't feel comfortable making the changes, can you do it for me?
A. Absolutely. Open a trouble ticket with our helpdesk, and provide as much information as possible. Always include your VEID when opening a ticket.
Q. Will this have any adverse affects on the visitors of my website trying to send me information via my form?
A. Unless they are spammers, no. I have yet to see any reason why anyone would ever need to legitimately add carriage returns, line feeds, or the strings "bcc:" or "multipart/" to any part of their name, email address, or a few lines of comments.
Comments and suggestions are welcome.