= SMTP pain and torture INTRODUCTION This page documents some difficulties I had attempting to send an email using a linux 'bash' script. In principle, I thought, this should be easy, because all I had to do was get an email into a mailbox on the 'localhost' using SMTP. No authorization was necessary. However I ran into all sorts of 'new-line' problems and protocol problems, mainly having to do with my sending the data to the SMTP server too fast. I suppose subconciously I was assuming that 'netcat' (which is greatly praised on the Web) would 'wait' for the server to reply before sending each line of 'input'. But how would it deal with protocols that allowing multiple line 'inputs' (such as the SMTP DATA section)? The answer to this may be to use the -e switch which executes a program AFTER connection. Then the question is going to be, how to read standard output within the bash script (readline?) This document also tries to provide some links to help resources for the 'netcat' tool. Netcat is a unix-born tool to create simple scripted tcp clients and servers. It has been rated by my knowledgeable 'network people', ie security etc people to be one of the most useful tools available Essentially why I have persisted with netcat is that it is very frequently desirable to be able to establish a tcp connection from within a shell script and to send text string to that connection. Netcat seems the right tool to use to do this. This file is pretty disorganized because I was pretty frustrated when I wrote it. mjb QUICK CONCLUSIONS Even more success. by putting a 'sleep 1;' directly after the DATA command, the whole email gets sent successfully. It is not necessary to put a 'sleep' command anywhere else. I took the netcat line out of 'bashmail' and piped the output of bashmail into netcat without the -i delay switch. The email now gets sent very quickly, like this: -->> ./bashmail.sh matthew@ella-associates.org hello | netcat -v -v localhost 25 --<< Despite this success, it would still be useful to have (say) a java equivalent that could do authorization etc. The tools and examples for netcat are not as extensive as I thought. To use netcat as an SMTP client you need to do the following use the -i switch to slow down the output of lines redirect a script file which has printf or echo -n -e lines use '\r\n.\r\n' to terminate the DATA section RESOURCES 'blat' The MS command-line e-mail sender (smtp agent) zless /usr/share/doc/netcat/README.gz On debian linux you should be able to read the netcat docs with this. http://www.cse.msu.edu/~westrant/symlink/pages/exploits/netcat.htm Some reasonable looking examples of netcat usage. http://evolvedcode.net/content/doc_alttelnet/#NetCat Some netcat explanations http://www.xlevel.org.uk/ a GPL java2 command line smtp client Is only provided in 'zip' format which seems a little dodgy MISCELANEOUS http://www.cpan.org/scripts/Networking/ Contains a reference to a perl script for sending sms messages through the sms gateway www.genie.co.uk. This requires a special perl module for sms, also available at cpan. I should work out how to install perl modules http://www.shelldorado.com/search/search-swish.cgi A (korn) shell script web site search engine. NOTES netcat is much beloved by various net-work hackers. Unfortunately these sort of people do appear to have a definable personality type, which to me is not particulary attractive. There seems to be a personal insecurity on the part of the hackers. They appear to have a need to appear 'cool' and their language is pretty immature. For example in the README file in the netcat examples, the author of netcat says something like 'the scripts are a bit *sick*'. This immaturity really gets in the way of decent documentation. Their documentation is more about trying to show how intelligent they are, rather than how the tool can be used. This 'culture' actually infects almost all the documents that are available on the Web about Netcat and makes the tool much less useful than it might be. THE SMTP SENDING WORKING FILES The 'doomail.txt' file --------------------------------------- echo "HELO ella-associates.org"; echo "MAIL FROM:"; echo "RCPT TO:"; echo "DATA"; echo "subject: testing the netcat mail script with a data section date" echo "from: " echo "to: " echo "date: $date " echo " " echo "this mail uses printf not echo. This took me hours and hours of pain and torture"; echo "the key seems to be using the -i switch to " echo "delay the output of each line" echo "and also using an \r\n.\r\n combination to terminate the" echo "the data section." #echo -n -e "\r\n.\r\n" printf "\r\n.\r\n" echo "quit" ---------------------------------------- The pertinent command line using the above file and netcat to send an email through an SMTP Mail Server which is running on the local computer. You should also be able to change 'localhost' to the host-name of any valid SMTP server which you are allowed to use. Actually, this script does no authorization so it will not work if the SMTP server requires authorization (which is the usual case). ./doomail.txt | netcat -v -v -i1 localhost 25 EARLIER STRUGGLES (14 april 2003) Last night, early this morning I was attempting to an email to matthew@ella-associates.org using Telnet and Netcat from the Local Server computer. When I say from, I mean from a 'Putty' window on a Windows Laptop in Ametlla de Mar into the 'Local Server'. Strangely I was able to send an email to this address but the actual body of the message was not transmitted (or not displayed). Using Telnet the entire SMTP session appeared to go swimmingly. I got all the right response messages and was told that my message has been received. However with 'netcat' I did not appear able to terminate the 'DATA' section of the email (the message body). Even when I typed . (dot) on a line by itself, it didn't work. I tried sending 'linebreaks' using 'printf' and all but nothing worked. Nevertheless, it seems that the message was still sent, but without a message body. My ideas as to what this problem really is: some strange 'putty' line-break mangling problem, a james SMTP protocal problem (unlikely), ... Here is an example session telnetting to yahoo: everything seemed to go just fine. This was using Putty into the Local Server. -->> Ella_Associates:~# telnet mx1.mail.yahoo.com 25 Trying 64.157.4.84... Trying 64.156.215.5... Connected to mta-v20.level3.mail.yahoo.com. Escape character is '^]'. 220 YSmtp mta109.mail.scd.yahoo.com ESMTP service ready mail from: 250 sender ok rcpt to: 250 recipient ok data: 354 go ahead subject: sending an email using telnet to yahoo (mx1.mail.yahoo.com) here is the body of the text. Will this be transmitted . 250 ok dirdel quit 221 mta109.mail.scd.yahoo.com Connection closed by foreign host. --<< This worked without a problem and the mesage text was transmitted. However there was no 'from' section, and perhaps James or Outlook express becomes morally righteous about not having a 'from' section. or perhaps not. And the session using telnet and putty to the local host, and attempting to send to myself -->> Ella_Associates:~# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 13:25:11 +0200 (CEST) mail from: 250 Sender OK rcpt to: 250 Recipient OK data 354 Ok Send data ending with . subject: sending email with telnet with putty from local-server to localhost from: matth3wbishop@yahoo.com I wonder if this text will actually be transmitted thats the thing . 250 Message received quit 221 Ella_Associates Service closing transmission channel Connection closed by foreign host. --<< This all seems to have gone OK, But once again Outlook express doesn't actually display the body of the text. How odd It seems hard to believe that the body of the text has not actually be sent, considering the above mail server messages. One possibility: Outlook or James expects something like 'body:' or something similar. They shouldn't. SUCCESS!! yep. I did the same as above but I put a 'to:' line in and I also put a blank line after the 'to:' line. The following also worked fine (and the message text was visible). This suggests that the blank line is the key. -->> Ella_Associates:~# telnet localhost 25 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 13:55:19 +0200 (CEST) mail from: 250 Sender OK rcpt to: 250 Recipient OK data 354 Ok Send data ending with . subject: blank line but no to line this is the message text . 250 Message received --<< Some conclusions ------------------------- the blank line after the 'subject:' line or possibly any other such line is the key here. I still dont know if this is outlook express or James. It seems hard to believe that this is actually part of the RFC. I would be inclined to bet that it wasn't Now to get Netcat to work, so that I can script the thing. There is a file 'doomail.txt' which outputs the correct SMTP strings however the final '.' doesn't work. the data does not get transmitted, although the mail header does. Now how to overcome this? Netcat clearly has something just a tiny strange about how it prints a newline or MORE LIKELY, netcat it adding something before or after the '.' character and therefore the . is not interpreted as the end of the mail. I have tried I think every possible combination of 'carriage returns' and line feeds with printf and echo. With double quotes and single, nothing works. Un gran rollo. http://cr.yp.to/smtp/mail.html A site with some SMTP examples for my 'netcat' super-dodge. This site shows examples of actually 'raw' SMTP sessions which can be handy for debugging, and saves me from reading the RFC's everytime again The inboxes for james mail server are stored at: /usr/bin/james/apps/james/var/mail/inboxes/matthew and appear to store the mail as some kind of streamed something (ie not plain text). ONce the messages have been downloaded into the users mail client they are deleted from the inbox immediately (they dont call it 'pop' for nothing) ----------------------------------------- This may well not belong in this page/document/file/piece of information but I've got to put it somewhere. Notes for creating an 'email' sending script (ie b2b n-tiered robust application-serving architectured rah rah rah) The point is; every form which wants to send email should have the same basic input element names and URL target. The http://www.ella-associates.org/ page currently does not have a 'Form' element. In fact in Netscae 4.61 the input element do not even 'render' or display or do anything else. Who are these developers? yes in modern browsers the input elements will display without even a Form element. But in old browsers, no. The page mentioned just now has the following input fields Nombre y apellido: input type="text" name="textfield" direccionn de correo electronico: input type="text" name="textfield2" idioma de interes: input type="text" name="textfield3" interes cultural: select name="select" option selected>musica option>teatro option>literatura/poesia horario preferido select name="select2" option selected>mananas option>medio dia option>tardes option>noches input type="submit" name="Submit" value="Enviar"> input type="submit" name="Submit2" value="Cancelar"> In retrospect I really dont think the netcat is particularly useful here. I can't see any way for netcat to 'wait for' a server response and react based on that response, which is really what you would usually want. Any java, perl, c, or bash smtp client would do the job Actually there dont seem to be terribly good internet resources for netcat. The does not appear to be a decent home page. This is probably because it is quite a small tool. The docs which come with netcat dont really contain any examples but these should be available on the internet because the tool has a wide user base and is very flexible. SMTP TORTURE AND PAIN AND VARIOUS STAGES OF HELL I was trying to write a simple smtp client for netcat but I couldn't get the end of the DATA section to happen. That is the CRLF.CRLF just wouldn't work, in other words, I would never get a 'message received' message from the mail server. This is particularly strange because all this works perfectly using telnet. I think that I have also got this to work in the past using putty and the raw protocol option. Hence this. Here is an example netcat smtp session which I have been having. -->> Ella_Associates:/var/www/utils# netcat localhost 25 220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 17:15:08 +0200 (CEST) mail from: 250 Sender OK rcpt to: 250 Recipient OK DATA 354 Ok Send data ending with . subject: test netcat with manual stdin input, line by line to: me I put a blank line and will terminate with a . he here goes . l . --<< As can be seen, the dot by itself on a line just does not terminate the DATA sections. There may be a reader of this doc shaking her head and saying 'but its so obvious'. Sadly to me the problem is not obvious at the momnet It must have to do with the way that netcat 'filters' the standard input. I tried all this with 'printf' and with 'echo -n -e' etc but this is not working. Another possibility would be to try all this from Windows to see if its is something about putty or linux. This is truly infuriating The same problem arises from NT (windows 2000) when using netcat to manually SMTP into the Local server. A possibility is that the 'manual' netcat method is failing for a different reason from the 'redirected input' method. Perhaps the redirected input method (which involves redirecting the output from a text file or from a shell script into netcat) is failing because there is no delay before the data section is sent. The only reason I am persevering with this, is that I think that netcat could be a very usefull tool if I can sort out these sorts of problems. I cant believe but it finally worked. Why. I dont know. The lines were these ./doomail.txt | netcat -v -v -i2 localhost 25 -->> echo "HELO ella-associates.org"; echo "MAIL FROM:"; echo "RCPT TO:"; echo "DATA"; echo "subject: hello" echo " " echo "Just testing our new mail system."; echo -n -e "\r\n.\r\n" #printf "\r\n\r\n.\r\n\r\n" echo "quit" --<< For outlook express you need to have a 'from:' section in the DATA section in order for the from label in you inbox to be filled. Also in Outlook express you need to have a blank line underneath the section headers lke 'subject:' or 'from:' as above. I think the -i switch was important. [Yes it was] So using either of the below can successfully send the email. echo -n -e "\r\n.\r\n" or printf "\r\n.\r\n" The 'delay switch' (the -i switch for netcat) is the absolutely crucial thing. Without a delay the mail gets sent but the mail has no content. This is because the 'DATA' section of the email gets sent to the James Mail Server befor the Server is ready to receive it. On the other hand, if the command 'printf "\n.\n"' is used with a netcat delay (the -i switch) the whole e-mail gets sent (including the body) but netcat has to be stopped with a control c. This is because the SMTP protocol expects a \r\n.\r\n and NOT just a \n.\n Outlook Express also want the from:, to:, and date: sections to all be in the DATA section rather than being send as headers. It doesn't crap out if not but it does not fill in the appropriate GUI labels. COMMENTS Added by: mjb, on Tuesday, 15 April 2003, 05:54 PM Netcat is not really the write approach here