
 = SMTP pain and torture

 INTRODUCTION

  This page documents some difficulties I had attempting to send an email using a linux
  'bash' script. In principle, I thought, this should be easy, because all I had to do
  was get an email into a mailbox on the 'localhost' using SMTP. No authorization was
  necessary. However I ran into all sorts of 'new-line' problems and protocol problems,
  mainly having to do with my sending the data to the SMTP server too fast.

  I suppose subconciously I was assuming that 'netcat' (which is greatly praised on the
  Web) would 'wait' for the server to reply before sending each line of 'input'. But
  how would it deal with protocols that allowing multiple line 'inputs' (such as the
  SMTP DATA section)? The answer to this may be to use the -e switch which executes
  a program AFTER connection. Then the question is going to be, how to read standard
  output within the bash script (readline?) This document also tries to provide some
  links to help resources for the 'netcat' tool. Netcat is a unix-born tool to create
  simple scripted tcp clients and servers. It has been rated by my knowledgeable
  'network people', ie security etc people to be one of the most useful tools available

  Essentially why I have persisted with netcat is that it is very frequently desirable
  to be able to establish a tcp connection from within a shell script and to send 
  text string to that connection. Netcat seems the right tool to use to do this.

  This file is pretty disorganized because I was pretty frustrated when I wrote it.
  mjb

 QUICK CONCLUSIONS

   Even more success. by putting a 'sleep 1;' directly after the DATA command, the
   whole email gets sent successfully. It is not necessary to put a 'sleep' command
   anywhere else. I took the netcat line out of 'bashmail' and piped the output
   of bashmail into netcat without the -i delay switch. The email now gets sent very quickly, like this: 
   -->>
   ./bashmail.sh matthew@ella-associates.org hello | netcat -v -v localhost 25
   --<<

   Despite this success, it would still be useful to have (say) a java equivalent that
   could do authorization etc.

   The tools and examples for netcat are not as extensive as I thought.

   To use netcat as an SMTP client you need to do the following
   use the -i switch to slow down the output of lines
   redirect a script file which has printf or echo -n -e lines
   use '\r\n.\r\n' to terminate the DATA section
   
   
 RESOURCES
   
   'blat'
     The MS command-line e-mail sender (smtp agent)

   zless /usr/share/doc/netcat/README.gz
     On debian linux you should be able to read the netcat docs with this.
     
   http://www.cse.msu.edu/~westrant/symlink/pages/exploits/netcat.htm  
    Some reasonable looking examples of netcat usage.

   http://evolvedcode.net/content/doc_alttelnet/#NetCat
    Some netcat explanations
    
   http://www.xlevel.org.uk/
     a GPL java2 command line smtp client
     Is only provided in 'zip' format which seems a little dodgy 

 MISCELANEOUS

   http://www.cpan.org/scripts/Networking/
     Contains a reference to a perl script for sending sms messages through
     the sms gateway www.genie.co.uk. This requires a special perl module 
     for sms, also available at cpan.

  I should work out how to install perl modules

  http://www.shelldorado.com/search/search-swish.cgi
    A (korn) shell script web site search engine.

 NOTES
   
   
   netcat is much beloved by various net-work hackers. Unfortunately these sort of 
   people do appear to have a definable personality type, which to me is not particulary
   attractive. There seems to be a personal insecurity on the part of the hackers. They
   appear to have a need to appear 'cool' and their language is pretty immature. 
   For example in the README file in the netcat examples, the author of netcat says
   something like 'the scripts are a bit *sick*'. This immaturity really gets in the 
   way of decent documentation. Their documentation is more about trying to show how
   intelligent they are, rather than how the tool can be used.

   This 'culture' actually infects almost all the documents that are available on the Web
   about Netcat and makes the tool much less useful than it might be.
   
THE SMTP SENDING WORKING FILES

 The 'doomail.txt' file
 ---------------------------------------
 echo "HELO ella-associates.org";
 echo "MAIL FROM:<matth3wbishop@yahoo.com>";
 echo "RCPT TO:<matthew@ella-associates.org>";
 echo "DATA";
 echo "subject: testing the netcat mail script with a data section date"
 echo "from: <matth3wbishop@yahoo.com> "
 echo "to: <matthew@ella-associates.org> "
 echo "date: $date "
 echo " "
 echo "this mail uses printf not echo. This took me hours and hours of pain and torture";
 echo "the key seems to be using the -i switch to "
 echo "delay the output of each line"
 echo "and also using an \r\n.\r\n combination to terminate the"
 echo "the data section."
 #echo -n -e "\r\n.\r\n"
 printf "\r\n.\r\n"
 echo "quit"
 ----------------------------------------
 
 The pertinent command line using the above file and netcat to send an email through
 an SMTP Mail Server which is running on the local computer. You should also be able
 to change 'localhost' to the host-name of any valid SMTP server which you are allowed
 to use. Actually, this script does no authorization so it will not work if the 
 SMTP server requires authorization (which is the usual case). 

  ./doomail.txt | netcat -v -v -i1 localhost 25

EARLIER STRUGGLES

  (14 april 2003)
   Last night, early this morning I was attempting to an email to
   matthew@ella-associates.org using Telnet and Netcat from the Local Server
   computer. When I say from, I mean from a 'Putty' window on a Windows Laptop
   in Ametlla de Mar into the 'Local Server'. Strangely I was able to send an
   email to this address but the actual body of the message was not transmitted
   (or not displayed). Using Telnet the entire SMTP session appeared to go
   swimmingly. I got all the right response messages and was told that my message
   has been received. However with 'netcat' I did not appear able to terminate
   the 'DATA' section of the email (the message body). Even when I typed . (dot)
   on a line by itself, it didn't work. I tried sending 'linebreaks' using 
   'printf' and all but nothing worked. Nevertheless, it seems that the message
   was still sent, but without a message body.

   My ideas as to what this problem really is: some strange 'putty' line-break
   mangling problem, a james SMTP protocal problem (unlikely), ...
   Here is an example session telnetting to yahoo: everything seemed to go just
   fine. This was using Putty into the Local Server.
   
   -->>
    Ella_Associates:~# telnet mx1.mail.yahoo.com 25
    Trying 64.157.4.84...
    Trying 64.156.215.5...
    Connected to mta-v20.level3.mail.yahoo.com.
    Escape character is '^]'.
    220 YSmtp mta109.mail.scd.yahoo.com ESMTP service ready
    mail from: <matthew@ella-associates.org>
    250 sender <matthew@ella-associates.org> ok
    rcpt to: <matth3wbishop@yahoo.com>
    250 recipient <matth3wbishop@yahoo.com> ok
    data:
    354 go ahead
    subject: sending an email using telnet to yahoo (mx1.mail.yahoo.com)
    here is the body of
    the text. Will this be transmitted
    .
    250 ok dirdel
    quit
    221 mta109.mail.scd.yahoo.com
    Connection closed by foreign host.

  --<<

  This worked without a problem and the mesage text was transmitted. 
  However there was no 'from' section, and perhaps James or Outlook express becomes
  morally righteous about not having a 'from' section. or perhaps not.

  And the session using telnet and putty to the local host, and attempting to
  send to myself

  -->>
    Ella_Associates:~# telnet localhost 25
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 13:25:11 +0200 (CEST)
    mail from: <matth3wbishop@yahoo.com>
    250 Sender <matth3wbishop@yahoo.com> OK
    rcpt to: <matthew@ella-associates.org>
    250 Recipient <matthew@ella-associates.org> OK
    data
    354 Ok Send data ending with <CRLF>.<CRLF>
    subject: sending email with telnet with putty from local-server to localhost
    from: matth3wbishop@yahoo.com
    I wonder if this text will
    actually be transmitted thats the thing
    .
    250 Message received
    quit
    221 Ella_Associates Service closing transmission channel
    Connection closed by foreign host.
   --<<

This all seems to have gone OK, But once again Outlook
express doesn't actually display the body of the text. How odd
It seems hard to believe that the body of the text has not 
actually be sent, considering the above mail server messages.
One possibility: Outlook or James expects something like
'body:' or something similar. They shouldn't.

  SUCCESS!! yep. I did the same as above but I put a 'to:' line in
  and I also put a blank line after the 'to:' line. 
The following also worked fine (and the message text was visible).
This suggests that the blank line is the key.

 -->>
    Ella_Associates:~# telnet localhost 25
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 13:55:19 +0200 (CEST)
    mail from: <mat@nowhere.com>
    250 Sender <mat@nowhere.com> OK
    rcpt to: <matthew@ella-associates.org>
    250 Recipient <matthew@ella-associates.org> OK
    data
    354 Ok Send data ending with <CRLF>.<CRLF>
    subject: blank line but no to line

    this is the message
    text
    .
    250 Message received
  --<<
  
  Some conclusions 
  -------------------------
  the blank line after the 'subject:' line or possibly any other such line
  is the key here. I still dont know if this is outlook express or James.
  It seems hard to believe that this is actually part of the RFC.
  I would be inclined to bet that it wasn't

Now to get Netcat to work, so that I can script the thing. There is a file
'doomail.txt' which outputs the correct SMTP strings however the final
 '.' doesn't work. the data does not get transmitted, although the mail
 header does. Now how to overcome this? Netcat clearly has something just
 a tiny strange about how it prints a newline or MORE LIKELY, netcat it
 adding something before or after the '.' character and therefore the
 . is not interpreted as the end of the mail. 

I have tried I think every possible combination of 'carriage returns' and
line feeds with printf and echo. With double quotes and single, nothing 
works. Un gran rollo.
 
   http://cr.yp.to/smtp/mail.html 
     A site with some SMTP examples for my 'netcat' super-dodge.
    This site shows examples of actually 'raw' SMTP sessions which
    can be handy for debugging, and saves me from reading the 
    RFC's everytime again
  
  The inboxes for james mail server are stored at:
  /usr/bin/james/apps/james/var/mail/inboxes/matthew
  and appear to store the mail as some kind of streamed something
  (ie not plain text). ONce the messages have been downloaded into the
  users mail client they are deleted from the inbox immediately
  (they dont call it 'pop' for nothing)
  
----------------------------------------- 
  This may well not belong in this page/document/file/piece of information
  but I've got to put it somewhere. Notes for creating an 'email' sending script
  (ie b2b n-tiered robust application-serving architectured rah rah rah)
  The point is; every form which wants to send email should have the same
  basic input element names and URL target. 

  The http://www.ella-associates.org/ page currently does not have a 
  'Form' element. In fact in Netscae 4.61 the input element do not even 'render'
  or display or do anything else. Who are these developers? yes in modern browsers
  the input elements will display without even a Form element. But in old
  browsers, no. 

  The page mentioned just now has the following input fields

    Nombre y apellido: input type="text" name="textfield" 
    direccionn de correo electronico: input type="text" name="textfield2" 

    idioma de interes: input type="text" name="textfield3"

    interes cultural:
    select name="select"
    option selected>musica
    option>teatro
    option>literatura/poesia

    horario preferido
    select name="select2"
    option selected>mananas
    option>medio dia
    option>tardes
    option>noches

    input type="submit" name="Submit" value="Enviar">
    input type="submit" name="Submit2" value="Cancelar">

      
  
  
  In retrospect I really dont think the netcat is particularly useful
  here. I can't see any way for netcat to 'wait for' a server response
  and react based on that response, which is really what you would usually
  want. Any java, perl, c, or bash smtp client would do the job
  
  Actually there dont seem to be terribly good internet resources for netcat.
  The does not appear to be a decent home page. This is probably because it
  is quite a small tool. 
 
  The docs which come with netcat dont really contain any examples
  but these should be available on the internet because the tool has
  a wide user base and is very flexible.

 SMTP TORTURE AND PAIN AND VARIOUS STAGES OF HELL

  I was trying to write a simple smtp client for netcat but I 
  couldn't get the end of the DATA section to happen. That is the 
  CRLF.CRLF just wouldn't work, in other words, I would never get
  a 'message received' message from the mail server. This is particularly
  strange because all this works perfectly using telnet. I think that 
  I have also got this to work in the past using putty and the raw
  protocol option. Hence this.

  Here is an example netcat smtp session which I have been having.
  
  -->>
    Ella_Associates:/var/www/utils# netcat localhost 25
    220 Ella_Associates SMTP Server (JAMES SMTP Server 2.1) ready Mon, 14 Apr 2003 17:15:08 +0200 (CEST)
    mail from: <mat@big>
    250 Sender <mat@big> OK
    rcpt to: <matthew@ella-associates.org>
    250 Recipient <matthew@ella-associates.org> OK
    DATA
    354 Ok Send data ending with <CRLF>.<CRLF>
    subject: test netcat with manual stdin input, line by line
    to: me

    I put a blank line and will terminate with a . he
    here goes
    .
    l


    .
  --<<

  As can be seen, the dot by itself on a line just does not terminate the 
  DATA sections. There may be a reader of this doc shaking her head and saying
  'but its so obvious'. Sadly to me the problem is not obvious at the momnet
  It must have to do with the way that netcat 'filters' the standard input.

  I tried all this with 'printf' and with 'echo -n -e' etc but this is not 
  working. Another possibility would be to try all this from Windows to 
  see if its is something about putty or linux.
  
  This is truly infuriating

  The same problem arises from NT (windows 2000) when using netcat to manually
  SMTP into the Local server. 
  
  A possibility is that the 'manual' netcat method is failing for a different
  reason from the 'redirected input' method. Perhaps the redirected input method
  (which involves redirecting the output from a text file or from a shell script
  into netcat) is failing because there is no delay before the data section is sent.
  
  The only reason I am persevering with this, is that I think that netcat could be 
  a very usefull tool if I can sort out these sorts of problems.

  I cant believe but it finally worked. Why. I dont know. The lines were these

   ./doomail.txt | netcat -v -v -i2 localhost 25
   
 -->> 
     echo "HELO ella-associates.org";
     echo "MAIL FROM:<matth3wbishop@yahoo.com>";
     echo "RCPT TO:<matthew@ella-associates.org>";
     echo "DATA";
     echo "subject: hello"
     echo " "
     echo "Just testing our new mail system.";
     echo -n -e "\r\n.\r\n"
     #printf "\r\n\r\n.\r\n\r\n"
     echo "quit"
 --<<

 For outlook express you need to have a 'from:' section in the DATA section in order
 for the from label in you inbox to be filled.  Also in Outlook express you need to
 have a blank line underneath the section headers lke 'subject:' or 'from:' as above.

  I think the -i switch was important. [Yes it was] So using either of the below can
  successfully send the email.
   echo -n -e "\r\n.\r\n"  or  printf "\r\n.\r\n"

 The 'delay switch' (the -i switch for netcat) is the absolutely crucial thing. Without a
 delay the mail gets sent but the mail has no content. This is because the 'DATA' section of the email
 gets sent to the James Mail Server befor the Server is ready to receive it.

 On the other hand, if the command 'printf "\n.\n"' is used with a netcat delay (the -i
 switch) the whole e-mail gets sent (including the body) but netcat has to be stopped with
 a control c. This is because the SMTP protocol expects a \r\n.\r\n and NOT just a \n.\n

 Outlook Express also want the from:, to:, and date: sections to all be in the DATA
 section rather than being send as headers.  It doesn't crap out if not but it does not
 fill in the appropriate GUI labels.

COMMENTS

  Added by: mjb, on Tuesday, 15 April 2003, 05:54 PM

Netcat is not really the write approach here 
