Interestingly it didn’t initially work 100% of the time, after a little bit of testing it looked like Microsoft were temporarily throwing away some of the test e-mails. Interesting as it must mean that they look for repetitious e-mails in a bid to fight SPAM. We of course simply contacted MS and got it sorted and now the e-mails come in every hour.

What this does show us though, is that under UNIX/Linux there is a lot of different solutions that can be made, but sometimes you have to work with the tools that are there.

First up I checked to see that I was able to get a POP3 connection to the other domain, as it is hosted by Microsoft for us. Whilst I was able to connect via POP3 I was unable to login. A quick check revealed that only POP3S was being accepted. This meant that I was going to need to utilize the simple s_client tool available with OpenSSL.

So to testing:

$ openssl s_client -connect SERVER:995
.... blah blah ....
+OK The POP3 Service is ready.
USER test.user@example.com
^C
$

Hrmm, what’s going on here, the server isn’t responding to my USER. A quick brain check and I spotted my issue, line endings!

$ openssl s_client -connect SERVER:995 -crlf
.... blah blah ....
+OK The POP3 Service is ready.
USER test.user@example.com
+OK
PASS mypassword
+OK User successfully logged in.
STAT
+OK 49 2758446
QUIT
DONE

That’s more like it. Now I know that I can connect, logged and run arbitrary commands. It may be perfectly OK to manually check like this but that isn’t the reason we started this, we want an automated solution.

The most obvious solution here is expect, ugh TCL! With the above information it is easy to write a simple expect script:

#!/usr/bin/env expect
spawn openssl s_client -connect SERVER:995 -crlf
expect "+OK The POP3 service is ready."
send "USER test.user@example.com\r"
expect "+OK"
send "PASS mypassword\r"
expect "+OK User successfully logged on."
send "QUIT\r"
expect "DONE"
expect eof

Simple enough, send a command and expect a response. Obviously it helps if you understand the protocol that you are dealing with. Running that script happily logs in and then quits. Sweet, we are halfway there already!

So to recap, I plan on sending an e-mail from the first domain to my test account on the second domain and then check that the e-mail actually arrived. Thus proving that the e-mail was getting through without issue. So when I do this I will wish to get the newest e-mail in my INBOX, which may not the be the first. So the script needs some modifying.

#!/usr/bin/env expect
spawn openssl s_client -connect SERVER:995 -crlf
expect "+OK The POP3 service is ready."
send "USER test.user@example.com\r"
expect "+OK"
send "PASS mypassword\r"
expect "+OK User successfully logged on."
send "STAT\r"
expect -re "OK (.*) .*" {
  set number $expect_out(1,string)
}
send "TOP $number 1\r"
expect "\."
expect "+OK"
send "QUIT\r"
expect "DONE"
expect eof

Now expect will grab the number of e-mails in the INBOX and request the headers of the newest one. So the final step is integrating this with a simple check. Lets just use a shell script to check with:

#!/bin/sh
TS=`date +%s`
echo "Subject: TESTING $TS
 
Hi,
 
This is a test please do not delete me!
 
Thanks" | mail test.user@example.com
sleep 60
expect collect.exp | grep $TS && \
 echo "Success The E-Mail Arrived!" ||\
 echo "Failure The E-Mail Was Not Found!"

That works, a test and we have a success! We find the TS in the subject line of the e-mail. There is a minor problem here, what happens if I e-mail the test and in-between I receive some other e-mails, hrmm! OK so why don’t we dump the headers of the first few e-mails, we certainly don’t expect a rush of e-mails on a test account within a minute. Let’s change the expect script just a bit more:

#!/usr/bin/env expect
 
spawn openssl s_client -connect SERVER:995 -crlf
expect "+OK The POP3 service is ready."
send "USER test.user@example.com\r"
expect "+OK"
send "PASS mypassword\r"
expect "+OK User successfully logged on."
send "STAT\r"
expect -re "OK (.*) .*" {
  set number $expect_out(1,string)
}
send "TOP $number 1\r"
expect "\."
expect "+OK"
set number [expr $number-1]
send "TOP $number 1\r"
expect "\."
expect "+OK"
set number [expr $number-1]
send "TOP $number 1\r"
expect "\."
expect "+OK"
send "QUIT\r"
expect "DONE"
expect eof

Placing the shell script into a crontab and we are done. Yet another example of how simple scripting can achieve anything.

First up you need to discover the storage group that your mailbox is in. This requires a quick search on an active directory server. Once you have that you can go on to create a ‘recovery storage group’ linking it to the correct storage group.

Then you start up your TSM restore and recover the storage group (we use TSM as our backup). On our setup that means that you are restoring approximately 20Gb of data. Once thats done (it can take a while), we can then mount the recovery group. At this point we are left with one of two options: copy all the mail and folders into your existing mail account effectively doubling your mail, or you can merge the contents back into your mail so only copying what is missing.

My issue is that neither of these options are really what a use wants. If we copy the mail back, then we effectively double the amount of mail that a user has. If we merge the data back, it can merge e-mails back into folders that maybe the user will not look at, thus leaving behind stray meant-to-be-deleted e-mails. Nine times out of ten all the user has lost is a single folder or a single mail.

The solution to this is to add another step into the process, extract the mailbox from the recovery group into a PST file, then merge the folder out of the PST file back into the mailbox in the main storage group. But even here there is a problem, if you don’t know the exact folder name you can’t find out what it is and therefore cannot restore it.

In reality what is needed here is a solution that allows the administrators the ability to extract individual folders within a mailbox. It is obvious that the exchange restores are built with the intent of only being used in a situation of disaster, total loss of a storage group or something similar. However almost all the restores undertaken will be for single users.

Looking back to the old method we used with exim makes this look even more silly. As there we could simply find the file representing the mail folder and restore it. Done!