Email

From Peyton Hall Documentation

Revision as of 17:55, 30 October 2013 by Lkoven (Talk | contribs)
Jump to: navigation, search

Email has become one of the most important means of communication among people here, in part because they may not be physically anywhere near their collaborators. Also, being a 'non-intrusive' communication is helpful - if you send someone an email and they're not available to reply right away, they can take their time formulating a response instead of needing to on the spot, or promising a return phone call later when time is available. Quite handy for those 4AM questions that pop up. This section will explain a bit about email, including links for setting up various mail clients in the building, as well as some of the details of our email setup here in Peyton Hall.


Contents

Email basics

While email has become an important and common method of communication, many don't quite understand the details that go behind pressing 'send' in your client. What follows is a brief explanation of how that all works.


How it works

When you compose an email to send somewhere, your mail client generates a series of text which describes the details of the message. There's a few headers written, including who the mail is from (your From: address), the destinations (To:, CC: and BCC: for blind carbon-copied messages - those which are sent somewhere, but not written in the message itself so other recipients are unaware it was sent to them), the Date: header saying when the message was written, and a Subject: header for the subject line of the message. Some clients will also include other headers, such as priority (which works differently in different mail clients) or headers to identify the client program and version. The email is now ready to be sent to a server that speaks Simple Mail Transfer Protocol, or SMTP.

In some cases, when you send a mail, the message is handed off to a SMTP client program such as Sendmail running from the command line (/usr/bin/sendmail). This client program will take the message and handle speaking SMTP to the mail server. In other cases, your mail client will speak SMTP directly to the server. In both cases, the conversation is fairly short; the client tells the server what information it needs, known as the "envelope headers", so called because they could be different than what the message itself has for recipients and headers (ie, if you're sending a BCC of an email to someone, the envelope recipient is that person, even though their name doesn't exist in the actual headers of the email). Once the headers are passed, the client tells the server it's now sending the data of the message, sends the data, then signals that the message is done. At that point, the mail server can either accept the message completely - usually giving a kind of tracking number for the email - or reject it, which should cause the client to pop up an error that the mail could not be delivered. Should the mail be accepted, it is now the responsibility of the mail server which accepted the mail to either deliver it one step closer to the recipient, or return a message to the sender to say that the message could not be delivered. It is unacceptable to have a legitimate email discarded by a mail server and no "delivery status notification" sent back to the sender of the message.

Once the SMTP server holds the mail, it looks at it to determine who needs a copy, and where they are. Through some clever Domain Name Service (DNS) entries, it can tell where a mail for a certain person needs to be delivered, and will contact that server (unless this SMTP server is configured to send all mail to a "relay host", in which case it will relay the message up the chain and that host will now be responsible for delivery - this is common if you have an ISP which requires all emails be sent through their own servers, and not directly from your home computer). Once the SMTP server knows where to hand off the message, it initiates a connection to that server and repeats the process, adding to the message a few headers to mark the email's travels through itself (in most cases, a "Received:" header to show that it received the message and handed it off, while in some cases there may be a few headers added if the message was scanned for spam & viruses). This process will continue to repeat, until an SMTP server holds the message and must deliver it locally somehow.

When the local SMTP server holds the mail, it may be delivered many ways depending on the local configuration. In many cases, the mail will be scanned for viruses and spam content, and assuming the message passes it will be delivered. Some mail storage systems work with plain text files for holding messages, so the mail server can deliver them internally. Others use the Local Mail Transport Protocol (LMTP) to pass a message on to a mail server daemon which clients will connect to for reading mail (using either Post Office Protocol v3 (POP3) or Internet Mail Access Protocol (IMAP)).

As mentioned before, if a SMTP daemon cannot deliver a message any closer to the recipient for some reason, it has two choices for what to do:

  1. Reject the message, and the sending program (SMTPD or client) is now responsible for notifying the sender somehow
  2. Generate a Delivery Service Notification (DSN) message to the sender, notifying them that their email could not be delivered and why

In some cases, DSNs are sent because the recipient does not exist, or their mailbox is full. Sometimes it may be because a virus was detected in the email, or because the mail appeared to be spam (or had enough characteristics to make it 'spammy'). It could also be that the recipient's SMTP server is not responding, in which case you'll see the familiar "Message could not be delivered for 4 hours; will keep trying for up to 5 days" DSN. These mean just what they say: the message could not be delivered in the last four hours, but the program will keep trying to deliver it (usually once every hour) until it is five days old, at which time it will send another DSN to let you know the mail could not be delivered at all. Because SMTP servers will hold messages like this, mail tends to always go through even if a mail server is down for a while. All those trying to send mail to it will just hold their messages and wait until a later time, so the worst problem you'll usually see is that messages are delayed an extra hour from when the server is brought back online.

If you'd like to know how our mail system here is configured, see the below section "The nasty details".


Using email in Peyton Hall

There's a few ways to read your email while in Peyton Hall, which are outlined here. A word on "support" first, however. Because there are so many different ways to access email, we cannot possibly support every permutation of them. However, we have selected a few programs which are widely used and easily setup so that we can guarantee support with those programs. This does not mean you must run only one of those programs; you're free to use whatever software you like for reading and sending emails. However, if the program you've chosen does not work properly, or has some kind of problems that are not a fault of the mail server itself, you will be on your own to figure out the problem and corresponding solution. See Other clients below for information if you wish to use an unsupported email client.

Note: You MUST login to your email inbox with a new account before it is created! If you do not login at least once, you'll have no mailbox yet.


Supported clients

Alpine

From any Unix/Linux machine in the building, there's two main options for viewing your email. The first is Alpine, a text-based email program which will work over SSH connections as well as while you're sitting in front of a machine in Peyton. While it may not have all the whiz-bang features of a GUI mail client, Alpine is still in use by many and is a good no-frills (or low-frills) program to use for accessing email. Because of the way we have it configured, we can also guarantee that even if nothing else works, Alpine should be able to read your mail no matter what - as long as the mail server itself is running. You can find more information about Alpine in the article linked above in this section.


Thunderbird

Another supported client in the building is Thunderbird. This is what became of the mail client portion of the Mozilla suite. Thunderbird is a great IMAP client with many plugins and extensions that can help you effectively and efficiently use your email. Since it supports encryption and secure connections, you can configure Thunderbird for use on your laptop, and it will work the same if you're in the building or sitting at home. More information on Thunderbird is available in its own article (linked above).


Roundcube

Roundcube is an IMAP client which is accessed via a web interface. We run Roundcube on the mail server directly, and using a SSL certificate for encryption means your user name and password are not sent in-the-clear for others to see. You can do just about anything through Roundcube as you would with any other mail client, with the added benefit that you don't need anything more than a web browser to access it. The downside, of course, is if you don't trust the computer in front of you (for example, in an Internet cafe or web kiosk). But if you're visiting some other institution and have web access, you can read and reply to emails with Roundcube. You can access Squirrelmail at https://mail.astro.princeton.edu/roundcube/


Other clients

For unsupported email clients, we have some information here which may be useful in getting it setup to access the mail server. If you have information about another client, you're welcome to either add it here, or if it's a large article create a new page for it and link to it in here instead.

  • Mail.app is the default email client that comes with Mac OS X machines.
  • VM is an email client that runs within Emacs.


Android

Unfortunately, different Android devices use different implementations of the e-mail client, so it's difficult to come up with a catch-all set of instructions. If you have trouble getting outgoing e-mail working with the settings under the section below, try disabling the option to encrypt connections to the outgoing mail server. Our mail server will reject non-encrypted attempts to login, anyway, so it should be safe to try. The problem is that by checking that box, some Android devices force a mode of SSL which our server doesn't support. Instead, our server uses TLS, which works in a different manner (and which Android should support). I've tested this setup on an Motorola Droid X running Android 2.1 (aka Eclair) and it works.


Use your own

If none of these clients fit your needs, you can always pick your own for whatever you'd like to do. The information you'll need to know is:

  • SMTP (outgoing) server: mail.astro.princeton.edu port 587
  • IMAP (incoming) server: mail.astro.princeton.edu port 993
  • Security/encryption must be turned on
    This means that the IMAP port will be 993 instead of 143, and SSL must be turned on. For SMTP, you'll need to tell your client to use TLS encryption. You will also have to supply your user name and password to the client, as it will have to authenticate to the mail server to be able to send mail to recipients not at the astro domain (ie, you're relaying mail through our server from home, and the recipient is in Berkeley).


Accessing email from outside Peyton

As nice as the building is, you can't always be here. So how do you access your mail from somewhere else? Simple!

  • If you have Thunderbird configured on your laptop or home computer, it will work just the same while outside the building as it will while connected to our network.
  • If you have SSH access from wherever you're connecting, you can login to any machine and run Alpine.
  • If all you have is a web browser, you can use Squirrelmail


Email FAQ

This section lists some frequently asked questions about email, and their answers (or links to their answers).


Because it's hard to understand and unnatural to read.

Why shouldn't I top-post an email reply (where the response goes at the top of the email, and the original message gets pushed down to the bottom)?


How do I ...

...get all my mail in one place?

If you have an account with OIT, you also have an email address on their servers. However, many people don't like to have to check multiple accounts, and would rather have their mail all in one location. We'll leave out issues of server availability and uptime :>


To forward OIT mail to here
  1. Go to http://support.princeton.edu/account/ and select "Forward your e-mail" from the list of choices under "Manage your e-mail account". Login with your OIT user name and password.
  2. Click the "Change entry" button.
  3. Select the radio button next to the blank text field, and enter your Peyton email address (username@astro.princeton.edu).
  4. Click "Submit changes" at the bottom of the page.


To forward astro email elsewhere

If you would like your email here to be delivered somewhere else, see the forwarding section of the SmartSieve article. Note that using a .forward file will not work.


...export my email?

If you're leaving the department and closing your account, you'll probably want to take your email with you when you go, either as live copies or as an archive. Or maybe you want to use some mail program that doesn't use IMAP, and needs to see the email folders as flat text files. There's a few ways to do this:

Alpine and Thunderbird can be configured so you can copy your mail from Peyton to your new mail server (as long as it's IMAP). Alpine can also (as can Fetchmail) be used to download text copies of your email, in a few different formats (the most common being 'mbox' format). Depending on your needs, follow one of the links above to see how to get the job done. For automated downloading of mail, Fetchmail is probably the best choice.


...filter email?

You'll want to either read a bit about Sieve - the filtering language used by Cyrus IMAPD - and create a Sieve script, or have a look at SmartSieve which we run here as a web frontend to the Sieve language. Most things you might need to do can be done through the interface in SmartSieve, but worst case scenario you can upload straight Sieve scripts to it as well.


...manage delivered (and tagged) spam?

See Filtering Spam for an example.


...subscribe to such-and-such mailing list?

See the Mailman interface.


...learn more about Mailman & mailing lists?

You can browse the documentation at the main Mailman website, http://www.list.org/docs.html. Chris Kolar wrote some very good docs for list managers and admins, which is available at http://staff.imsa.edu/~ckolar/mailman/ .


...know this email was sent by someone?

If the email is not cryptographically signed (say, with GPG), you don't. Emails can be forged, very easily. They can appear to come from anywhere, and with varying degrees of believability; anything from "no way in hell" to "only the NSA knows." If your work requires you verify that an email came from the person who appeared to send it, then you both need to install GPG (and if you use Thunderbird, you'll want the Enigmail extension; see here) and learn how to use it.

As of 2006/02/20, any automated emails sent from the address listed in Requesting assistance are GPG signed. This doesn't include the actual trouble ticket mails, however it does include the password expiration reminders, and user account expiration notices that are sent out. This way you can verify that the email is legit and is from us here. The key used for signing the mails has the ID 0x5BF560E9 and the fingerprint A197 9F2E 99E1 1CC9 92AC D93C BF8E 880F 5BF5 60E9. The key is also signed by Steve and Leigh (keys 0xE6E97FD4 and 0xB220F113).


...send or receive large emails?

See the article on large emails which describes what they are and what to do.


...find out why a mail got marked as spam?

In order to see why a message was flagged as spam, you must look at the headers of the message. Spamassassin will include a header named X-Spam-Status which includes the score and all the tests which matched the message, and what their result added to or subtracted from the score. If you would like us to look at it and describe it more, you need to send us the message with all headers intact. You can also see below for another way to easily get us the message for inspection.


...teach the spam filter to learn from mistakes?

The spam filter can be taught that a message is spam, or not spam (aka 'ham') and will learn over time. Using Bayesian logic, the filter teaches itself when a message has a very low score that its contents tend to not be spam, and when a message has a very high score that its contents tend to be spam. This way, those messages which are in the middle can be pushed one way or another based on the content of the message and Bayesian inference.

If a message was erroneously flagged as spam, or let through when it clearly was spam, you can help to teach the filter where it went wrong. There is a global folder on the mail server, called LearnSpam, where you can put those messages which were incorrectly classified. You don't need to let us know about a message being there, as we can see them and shouldn't have trouble figuring out if it was wrongly flagged or allowed to pass, unless you want us to explain why a message was flagged. In that case, please send us a message describing the mail you copied there and what you'd like us to tell you about it.

Note: Everyone can copy messages into this folder, but cannot read messages there. Only the sysadmins can read messages placed in this folder, and in the course of training the spam filter we may do so. Rest assured that messages which may find their way to the folder will be kept in confidence, and that only the admins can read messages there (and we only care about the technical aspects of the spam filtering, not really about the content of the message anyway except how it may have been detected as spam).


I created folders in my client, but they're not in Squirrelmail!

Fear not! Squirrelmail uses the IMAP subscription model. Similar to how Usenet worked, before Squirrelmail will let you see the contents of a folder you have to subscribe to it. In this way, if you have a bunch of folders you don't want to see if you login via Squirrelmail, you can turn them off and they won't be visible. Conversely, if you create new folders and want to see them, you have to subscribe to them first. Click on "Folders" at the top of the page, and near the bottom you'll see two sections. The left side shows folders to which you are subscribed, and the right folders to which you are not yet subscribed. Select whichever folders you desire to change (one side at a time, please) and then click the corresponding button beneath the section to submit the changes. You may need to refresh the folder view on the left panel (click "Check email") for the changes to appear.


Why don't I see duplicate emails?

Say for example, you're subscribed to two mailing lists. One list, however, is subscribed to the other, so normally you would see two copies of the message - one from the first list, and one from the second. However, Cyrus IMAPD will discard the duplicate. Why? Because duplicate emails take up more space and are wasteful, and rarely do you need to see the same mail twice. Also, in cases where someone accidentally sends out the same message many times, you'll only see it once. Cyrus checks the actual "Message-ID:" header to verify that the message is a duplicate, so in some cases you may see dupes if the mailing list software (erroneously) rewrites the Message-ID.


Why am I not getting important system announcements?

Make sure you whitelist the address listed in Requesting assistance in whatever mail client you use. When we send out announcements, we will try to always use that as the "From" address. Failure to whitelist the address could mean that mails from help are flagged as spam in your email client, and discarded (or hidden).


People said mail to my new account bounces!

Have you ever logged into your mailbox yet? If you have not, then the mail system hasn't created your account yet. Most people will login within a day or so of getting their account, but if you didn't login yet, then your inbox hasn't been created yet, and mails to your account will of course bounce. Login with one of the mail clients listed, and your inbox will be created for you.


I tried mailing a new user and it bounced!

See above, and tell the user to go login to their mailbox so you can email them. Or perhaps they're not using our mail accounts here, and they use their Princeton account, or somewhere else. Either way, while help could create their account, we can't force them to use it, so this isn't a problem we can fix.


I got this strange email...

While our mail server does a pretty good job of keeping viruses out of your mail, and the majority of people here read their mail on Unix machines that are impervious to Windows viruses anyway, some things occasionally "get through" (especially if a virus starts running rampant before the virus scanner's database is updated, either because the company has not released a new version yet or because the update hasn't run yet). Here's some things to keep in mind:

  1. If you read mail on a Unix machine, and you opened an email with a virus in it, don't fret.
    Just delete the mail and move on. You're fine. Trust me.
  2. If you read mail on a Windows machine, and opened an email with a virus, don't fret.
    Delete the mail, and run your favorite virus scanner to make sure you're fine. More likely than not, you will be uninfected (especially if you don't use MS Outlook). And either way, NEVER OPEN ATTACHMENTS YOU DON'T RECOGNIZE!!
  3. If you get an email that seems strange, but either has no attachments or has a text attachment that says something about a virus being deleted, just delete the mail.
    Some admins subscribe to the theory that all mail should always get through, so if their scanners find a virus in an email they will "cleanse" the mail (ie, remove the virus, sometimes replacing it with a text file stating that a virus was removed) and send the body of the message through unharmed. Due to the way viruses act anymore, I subscribe to the theory that a mail should only go through in its original form; if there was a virus in the mail, then the entire message is blocked. In the cases of non-mass-mailing viruses (more later), a note is sent back to the mail sender telling them that the mail was discarded and why.
  4. If you get emails about a message from *yourself* being returned or blocked due to a virus, and you didn't mail anyone anything, DON'T WORRY ABOUT IT.
    Nobody has hacked into your account, and nobody is sending mail from your machine. One of the biggest virus methods of transport nowadays is through email, and many viruses (those that end with "@mm", meaning "Mass Mailer") will take the address book of someone infected, and send itself to everyone in the list. The trick is, to hide the true location of the virus, they will also pick *another* name in the address book, and use that as the "From:" address. This is quite common, and why things like GPG have come about with digital signatures on email to guarantee that a mail is from the person it claims to be from. It is very trivial for anyone to send mail as anyone else, without ever breaking into anything or knowing another password, and only a trained eye can spot the trickery in the headers (and sometimes only when one looks through the mail logs on the servers in question). Also, a mail such as this does not mean your computer has a virus (unless you actually sent something to the person whose mail server bounced the message, in which case you should run your virus scanner to be sure). If you want to find out who is actually infected, look at the headers of the message, and find the last header that mentions "mail.astro.princeton.edu" - it will usually say something like "Received: from smtp.foo.com by mail.astro.princeton.edu". In the example here, "smtp.foo.com" is who gave the mail to us, so either that machine or one that it trusts is likely infected. You can narrow this down by trying to find someone who has both yourself and the "recipient" of the bounced email in their address book. That is likely the person with an infected computer.
  5. If you want a good resource for virus information, including links to removing some viruses and worms, see the Symantic Threat Explorer.
    This is not only a good place for reading about viruses old and new, but it also explains what they do, how they work, and how to remove them. Also, if you get a mail that you think might be a hoax, you can usually find it here as well.

Here's an article on Backscatter - one of the reasons for receiving mails that "you" sent that bounced back from the recipient - courtesy of Computerworld: http://www.computerworld.com.au/index.php/id;1698505531;fp;16;fpid;1


The nasty details

This section explains all the nitty-gritty of the mail system in use here. For casual users, it's probably not important, but the more curious may want to know exactly what programs touch their email on its travels through the Ether.


SMTPD

After running Sendmail for a long time, I had a look at Postfix before one of the mail server upgrades. While Sendmail configuration has become so cumbersome that there's entire books dedicated to the configuration files (O'Reilly's Bat Book), Postfix's configuration is fairly straightforward. With mostly plain-English directives, I was able to setup a Postfix system that mirrored our Sendmail system in a fraction of the time. Whats more, the Postfix system allowed for sending a mail through a filter and then processing it on the other side, and authentication/authorization and encryption for security. Sendmail is only used for secondary systems now, and only because it's the default installation at the time.

We're running Postfix 2.3.x compiled with SMTP AUTH via SASL (which in turn gets information through PAM using LDAP; there's also a local LDAP slave running on the mail server for its own use and to make the machine autonomous). Postfix is configured to use TLS encryption over the standard port (25), allowing for secure authentication to the server for relaying mail. We also have it listening on port 587 (mail submission from clients), which requires TLS and SMTP AUTH for all connections (even local deliveries).


IMAPD

Originally, all email was delivered to a file named for each user, which was cross-mounted on all machines in the building. While this allowed for the use of things like 'grep' to search your inbox, it had a number of downsides:

  • Mail server downtimes could cause strange NFS locking issues on every machine in the building
  • Lock contention
    If the mail server process and your client were both trying to write to the file at the same time, there was a very real chance that mail would be lost, and/or the file completely corrupted.
  • Remote access to mail was impossible - only option was to login and use Alpine

To make the system more robust, I moved everything over to Cyrus IMAPD. Cyrus is a "black box" mail system - end users do not (and in fact, can not) login to the mail server, but only speak to it over IMAP. This makes the system better for us, since it no longer needs to rely on the availability of the home directory server - in fact, our mail server could be physically removed from the building, and plugged in somewhere else on campus, and as long as it can get the same IP address it uses while here, it will continue to function fully. Cyrus IMAPD allows for filtering via the Sieve filtering language (similar to how Procmail works). Cyrus is also compiled to use SSL or TLS encryption depending on the connection.

We use Cyrus IMAPD 2.3.x.


Spam & virus filtering

In this day and age, it's impossible to run a mail server without filtering for spam (and it's irresponsible to run one without virus scanning as well). Early on in the process, I added AMaViS as a filter in the mail processing pipeline to do just that. It scans every incoming and outgoing mail for viruses, and scores mails based on their content to see if they're spammy or not. The spam filtering comes from SpamAssassin, a powerful filter which is very configurable and routinely updated.

We're running amavisd-new version 2.5.x, and SpamAssassin 3.2.x.


Mailing lists

Originally, mailing lists here were nothing more than "exploders"; an alias which mapped to a long list of recipients, so an email sent to the alias would just be expanded to all the people on the list. While effective, these are harder to maintain and not very powerful. By installing and using Mailman, there's a lot more flexibility available in our lists, including the ability to delegate someone other than us as the maintainer and moderator (important in the case of "user-run" lists for projects and such). We can also offer archiving for mailing lists this way, so that all mails can be stored on the server for future reference.

We currently run Mailman 2.1.x.


Squirrelmail

As mentioned above, Squirrelmail allows anyone to read their email with just a web browser over a secure connection. A web-based IMAP client, Squirrelmail has helped those who might not be at a computer capable of doing SSH, or installing Thunderbird for a full-fledged IMAP client. Some people even use it full-time, instead of a regular IMAP client.

Squirrelmail 1.4.x is the version currently installed.


How it all fits

An email coming in from outside the building to be delivered to a local account takes the following path:

  1. External Postfix
    Postfix takes the details of the email, and when the remote client/server finishes sending it does not immediately reply with an acceptance or rejection message - instead, the connection is held open while processing takes place. Because of this, we never accept responsibility for an email until it's been delivered, so if any stage fails it is still up to the sending computer to either retry or notify the original sender of the failure.
  2. AMaViS
    amavisd-new runs the email through a virus scanner, and through SpamAssassin to assign a score to the message. If the virus scan fails, the message is rejected and Postfix will inform the sending system. SpamAssassin tags all emails with their score, no matter the value, for debugging purposes. The process of spam scanning is:
    1. If the score is above 3.0:
      • X-Spam-Flag is added to the headers with the value "YES".
      • X-Spam-Status header (added regardless) will now start with "Yes", followed by which tests were related to the mail and the score output from each test.
      • X-Spam-Level will contain a number of stars according to the whole number value of the score (3.1=***, 3.9=***, 4.0=****)
    2. If the score is 5.0 or above, the mail is rejected entirely.
  3. Assuming above passed, mail moves to an internal Postfix queue
    This queue deals with local delivery; either handing the message off to Cyrus, or forwarding it to another computer for processing (SDSS and some other lists are processed elsewhere). The internal Postfix checks to make sure the recipient exists, and if not will reject the mail (which carries through the open pipes back to the sender). At this point, if this Postfix accepts the message, amavisd-new sends back an acceptance to the external Postfix, which in turn tells the sending computer that the email was accepted.
  4. Cyrus via LMTP
    The internal Postfix speaks to Cyrus with the Local Mail Transfer Protocol (LMTP), a lightweight version of SMTP for local intra-network processes. Cyrus will accept the message from Postfix, process it through Sieve (if needed) and save it to a mail folder for later viewing.
Retrieved from "https://www.astro.princeton.edu/docs/Email"
Personal tools