Unix Systems Group

If you have a question that is not covered by one of the other blog postings, please post it here. Please be sure to leave an Email address we can reach you at, we’ll respond to you directly as soon as possible.

After an extensive evaluation headed by Student Government, the University of South Florida and Google have entered a partnership to bring Google Apps to all USF students. Google Apps is an exciting new system that will provide us with email, as well as a suite of other Google products that will enable USF students to better communicate, share, and collaborate.

To sign up, visit http://mail.usf.edu

Don’t worry, your email address won’t change, and you won’t lose your old messages! The nice thing about Google Apps is that everyone will be able to keep the same email address and we’ll migrate the mail from your existing email account to Google Apps. This means that you can search and access all mail in one central location.

Here are some of the other key benefits:

• 6 Gigabytes of email storage — No more worries about having to delete mail.
• Instant messaging & voice calls — Connect instantly with others when you have a question.
• Google Calendar — Schedule meetings, book rooms, create events, and share calendars with others.
• Docs and Spreadsheets — Create and collaborate in real-time with others at your location, and across the globe.
• Start Page — Start off the day on the right foot by adding your favorite gadgets to your personalized start page.
• Access your information anywhere, anytime — All content is available online. You can also access your email with a mobile device .
• Reliability — Google Apps products are built with speed and reliability in mind.

You can use these URLs to access the service once you’ve registered:

• Mail - http://mail.mail.usf.edu
• Calendar - http://calendar.mail.usf.edu
• Docs - http://docs.mail.usf.edu
• Start Page - http://start.mail.usf.edu

Due to a required system upgrade, one of the Blackboard servers must be restarted at Midnight on Friday, 1/12. This server is also providing the storage for mail.usf.edu accounts, so we must bring the mailserver off-line while the upgrade is completed. Services will unavailable for about 30 minutes — 12:00AM-12:30AM. WebMail and POP/IMAP services will be unavailable during this period, but incoming messages will be delivered when the server is brought back on-line.

We’re getting a lot of questions about spam, so I thought I’d go over what we are doing about the problem.

Please be aware that the following information is for @mail.usf.edu accounts only. If you have an @eng.usf.edu or @stpt.usf.edu account you can use WebMail, but none of these anti-spam features are available to you.

The Problem

By now everyone has heard of, and received spam, so I’m not going to explain what it is, but I want to give you some perspective on the size of the problem we are dealing with. We receive around 300,000 email messages on an average day and we’ve had peaks of over 500,000 per day. That’s a lot of mail, and scanning each message for viruses and spam is very CPU-intensive. Spam scanning is especially hard, because of the nearly-infinite variations that spams come in, thousands of tests have to be run on each message. Up until now, scanning was done on the mail server itself, just before the message was placed in your mailbox. This was sufficient when the mail server was put into production back in 2004, but we were only receiving about 1ooK messages per day then. In order for the mail server to handle the increased workload since then, we’ve had to cut down on the number of tests that we used to scan for spam, which limited the effectiveness of the filters.

The Solution

Just before Fall semester, we moved to a different architecture: the scanning is done on a separate set of machines (called MailGate) which then hand the messages to the mail server for final delivery. The new system is working really well, and with your help (more on that later), it will get even better. However it is not perfect. Some spam will still get through, but it will make a huge difference in the amount of spam you receive. MailGate reduces the number of spams you receive in a couple of ways:

Blacklisting

The first step in combating spam happens before a message has even been transferred. When an Email server tries to contact MailGate to send a message, MailGate checks several blacklists and if the server is listed, the connection is denied and no mail is transferred. MailGate also denies access to badly mis-configured or non RFC-compliant mail servers, which are usually spam zombies.

Virus Scanning

At this point, MailGate looks at the message and determines what (if any) files are attached. All files that are executable on Windows (.exe, .bat, etc) are automatically rejected. We are doing this because most Email-borne viruses use these file formats. If you need to send an executable file for some reason, put it into a “zip” archive to get past this check. If the file is not an executable, it is sent to the virus scanner. All archived files are unpacked at this point and the contents are also scanned. If all of the contents are virus-free, the message is then ready for spam scanning.

Rules-Based Spam Scoring

We use SpamAssassin to determine if a message is spam. Spamassassin (SA) uses thousands of rules and text patterns to make this determination. In addition to SA’s built-in rules, we are also using sets of rules that are updated daily to detect the latest types of spam We are also using Razor and DCC which are massive spam databases that messages can be checked against. Each rule has a “spam score” associated with it and once the message has been tested against all of the rules, the message’s total score is added up. If this score is greater than 5.0 (this score may change at some point), the message is considered spam.

Bayesian Filtering

In addition to the rules-based spam scoring, SA also uses Bayesian Filtering to determine the spam score. I’m not going to go into all the details, but basically a bayesian filter “learns” what you think of as spam and non-spam (”ham” in SA terms). In order for a bayesian filter to work, however, you must train it. Here’s where you come in. You may not have noticed, but there is a new link in WebMail when you are reading a message: “Mark as Spam”. This link sends the message to MailGate’s bayesian filter to help train it to see that message as spam. There is a similar link ( “Mark as Non-Spam”) on every message in your SPAM folder which trains the filter to look for Non-Spam. Whenever the spam filter misses a spam message, make sure to mark the message as spam and whenever it mistakenly marks valid mail as spam, make sure to mark it as non-spam.

Delivery

Once all of these filters are run, the message is finally sent to mail.usf.edu for delivery. If you have spam filtering enabled, messages marked as spam by MailGate will be moved into your SPAM folder, if not, the message is delivered to your mailbox as usual. Again, this will NOT catch every spam! For me, it’s catching about 97% right now and with more training, it should get over 99% of the spam.

To make sure that you have the spam filtering enabled:

1. Login to WebMail
2. Click on Options
3. Click on Spam Filtering
4. Choose the destination for your spam
5. Click on Update Spam Filter Action

The spam/virus-scanning system for mail.usf.edu will soon see a major upgrade. In preparation of this upgrade and to increase the amount of available storage for email accounts, some policy changes must be made. Beginning on Wednesday, August 10th 2006, messages marked as Spam and moved to your SPAM folder will be kept for two weeks and then deleted automatically. This change affects your SPAM folder only and messages in any of your other folders will not be affected. If you have any questions, please email us at usg@mailman.acomp.usf.edu or post a comment to this entry.

blog.usf.edu and myweb.usf.edu will be unavailable during the Blackboard maintenance window (12AM-2AM) on Aug 4, 2006. This outage is necessary to accomodate changes needed for the upcoming Blackboard upgrade. No other services (WebMail, mail.usf.edu accounts, etc) will be affected. We apologize for any inconvenience this may cause.

What new features are available?

The most obvious change is the new design. The basic layout has stayed the same, but we have added new graphics and adjusted the design to make WebMail easier to use. You might also notice that there is a new menu option: Remote. This module allows you to grab mail from any POP3 mail server so you can read all of your mail in one place (more on this later). The other major addition is the new WYSIWYG editor — this allows users to write HTML-based messages in an editor similar to MS-Word.

How can I check another Email account through WebMail?

Before trying this, please make sure the account you are trying to check can be reached using the POP3 protocol. This protocol is a standard used by many systems for transferring mail. The remote Email function is WebMail ONLY works with POP3, it will not use IMAP or any other mail protocol.

You need 3 pieces of information before starting this process:

• POP3 Server name (and port number if it is not the default, 110)
• Your Username
• Your Password

Once you have these pieces of information, login to Webmail and click on Remote

Click on Define new server

Fill out each of the fields with the information from above.

Adding a Server

You can leave the password field blank (recommended) and WebMail will ask you for the password whenever it tries to check your mail. If you do enter your password, it will be encrypted and stored in the WebMail database.

Alias - A short description used by WebMail to identify the account (e.g. Work Mail or ENG account)

Leave Mail on Server - Choose this option if you want to download a copy of your messages to WebMail, but keep the original messages on the server. If you de-select this, the messages will be deleted from your mail server after they are transferred to WebMail.

You can choose when to check this account automatically, either at login or whenever you click the “refresh” button on the left-hand pane. If neither of these choices are selected, you must click on the “Remote” icon and started the mail download process manually.

Click on “Add Server”

Downloading Remote Mail

Once the account is configured, click on the “remote” icon.

Select the server you want to check

Type in your password if required and click on “Fetch Mail”

Your new messages will be added to your USF email account.

Will all of my settings (Addressbook, filters, etc) be retained?

Yes. The only thing that they may have been changed when moving to the new version is the “Custom CSS” setting. If you have set a custom font size, you may need to set that again.

How can I compose HTML emails?

Click on the “Compose in HTML” button when you are writing an Email message. A set of toolbars will appear that are very similar to the ones in MS Word. By clicking on the toolbar icons, you can set font size & color, add links or images, you can even use a “virtual keyboard” to add characters from different languages. To make HTML messages the default for your account, Click on “options” and then “Display Preferences”. Scroll down to “Default Email Composition Format” and set it to HTML. Click the Submit button to save your changes.

I don’t see a “Compose in HTML button!!!

You are using an unsupported browser. Unfortunately, HTML mail is only supported in IE 5.5+ and Mozilla/Firefox 1.4+. Safari and Opera support will be added once those browsers support all of the necessary features.

I like the old design!

We’ve included a WebMail theme that is very similar to the old design. Click on “Options” and then “Display Preferences”. Change the theme to “Old USF WebMail”and click Submit to save your changes.

We’ve had some questions about the mail storage quota on mail.usf.edu accounts. The quota was raised to 100MB during the restore process because the mailbox restores caused multiple versions of a mailbox to exist on some accounts (i.e INBOX backed up on 12/13 and 12/20 were both recovered — one as INBOX, the other as INBOX-recover) so there were many people using more than their normal amount of storage. The quota was decreased to 75MB two weeks ago and has now been moved back to the original 50MB.

We’d love to be able to offer more storage, but that is most that we can offer to everyone at this time. If you have a class-related need for more storage, quota increases are available, but the request must be approved by the faculty member you are working with. To request an increase, have your professor send an email to usg@mailman.acomp.usf.edu (make sure they include your NetID in the message!)

Eric Pierce has recovered the hardware from a catastrophic failure on 12/22. We had to retrieve backed up data from tape. Unfortunately, the tape recovery is a long processes. In order to expedite things, we did an initial, partial, recovery of a data set which included about 65,000 or the 70,000 or so accounts we maintain. That took 2 days. Eric then sent a test message to all accounts in order to pinpoint which accounts still have issues. We now started working on retrieving those accounts.

The bottom line:If you are having problems login in to the system, chances are your account is one of the many we are still retrieving. We are working as quickly as we can to restore access to all accounts but we are constrained by the speed of the recovery from the tapes. Once we believe the process is done we will post a follow up, asking those who still may be having problems to contact us directly. If I am pressed for a time frame right now I would say we will probably be done with the restores by Tuesday morning, but keep an eye on this space for further info.

Again, our apologies for the situation.

The campus chilled water supply will be interrupted for 24 hours starting on December 22 at 3:45PM. The air conditioners at USF use chilled water for cooling, so this means that all A/C will be off-line until chilled water is restored. The Blackboard servers will be down during this period. (details here), but we will attempt to keep all other student services provided by Academic Computing (Email, webpages, blogs) running during the outage. These services will also be shut down if they are in danger of over-heating.