Unix Systems Group

After an extensive evaluation headed by Student Government, the University of South Florida and Google have entered a partnership to bring Google Apps to all USF students. Google Apps is an exciting new system that will provide us with email, as well as a suite of other Google products that will enable USF students to better communicate, share, and collaborate.

To sign up, visit http://mail.usf.edu

Don’t worry, your email address won’t change, and you won’t lose your old messages! The nice thing about Google Apps is that everyone will be able to keep the same email address and we’ll migrate the mail from your existing email account to Google Apps. This means that you can search and access all mail in one central location.

Here are some of the other key benefits:

• 6 Gigabytes of email storage — No more worries about having to delete mail.
• Instant messaging & voice calls — Connect instantly with others when you have a question.
• Google Calendar — Schedule meetings, book rooms, create events, and share calendars with others.
• Docs and Spreadsheets — Create and collaborate in real-time with others at your location, and across the globe.
• Start Page — Start off the day on the right foot by adding your favorite gadgets to your personalized start page.
• Access your information anywhere, anytime — All content is available online. You can also access your email with a mobile device .
• Reliability — Google Apps products are built with speed and reliability in mind.

You can use these URLs to access the service once you’ve registered:

• Mail - http://mail.mail.usf.edu
• Calendar - http://calendar.mail.usf.edu
• Docs - http://docs.mail.usf.edu
• Start Page - http://start.mail.usf.edu

Mail services will be unavailable from 3A.M. to 6A.M. Tuesday August 7th.

As of 5:40A.M. mail services are restored.

Mail services have been down since 3:50 A.M. The services are being switched to the failover server and should be back up shortly. No mail should have been lost during this time.

-Chance Gray 5:45 A.M. 2007/07/27

UPDATE: Mail services were restored at 6:10AM

Email services will be down for about 15 minutes between midnight and 12:30 A.M. The email data has been moved to a new storage device and the mail services will be unavailable as some final adjustments are made. If you have problems with your account after 12:30 A.M. 2/9/2007 please post here.

00:20 A.M. 2/9/2007
Mail services have been up for about 10 minutes now since the switch. Please post here if you are experiencing any problems with your mail.usf.edu account.

A few of you may have noticed the email services actually recovered twice yesterday but failed immediately after coming back online (once at ~ 5 P.M. for about 1/2 hour and once ~ 9 P.M. for about 10 minutes). This time we opted to move the data once again to a new home. This is not our permanent solution however. Ideally the next time we relocate will not be because of hardware failure. We will do it during Spring break or after the end of Spring session giving advanced notice to all of our intentions. Such a move would require minimal downtime.

The storage system serving mail.usf.edu accounts is having problems again. Mail and Blackboard admins are working on the problem with Sun engineers. We will post updates as they become available.

Update (5:37PM) The system was available briefly, but it is still having problems. We are continuing to work with Sun engineers on this issue.

As of 3A.M. the mail server was brought back online. All mail waiting in the queue has been delivered. Some users are experiencing problems with corrupt files. Most of these are showing up in the log but to be sure post here if you encounter any problems with your account.

Both services have been affected. We are working to restore them. More details within the hour.

Update: my.usf is online again. The filesystem for the mail services is still being checked. It will be 10+ hours before that filesystem is available again. We are going to attempt a restore elsewhere as that may be quicker.

ETA: 3A.M. The filesystem check completed and when the system was brought online it immediately crashed. Due to the nature of the crash (Write Errors) we deem it prudent to relocate the data once again. We have since run multiple filesystem checks ( filesystem checks on a relatively clean filesystem are much quicker) until no errors were present. We have mounted the filesystem in Read-Only mode and are copying the data to its new home. We estimate the email system should be available again by 3 A.M. We appreciate your patience during these trying times. The situation will improve.

The email system is back online as of approximately 2A.M. Since we are running from a copy from a damaged filesystem it is likely that some files (primarily those active during the crash) will be affected. Most likely those will be files in the queue waiting for local or remote delivery. So far it looks like those are bulk email and spam. We will be monitoring the logs as always to pick up on those files that are having problems. It is possible that your account could be affected without any errors showing up in the logs so please post here any problems that you are seeing with your accounts and we will remedy them ASAP.

I suspect that the precautions we have taken to copy the the mail data to another filesystem will result in a more reliable solution than trying to run off of the recovered filesystem. Though this resulted in much greater delays there should be a lower incidence of residual effects compared to previous recoveries.

Due to a required system upgrade, one of the Blackboard servers must be restarted at Midnight on Friday, 1/12. This server is also providing the storage for mail.usf.edu accounts, so we must bring the mailserver off-line while the upgrade is completed. Services will unavailable for about 30 minutes — 12:00AM-12:30AM. WebMail and POP/IMAP services will be unavailable during this period, but incoming messages will be delivered when the server is brought back on-line.

The mail server seems to be behaving pretty well since last Wedsnesday. We are still working with the software developer trying to figure out the cause of apparently random crashes happening here and there:

• Friday, 12/15, 10:23 am
• Friday, 12/15, 2:14 pm
• Sunday, 12/17, 12:40 pm
• Monday, 12/18, 0:55 am
• Tuesday, 12/19, 0:49 am

We now have an automated script that will detect when the server application goes down and restart if.

Some of the machines that handle the IMAP/POP caching have been acting up in the past few days. This did not affect most of the users (IMAP users) reading emails, but it seemed to have intermittent problems with folks using POP to check their messages, usually through client such as Outlook and others. We are looking into fixes today.

Overnight we transferred the remaining local accounts back to a shared drive. This will allow us to switch servers temporarily and apply firmware updates to the new Dell server without interruption of service. You may see a blip here and there but we are not expecting any extensive downtime at this point.

I think that’s it for now. More updates as they are needed.