Unix Systems Group

As of 3A.M. the mail server was brought back online. All mail waiting in the queue has been delivered. Some users are experiencing problems with corrupt files. Most of these are showing up in the log but to be sure post here if you encounter any problems with your account.

Both services have been affected. We are working to restore them. More details within the hour.

Update: my.usf is online again. The filesystem for the mail services is still being checked. It will be 10+ hours before that filesystem is available again. We are going to attempt a restore elsewhere as that may be quicker.

ETA: 3A.M. The filesystem check completed and when the system was brought online it immediately crashed. Due to the nature of the crash (Write Errors) we deem it prudent to relocate the data once again. We have since run multiple filesystem checks ( filesystem checks on a relatively clean filesystem are much quicker) until no errors were present. We have mounted the filesystem in Read-Only mode and are copying the data to its new home. We estimate the email system should be available again by 3 A.M. We appreciate your patience during these trying times. The situation will improve.

The email system is back online as of approximately 2A.M. Since we are running from a copy from a damaged filesystem it is likely that some files (primarily those active during the crash) will be affected. Most likely those will be files in the queue waiting for local or remote delivery. So far it looks like those are bulk email and spam. We will be monitoring the logs as always to pick up on those files that are having problems. It is possible that your account could be affected without any errors showing up in the logs so please post here any problems that you are seeing with your accounts and we will remedy them ASAP.

I suspect that the precautions we have taken to copy the the mail data to another filesystem will result in a more reliable solution than trying to run off of the recovered filesystem. Though this resulted in much greater delays there should be a lower incidence of residual effects compared to previous recoveries.

Due to a required system upgrade, one of the Blackboard servers must be restarted at Midnight on Friday, 1/12. This server is also providing the storage for mail.usf.edu accounts, so we must bring the mailserver off-line while the upgrade is completed. Services will unavailable for about 30 minutes — 12:00AM-12:30AM. WebMail and POP/IMAP services will be unavailable during this period, but incoming messages will be delivered when the server is brought back on-line.