Unix Systems Group

The retrieval processing is still in progress. We are doing this in two ways: there’s an automated process running 24 hours/day since Monday, restoring files from a list of users we generated. In addition to that, we will manually restore the files from anyone who has been posting to the blogs. If you have problems accessing your email, please first refer to the FAQ. If you still cannot access it, simply post here and someone will be in contact with you.

Alex Campoe

Frequently Asked Questions

1. Will I lose my email?
As of right now we have no reason to believe we will not be able to restore the majority of emails you may have had stored. Some may be missing due to the nature of the backup process. Backups are a snapshot in time: if a particular message was in your INBOX when the snapshot was taken, it will be retrieved. Once the program we wrote finishes retrieving data we will let you know, and we will start dealing with user issues manually.

2. When should I start worrying?
All users should now be able to log in to the system. If you login and don’t see anything, check the answer to one of the FAQ below. We are still retrieving data but this is a slow process - we are only about 20% done as of 12/27 morning. We will post a note when the process is finished.

3. Is there anything I can do?
We are using the blog comments to monitor issues. If you are still having problems, post your problem to this blog entry. Please do not forget to let us know your @mail.usf.edu address. Some folks are posting using off campus email addresses (which I understand) but we can’t track anything down if we simply know you as “bigmouthtoad@aol.com”. Also just as important: once your problem is resolved, let us know.

4. What about emails sent to me since the system went down on the 22nd?
That is up to the configuration of the server that sent the note. Most servers are setup to retry for anywhere from 3 to 5 days. If it is unsucessful after that period, the user sending the note should receive a note stating that the server was unable to send the message.
Update: Any messages sent to you during the outage should have reached you by Friday, if it was resent by the sending server. If you were expecting a message during that time and you have not received it I would strongly suggest you contact the sending party and request a resend.

5. My INBOX is back but I don’t see anything I had saved on my folders. What do I do?
Go to https://mailbox.acomp.usf.edu and select the Folders option after you login. You may have to re-subscribe to your folders after we have restored them.

All of the people who could not previously login to the system now should be able to login. Backups are still being restored and probably will continue for the next couple of days. We are still looking into ways in which we can speed up the restore process. The difficulty is that we have backups to perform (both Blackboard and the Help Desk Online software have been upgraded over the past couple of days) and we only have a limited number of tape drives to work with.

All people who posted to the threads are being contacted for a follow up. If we can manually resolve any issues we will do so.

A reminder: some of the folks posting are claiming significant loss of money due to the lack of service. I strongly suggest these users refresh their memory and read the Acceptable Use Policy on UNA

Thanks,
Alex Campoe

If you are reading this, you are probably one of the users who are still having issues either logging in the system, or you have missing mail files (INBOX, mail folders, etc). Here’s the situation:

1. The standard way to restore files from backup tapes was not being efficient enough. At the speed things were running, it would take us 9 days to restore all the files. Eric Pierce has worked out a different way to restore these files. We manually tested this approach on a few accounts and are now ready to do them automatically.
2. We have also determined the reason some folks were not able to login at all. We are also ready to initiate the correction automatically.
3. We wrote a program to fix both issues outlined on (1) and (2). This program will start running some time this morning.

The files recovered by (3) will be named in the format “INBOX-recovered.” You will be able to see them as one of your subscribed mailboxes.

Thanks
Alex Campoe

Update (12:24pm):The first stage of the script, checking which accounts have login issues, has processed 15,000 accounts. Of these 15,000, approximately 200 have to be fixed manually — the others have been recovered successfully. We also restored more mailboxes manually. It may be nessessary for you to re-subscribe to those folders we are restoring.On mailbox.acomp.usf.edu, that is done in the Folders menu.

Update (5:00 pm):Added to FAQ

Update (12/27, 9:30 am):Moved FAQ to its own section

Eric Pierce has recovered the hardware from a catastrophic failure on 12/22. We had to retrieve backed up data from tape. Unfortunately, the tape recovery is a long processes. In order to expedite things, we did an initial, partial, recovery of a data set which included about 65,000 or the 70,000 or so accounts we maintain. That took 2 days. Eric then sent a test message to all accounts in order to pinpoint which accounts still have issues. We now started working on retrieving those accounts.

The bottom line:If you are having problems login in to the system, chances are your account is one of the many we are still retrieving. We are working as quickly as we can to restore access to all accounts but we are constrained by the speed of the recovery from the tapes. Once we believe the process is done we will post a follow up, asking those who still may be having problems to contact us directly. If I am pressed for a time frame right now I would say we will probably be done with the restores by Tuesday morning, but keep an eye on this space for further info.

Again, our apologies for the situation.

There is good news and bad news for users of mail.usf.edu:

Good News

• The server is back up and you can send & receive Email again!
• Any mail sent to your mail.usf.edu address during the outage should be delivered to your account in the next few hours.

Bad News

• You may be missing some mail. To expedite the recovery process, which had already taken WAY too long, I recovered the mailboxes from the FULL backup on 12/18/2005. The data from the INCREMENTAL backup which took place on 12/22/05 will be recovered, but it may be a few days before you see the new data in your account. It is not possible to recover any mail or changes to your account that were made after the backup took place on 12/22/05 (it finished around 6AM).

We’re working on getting on your Email restored as fast as possible. Thank you for your patience.

The hard-drive array supporting mail.usf.edu failed just after 2PM on 12/22. The failure has been dealt with, but user mailboxes must be recovered from backup tape. The system will be down until noon on 12/23 while accounts are restored. We apologize for this inconvenience and are working to restore access as fast as possible.

UPDATE: The recovery from tape is taking longer than expected. We will post updates here as they become available.

The campus chilled water supply will be interrupted for 24 hours starting on December 22 at 3:45PM. The air conditioners at USF use chilled water for cooling, so this means that all A/C will be off-line until chilled water is restored. The Blackboard servers will be down during this period. (details here), but we will attempt to keep all other student services provided by Academic Computing (Email, webpages, blogs) running during the outage. These services will also be shut down if they are in danger of over-heating.