9 am October 20 update

I have new, updated info regarding email issues that I wanted to share with you. Essentially there are two points to make about missing emails. The first case is a situation that will impact ALL users.

(1) There is an unfortunate 21 hour window of time (from Wed 5am - Thu 2am) for which we physically do not possess emails for any user.  This is the window of time between the backup we used to restore users email and the time of the campus power outage Thu at 2am. In other words emails you may have seen, responded to, deleted, etc. during business hours on Wed and into the night and wee hours of Thu could not have been included on the Wed early morning backup and therefore can not be retrieved unless you happened to move them into an alternate folder before the power outage. Sadly, the Thu night backup was interrupted by the power outage.

In summary, all users will be impacted by the reality of this unfortunate timing and the fact that the power outage created a situation in which we lost the entirety of the original mailstore for all users and needed to restore from backup.

The second issue is more nuanced and impacts a much smaller group of users that we are working to identify and help work through the situation.

(2) Some users have identified a situation where they are missing large amounts of email from their INBOX or are missing some emails from Oct 14, 15, 16, 17 and/or 18th. Please contact me directly if you think this second issue applies to you.  I have already heard from a number of you and we are working on the following steps to lessen the impact of the (potential) data loss.

  1. RESTORE. We will do an individual mailstore restore on your account from two different backups. We have seen good results in adding more emails to a users mailstore from this process.
  2. TRANSACTION LOGS. We will provide you with the mail server transaction logs during the above mentioned time period that will allow you to know WHO sent you email so that you can contact them and kindly ask them to resend you any communications they may have sent you.
  3. TECH SUPPORT. Lastly, we continue to work with our technical support resources in the hopes that there are additional measures we can take to restore missing emails to users INBOX.

In summary, IF you believe you are missing significant amounts of email in your INBOX or between the dates of Oct 14 - 18 please contact me directly and we will help lead you thru the above steps to potentially retrieve email and identify those folks who emailed during this time frame.  If you have already contacted e you are on our list and we are already going thru the above process.

I cannot express how much it pains me to not be able to respond, react and remediate this problem with 100% effectiveness. I know some of you will have lost important emails or timely communications and I sincerely apologize for the impact to your communications. We will continue to work diligently on our users’ behalf.

Thanks for your understanding,
Bill

2:15 pm October 19 update

Thanks for your patience as we have worked to restore email services. At 10:30 pm Sunday night we finished applying our solution to the mailstore and have a very high percentage of success for individual email accounts. You should be receiving emails, sending emails and seeing all emails in your INBOX. IF you are still not sure if your email account is back to normal FIRST ensure that you have SUBSCRIBED to ALL you server folders in your email client, Thunderbird (see the thread below).

If you still have anomalies with your email account please email me directly as I am organizing the remediation efforts so that John can focus on individual issues that develop. I have heard from a handful of you -  we are actively working to resolve the issues.

Thanks for your patience and THANKS to John who has been working continuously on this issue!

I will continue to update the community via email and will continue to post updates on the NEWS section of the home page.

Bill

12:15 pm October 18 update

We wanted to let you know that John has been working tirelessly with the experts on a solution to resolve both email related issues that users have been experiencing. John has done multiple test implementations of our current expected solution and we have had great results from those tests.

We are now working on an implementation plan to apply this solution to ALL email users. We are waiting on some final confirmations from our expert support resource before we commit to a plan.

Please note, that in the next few hours we WILL begin putting this solution in place and it MAY involve limiting access to email for a few more hours while we restore your email service to normal operation. We will continue to POST updates to the News section of the website.

If you have concerns or questions feel free to email me. Thanks for your patience as we restore your email service

For those concerned about accessing the *Admin Fileshares* we have also created a plan to restore that service first thing Mon AM. We modified our original plan to allow us to more quickly get this service operational first thing.

Bill Doering

4:45 pm October 16 update

We are stil investigating the email issues, but at this time you can reliably send and the system will receive email. The problem appears to be that while we are receiving and putting emails in email inbox folders, users are not necessarily able to see all of those emails. We wil be working over the weekend to resolve the issue.

12:38 pm October 16 update

As for people within the school, here's the latest:

Most all services were backup on Thurs except for email.

Email services are currently up and mail is being sent and received.  I suggest if you have any critical email to send that you do so immediately as their are still concerns with the email service and we will need to bring the system down again to remedy them.

Email Issues:
- Users may see a server INBOX that holds some or no old emails - this is the primary problem we are working on now.
- Users may not see Folder(s) under their server INBOX but, this can be easily fixed by "subscribing" to those folders

After clicking subscribe a dialog box will open up. Make the window as long as necessary to see all the folders.  Select ALL the folders. Then click  the button "Subscribe" on the right hand side. Now click "OK" and you should see all your INBOX folders and the emails within each.

Please do NOT send emails to HELP during this time. We would appreciate it if you would call (x8031) or try to stop by (ED 4203) to get HELP.