Contents

Member Sign in

admin

Data Center Updates

Support.DataCenterUpdates History

Hide minor edits - Show changes to markup

Changed line 4 from:

Update 1, 6:30pm: The web server has been restarted and is being tested.

to:

Update 1, 6:30pm: The web server has been restarted and is being tested.\\

June 25, 2010, at 10:33 AM EST by Dave Marney - Update: Tests complete
Changed lines 1-4 from:

June 24, 2010 4:30pm

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and the system is operating normally.

March 13, 2010 10:00pm

to:

June 24, 2010 4:30pm

We experienced an unplanned shutdown of our primary web server. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours.

Update 1, 6:30pm: The web server has been restarted and is being tested. Update 2, 7:10pm: Tests have passed, and the system is operating normally.


March 13, 2010 10:00pm

Changed lines 12-14 from:

Update: March 14, 2010 9:54am The changes have shown up in ISPs on the east coast of the US.

August 27, 2009 11:33am

to:

Update 1, March 14, 9:54am: The changes have shown up in ISPs on the east coast of the US.


August 27, 2009 11:33am

Changed lines 19-21 from:

August 8, 7:39am

to:

August 8, 2009 7:39am

Changed lines 24-26 from:

July 23, 3:09pm

to:

July 23, 2009 3:09pm

Changed lines 31-33 from:

January 23, 2009 3:09pm

to:

January 23, 2009 3:09pm

June 24, 2010, at 11:28 PM EST by Dave - Update: tests passed
Changed lines 2-3 from:

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and we are currently testing the system.

to:

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and the system is operating normally.

June 24, 2010, at 10:42 PM EST by Dave - Update: system test required
Changed lines 2-3 from:

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and everything is operating normally.

to:

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and we are currently testing the system.

June 24, 2010, at 10:40 PM EST by Dave - Added June 24 notice
Added lines 1-3:

June 24, 2010 4:30pm

We experienced an unplanned shutdown of our primary web server due to human error. The primary sermonsonline.com web site, the Sermon Site Manager, and sermon content views could not be displayed for several hours. The web server has been restarted and everything is operating normally.

March 14, 2010, at 01:56 PM EST by Dave Marney - Added domain name alert
Changed lines 1-6 from:

August 27, 11:33am

to:

March 13, 2010 10:00pm

We experienced an outage that affected the routing of the doman name, "sermonsonline.com". We have corrected the issue, but it will take some time for the changes to propagate to all the Internet Service Providers. Typically, this type of change takes 6-12 hours to complete within North America, but the timing is highly variable.

Update: March 14, 2010 9:54am The changes have shown up in ISPs on the east coast of the US.

August 27, 2009 11:33am

September 02, 2009, at 06:33 PM EST by Dave Marney - Updated media2 entry
Changed lines 2-3 from:

We are experiencing problems with one of our media stores that is affecting downloads of sermons published between October, 2006 and June, 2007. *UPDATE* We are restoring files from backup. It is estimated that the restore process will be completed by Wednesday, 9/2/09.

to:

We experienced problems with one of our media stores that affected downloads of sermons published between October, 2006 and June, 2007. The affected files were restored from backup and the system is now operating normally.

September 01, 2009, at 10:55 PM EST by Dave Marney - Updated media2 backup/restore issue
Changed lines 2-3 from:

We are experiencing problems with one of our media stores that is affecting downloads of sermons published between October, 2006 and June, 2007. We are investigating the problem, and do not yet have an estimate of when the problem will be resolved.

to:

We are experiencing problems with one of our media stores that is affecting downloads of sermons published between October, 2006 and June, 2007. *UPDATE* We are restoring files from backup. It is estimated that the restore process will be completed by Wednesday, 9/2/09.

August 27, 2009, at 03:42 PM EST by Dave Marney - Added note on media2 downtime
Added lines 1-3:

August 27, 11:33am

We are experiencing problems with one of our media stores that is affecting downloads of sermons published between October, 2006 and June, 2007. We are investigating the problem, and do not yet have an estimate of when the problem will be resolved.

Changed lines 2-3 from:

We have received reports that the Sermon Publisher client will fail on upload if the length of time needed to encode a multimedia file is longer than 15 minutes. We have identified the cause of this error, and have made corrections on the server side to correct it. We have contacted the customers who have been affected by this problem, and are monitoring the upload logs daily to make sure the problem does not recurr.

to:

We have received reports that the Sermon Publisher client will fail on upload if the length of time needed to encode a multimedia file is longer than 15 minutes. We have identified the cause of this error, and have made changes on the server side to correct it. We have contacted the customers who have been affected by this problem, and are monitoring the upload logs daily to make sure the problem does not recurr.

August 08, 2009, at 11:41 AM EST by Dave Marney - Added note regarding RequestTimeTooSkewed
Added lines 1-3:

August 8, 7:39am

We have received reports that the Sermon Publisher client will fail on upload if the length of time needed to encode a multimedia file is longer than 15 minutes. We have identified the cause of this error, and have made corrections on the server side to correct it. We have contacted the customers who have been affected by this problem, and are monitoring the upload logs daily to make sure the problem does not recurr.

July 27, 2009, at 10:56 AM EST by Dave Marney - Added alert re:new media server
Added lines 1-5:

July 23, 3:09pm

We added a new media server last week which uses digital signatures to authorize uploads. A few customers have reported problems with the signing process, which has interrupted their ability to add new sermons. We believe that we have identified all of these issues and resolved them.

During the process of adding the new media server, some domain name routing information was not transferred properly, and customers with custom sermonsonline.com subdomains were directed to the wrong server. We have since corrected the routing, and customers should see a resumption of normal operation over the next 24 hours.

January 23, 2009, at 08:17 PM EST by Dave Marney - Added information on server configuration
Changed lines 1-41 from:

June 24, 7:00 am

On Friday, June 20, our hosting provider completed the cut-over to permanent utility power, and is operating in normal mode, backed by a permanent UPS and generator power infrastructure.


June 4, 7:18 am

All servers are up, and are running on backup power generators.

The power distribution equipment that was destroyed is being rebuilt, and should be installed by this weekend. When we switch back to the normal power, there will be another service outage of 4-6 hours. No final schedule on when that outage will be, but will likely be done in the wee hours of Sunday or Monday morning.

We have begun switching back to our original web site from our read-only backup web site. It will take some time for the domain name changes to be updated, but most people should be able to access the original web site during today and start uploading sermons again.


June 3, 7:32 pm

Within the last hour, our hosting provider brought a second power backup generator into operation, and we now have power restored to all Sermons Online servers.

During the day, we restored a backup of our web server to a new server, and changed our domain name records to point to this new location. It can take some time for domain name changes to be completely updated across the Internet, but most people in North America should be able to view and playback your sermons now.

While we switch back again to our original server, Sermons Online is in "read only" mode. Any content you uploaded as of June 2 at 5:45 PM CST will be available, but you will not be able to add any new sermons or update any records. Because of the time delay in updating domain name records, it will probably take another 1-3 days before you can start uploading sermons again.

We will be issuing some type of credit for the loss of service, but we will get to that once we have the service fully restored.


June 3, 9:09 am

Our hosting provider was able to restore power yesterday afternoon, but their backup power generators have not been able to sustain power reliably. As a result, one of Sermons Online's servers lost power again, and unhappily it was our main web server. Our hosting provider is repairing the backup generator, but we have not yet received an estimated time when that will be done.

We are receiving updates on the half-hour, and expect to hear from them soon on when they will have the power to our main web server restored again.

We learned more about the nature of the power loss last night. There was an explosion and fire in the underground power cable which feeds the data center, and this destroyed the main power distribution panel. No one was hurt, but the fire marshal would not allow the backup generators to be turned on until it was safe. The damage to the power distribution equipment has severely hampered their ability to get the backup generators working reliably.

This is an exceedingly rare event, and our hosting provider is making what appears to be a truly heroic effort to get things restored.


June 2, 4:13 pm

The power to our data center has been restored, and Sermons Online is back online. Thank you for your patience!


June 2, 9:14 am

Over the weekend, the data center where Sermons Online's servers are located experienced a severe loss of power. No sermons have been damaged or lost, and our hosting provider is working around the clock to restore power. They expect to restore partial service by this afternoon, and full service by tomorrow evening.

We are monitoring this situation closely, and will keep you updated on their progress.

to:

January 23, 2009 3:09pm

Over the last two weeks, our primary web server has been taking an unusually long time to display pages. Although this has not affected playback of sermon multimedia, it has affected the display of sermon views and the Sermon Publisher.

We traced the problem back to a configuration problem, and corrected it. The server is now operating normally, and we are monitoring it daily to make sure that performance is acceptable.

Changed lines 2-3 from:

On Friday, June 20, our hosting provider completed the cut-over to permanent utility power, and is operating in normal mode, backed by our permanent UPS and generator power infrastructure.

to:

On Friday, June 20, our hosting provider completed the cut-over to permanent utility power, and is operating in normal mode, backed by a permanent UPS and generator power infrastructure.

Deleted line 4:
Added lines 4-5:

June 24, 2008, at 11:02 AM EST by Dave Marney - Added announcement regarding permanent power cut-over
Added lines 1-3:

June 24, 7:00 am

On Friday, June 20, our hosting provider completed the cut-over to permanent utility power, and is operating in normal mode, backed by our permanent UPS and generator power infrastructure.

June 04, 2008, at 11:47 AM EST by Dave Marney - Created page
Added lines 1-37:

June 4, 7:18 am

All servers are up, and are running on backup power generators.

The power distribution equipment that was destroyed is being rebuilt, and should be installed by this weekend. When we switch back to the normal power, there will be another service outage of 4-6 hours. No final schedule on when that outage will be, but will likely be done in the wee hours of Sunday or Monday morning.

We have begun switching back to our original web site from our read-only backup web site. It will take some time for the domain name changes to be updated, but most people should be able to access the original web site during today and start uploading sermons again.


June 3, 7:32 pm

Within the last hour, our hosting provider brought a second power backup generator into operation, and we now have power restored to all Sermons Online servers.

During the day, we restored a backup of our web server to a new server, and changed our domain name records to point to this new location. It can take some time for domain name changes to be completely updated across the Internet, but most people in North America should be able to view and playback your sermons now.

While we switch back again to our original server, Sermons Online is in "read only" mode. Any content you uploaded as of June 2 at 5:45 PM CST will be available, but you will not be able to add any new sermons or update any records. Because of the time delay in updating domain name records, it will probably take another 1-3 days before you can start uploading sermons again.

We will be issuing some type of credit for the loss of service, but we will get to that once we have the service fully restored.


June 3, 9:09 am

Our hosting provider was able to restore power yesterday afternoon, but their backup power generators have not been able to sustain power reliably. As a result, one of Sermons Online's servers lost power again, and unhappily it was our main web server. Our hosting provider is repairing the backup generator, but we have not yet received an estimated time when that will be done.

We are receiving updates on the half-hour, and expect to hear from them soon on when they will have the power to our main web server restored again.

We learned more about the nature of the power loss last night. There was an explosion and fire in the underground power cable which feeds the data center, and this destroyed the main power distribution panel. No one was hurt, but the fire marshal would not allow the backup generators to be turned on until it was safe. The damage to the power distribution equipment has severely hampered their ability to get the backup generators working reliably.

This is an exceedingly rare event, and our hosting provider is making what appears to be a truly heroic effort to get things restored.


June 2, 4:13 pm

The power to our data center has been restored, and Sermons Online is back online. Thank you for your patience!


June 2, 9:14 am

Over the weekend, the data center where Sermons Online's servers are located experienced a severe loss of power. No sermons have been damaged or lost, and our hosting provider is working around the clock to restore power. They expect to restore partial service by this afternoon, and full service by tomorrow evening.

We are monitoring this situation closely, and will keep you updated on their progress.