More information regarding the problem is available here: http://engineyard.wordpress.com/2009/11/09/ey04-io-bandwidth-issue/.

I want to apologize for our down time but more so I want to make sure we’re learning from this experience and improving the service. More on that in a bit.

UPDATE: We’re working on some improvements to the feedback tab widget that will ensure that our service interruptions don’t affect your page loads. We hope to have these improvements out later on today but they will require you to change your embed code. You can vote for this issue here (http://uservoice.com/a/vHVF6) and you’ll be updated when it goes out (we’ll also blog/tweet about it as well when it’s ready).

Thanks for your patience,

Richard White, CEO

Full details of this issue are available in the email below which we received from EngineYard.

We apologize for any inconvenience. Further information will be available here: http://engineyard.wordpress.com.

Thank you for your understanding,

The UserVoice Team

Date: Thu, Oct 29, 2009 at 6:16 PM
Subject: Emergency Shelf Replacement

There will be emergency maintenance scheduled tomorrrow Friday Oct 30th at 11:00PM PDT. Our monitoring alerted us of a failed drive in one of the disk raidsets on the shelf. The shelf has an older backplane version that may cause a cascading array failure if we swap out the drive and rebuild the degraded raid array. To reduce the risk of data loss on this system the shelf must be swapped tonight and the affected raidsets parity rebuilt.

During the raidset rebuild which will be a few hours, I/O performance may be affected. During the swap, slices on the shelf will have their IO frozen and automatically resumed once the swap is complete. The frozen IO (outage) should last 15-30 minutes but we will set a maintenance window of 1 hour.

Regards,
Cluster Engineering

—

You may have noticed some quirkiness happening today.  We noticed some of this too.

We’re about to roll out some major changes at UserVoice that we think you’re going to be thrilled with – in preparation, we rolled out a major backend upgrade this weekend that hasn’t gone as smoothly as planned. Don’t worry, nothing major is broken — but you may see slower than normal response times and a formatting issue with some of the admin activity emails that went out this morning.

We’re actively working on these issues and hope to have resolution this evening. In the meantime we’ve suspended this afternoon’s email run. We usually try to reserve deployments until the evening (PST) but this is an important enough issue that we’ll push these fixes as soon as they’re tested.

We’ll keep you posted and I appreciate your patience.

Richard White, CEO UserVoice