Today we were hit by a massive DDoS (distributed denial of service attack). We’ll be putting up a more detailed post soon – right now we’re all hands on deck trying to resume normal service for everyone. This affected all UserVoice admin consoles, customer-facing sites, and tickets. It affected our ability to respond to incoming tickets from our customers. Please note, though, that no data was lost or compromised.
The main attack should be over (but there may still be smaller ones), and customer tickets are still somewhat backlogged as of 3:45pm PST. While these things cannot be prevented, there might be steps which our service provider can take that would possibly reduce the impact of future attacks, and we are working with them on this.
We are extremely upset by this and very, very sorry that you have not been able to use our service today. We’re doing everything we can to get back to normal operation.
Please note that you can get up-to-date statuses by subscribing to the incident report on our status page.
Head of Customer Success
The attack has ended and all systems are back to normal including ticket emails which are no longer delayed.
As mentioned, yesterday we were hit by a DDoS attack (Distributed Denial of Service) attack. This is a brute force attack where the attacker simply takes down a service by overwhelming a service with traffic. It’s hard to defend against. DDoS attacks have taken down almost every major website at one point or another from Twitter to major banks. We don’t know why would someone target UserVoice for such an attack but our best guess is that they mistakenly thought we were actually one of our (probably larger) customers.
Our infrastructure provider provides DDoS protection for all of the UserVoice servers and that system did detect the attack yesterday morning. Unfortunately the attack was larger than their defensive systems were designed to handle so instead of simply filtering UserVoice traffic they moved to immediately block all traffic both good and bad. Had they not done this the attack would have taken down their switches and knocked other customers offline. We immediately started working with them and when the traffic has somewhat subsided we were able to switch to having our traffic “filtered” instead of being outright blocked. Once that happened most people were able to access our services though some might have been incorrectly marked as “bad” traffic and filtered.
Our email provider attempted to deliver emails to the UserVoice system during that time and, when it couldn’t, stored them to be delayed at a later time. The longer the downtime the longer Mailgun waits to try and deliver the mail which is why people were having emails from that morning trickle in hours later.
There’s no magic bullet for stopping DDoS attacks but we’ve already making a few changes in response to yesterdays unfortunate events:
We’ve lowered the TTL on our primary DNS entry. This means that next time there’s an attack we can more quickly shift all traffic to a new IP. The attackers will likely still attack the old IP while all legitimate users are forwarded to the new one.
We’ve changed the way in which our email provider sends emails to us so that we can be sure that they never get filtered out during an attack.
We’re investigating using additional 3rd party solutions to provide more robust DDoS protection than what our infrastructure service provides. (If you are one of those vendors please don’t call us, we’ll call you)
On behalf of the entire UserVoice team, thank you for your patience and understanding through what was a very frustrating day for everyone but especially you who depend on us to do your job. Keep calm and UserVoice on!