Server Failure and Progress Report
I have the unfortunate task of having to report an unusual server outage that affected some clients today. Earlier we noticed some errors coming from one of our core monitoring servers. Shortly thereafter, the server failed to respond. We escalated the issue to our hosting provider who immediately began to triage the situation. Unfortunately, the diagnosis was not good.
It appears that at some point in the day the server, which runs on the Linux operating system, experienced a Kernel Panic. This is the equivalent of the “Blue Screen of Death” in the Microsoft world – otherwise known as a non-recoverable system error.
System crashes are a normal part of IT operations, but what is extraordinary in this case is that the system seemed to continue to operate partially during the crash, meaning that it went undetected for much of the day. Although the system was acting normal, it was not collecting data. Once the problem was isolated and the system restarted we learned of the missing stats for many of the customers on this server, which brings me to the point of this update.
Although the physical machine was restarted and is functioning properly at this time, frankly we no longer trust it. So in the next day or two we will be migrating the data to a new server as a precautionary measure. In addition to replacing this server, we were unnerved by the fact that it escaped notice for so long and we are working on new plans for earlier detection and greater redundancy.
Only a very small percentage of clients was affected, but if you are wondering if you were among them you can simply take a look at your stats for today and if you have any then you were not affected. If you see several hours of zero visitors when you would otherwise have expected them, then it’s likely your account was on the server in question. Please keep in mind that as we move the data in the near future there may be a period of up to an hour where the server will be unavailable; however, we will do our best to minimize any further downtime.
On behalf of the team, we apologize for any inconvenience this incident may have caused, and we wanted to let everyone know that we continue to refine and improve the system every single day.
Thank you for your patronage and your kind understanding as we work through these issues.
Sincerely,
John P.
@johnpoz
Thanks for the quick response, I was one of the unfortunate ones who lost data but just wanted to say Thanks for keeping me posted. It sucks to lose data but things happen. I am glad to be part of this blossoming app since beta and can’t wait to see what the future holds for new features.
is the server still down?
I’m still having trouble with my stats, I’ve had nothing for almost 17 hours and an error message from the wordpress plugin.
When tried to login to website:
504 Gateway Time-out
The server didn’t respond in time.
Boris, please keep an eye on http://twitter.com/wooprastatus/ as we posted some updates about being under maintenance while rolling out new features…
Cheers,
John P.
I only found out about Woopra a few weeks ago and installed it on my site. I am quite impressed by the level of data that Woopra provides. You may be aware of this already, but I think you are still having server problems as I have had 0 stats for the past 2 days.
You should have received an email from us, and this post describes what happened to impact only a few of our members. If you continue to have issues, please let us know in our forums so we can respond quickly to track down the issue.
I specifically checked your website and found it has many HTML and coding errors, and our tester did find Woopra installed but not active within the body of code. Your site is designed with tables, an archaic coding method. It takes only a tiny error to block the flow of search engine crawlers, and Woopra, from working smoothly with such sites. You might want to also check your code and fix it or update it, or see if something you added or changed recently might be interfering with Woopra’s ability to function. Thanks.
is the server still down
I cannot find your information listed in our database. There are no servers down or issues currently. If you could leave detailed information for me to help me find your account information (username and site) in the Woopra Forums, I can help you further. Thank you.