ATP’s Disappearance: No, We Were NOT Blocked (or hacked). Not Yet.

Adil Najam and Owais Mughal

Many of you have been writing to us today about the”disappearance” of the ATP page for nearly 13 hours.

We apologize for this inconvenience – believe us, we were even more concerned about this sudden “disappearance” than you were. It was caused because of technical problems at our server host, mediatemple, and not because of mischief on the part of government agencies or of hackers.

The government of the day in Pakistan should rightly be blamed for many things, but (despite the fears of some of our readers) cannot be blamed for this!

What is interesting, however, is that the dozens of messages we received (and even our own initial impulse) assumed that ATP had, in fact, been either blocked by the ‘authorities’ or hacked. This is important and worthy of note because we are either so very cynical as a people as to always assume the very worst, or because the probability of this happening is actually so very high that this becomes not only a probable but the likely reason for the inaccessibility of the site.

Our own sense of things is that it is both. We are, in fact, a fairly cynical bunch, quite prone to conspiracy theories (and the more fanciful it is the more we fall for it). But it is also true that a whole string of actions (many of which we have written about here, here, here, and here) by the omnipresent authorities would make this a fairly reasonable assumption to make. This is quite sad. On both counts. It is an unfortunate commentary not only on our state but also on our society. Maybe we should take a few seconds to think about why this is so.

While you do so, for those of you more technically inclined, according to mediatemple the original task was to be done between Midnight and 6AM, Saturday, Dec (USA Eastern Standard Time) and it was to:

facilitate proactive replacement of certain electrical systems in one of our Facility Power Segments at our EL-DC3 data center. In addition, an upgrade will be taking place to core components of the (gs), including the storage subsystem. This vendor-recommended upgrade required additional time and thus has been grouped with the data center activity in order to reduce customer impact.

This larger than normal maintenance period is a proactive measure to prevent power failure incidents as experienced by various other data centers mentioned in recent news. We would like to remind all customers that scheduled infrastructure maintenance and security related updates are a necessary and vital aspect of web hosting that ensures the long term uptime and reliability of your services.

Later messages from to mediatemple reported that the task was taking much longer than anticipated and, at around 1:13PM (EST) a message from mediatemple reported the following:

Given a failed upgrade from the vendor on (gs) Grid.Cluster.2’s storage segment, and then a failed rollback attempt, we either had to work to repair the systems so customers would have their “live” data — or recover from backup, potentially taking several days to get fully back online and rolling back some customers to their “last backed up” date. Though neither option was “pleasant”, we had good confidence that the “up to date” data was safe and accessible with the appropriate vendor involvement — this was seen as the best overall customer outcome. And so after several hours of troubleshooting, we have have managed to repair the systems preventing a restore situation.

We are continuing to validate the auxillary systems of the (gs) — database, database container, containers — at this time. All basic services such as web, email, and FTP have been restored.

We understand how frustrating situations like these can be and we sincerely appreciate your continued patience and understanding.

And, yes, the situation was frustrating. Very frustrating. Our ‘patience and understanding’ was also being stretched; especially because those managing the mediatemple phonelines were not always helpful. Our feelings about the company have been mixed. On the one hand some of their technical services (e.g., design of backend) is quite spectacular. On the other hand their customer service is a case study in how not to deal with customers and is singularly bad. We have been and continue to think about moving elsewhere. We have not yet decided whether we will, but this incident just might push us over the edge. So, there, now we have vented!

All that remains is for us to repeat to you what mediatemple said to us:

We understand how frustrating situations like these can be and we sincerely appreciate your continued patience and understanding.