Anthony, E-Reads' Technical Director, conducted the greatest migration since the Ice Age

My phone console has a number of speed-dial buttons. There is RC Phone Home, of course. There’s one for our foreign sub-agent with whom I talk daily, and there are those for frequently called clients. There are intercom buttons for buzzing staff. And then there’s the Anthony Button. Anthony is E-Reads’ technical director. The button for his station is bright red. On a Friday morning in December I hit the Anthony Button. Hard.

As that day dawned I noticed that I had not received emails for eight hours. Refreshing and other tried-and-true techniques for goosing the get-mail function availed nothing. Then I clicked on the E-Reads home page. Some of it came in, but where the banner should have been was an error message.

I leaned all my weight on the hot button: ANTHONY! PICK UP!

Anthony had already seen the outage and analyzed it.  “Our cloud server is running on fumes.  We’ve loaded so many files recently that it’s maxed out.  We have to migrate our files to a larger server.”

“Why didn’t you tell me this was happening?” I asked, logically.

“I did,” he reminded me.

And he had.  But a move from the 60 gigabyte capacity of our current server to the 300 gig one we needed was a jump of four or five times the cost and I had dragged my heels.  To make room for more uploads Anthony had trimmed a bunch of junk files but it was hard to tell exactly how close the meter was hovering over Empty.

Once the system went down he didn’t wait around to determine if I was being penny wise and pound foolish (that was now a given). He immediately committed us to the larger server. But the changeover is not implemented with the snap of one’s fingers. Hundreds of thousands of files great and small, text files and jpegs, excel spreadsheets and metadata folders plus backups had to be migrated from computer A to computer B. But even with supercomputers running at peak speed, the transfer would take a minimum of 72 hours. In fact it took 96. Down so long it looked like up to me.

Meanwhile, though our resourceful technical director managed to rescue enough gigabytes to enable us to use our email, our website had to be taken offline.

If you’re looking for a definition of helplessness, try gazing at a “Sorry” message on one’s own website for 24, 48, 72, 96 hours. Though I was assured we had triple backup redundancy, excluding the portable hard drives I ferry once a week between home and office, the paranoid terror of a permanent failure haunted my dreams for four consecutive nights.

On the fifth day it came back, and it was good. Thank you, Anthony.

Wretched though this incident was, the bright side is that it was the result of a company that is growing – growing at a rate of of about 40 gigabytes a year.

Watch this space for the announcement that we’ve topped one terabyte.

Richard Curtis