Just think about what it would mean if your website, application, or server was down for a minute. How about 5 minutes? An hour? A day? A week?? Insurance is for people who have something to lose. If your website being offline won’t affect anyone substantially, you don’t need to have a very complicated or expensive redundancy plan in place. But you probably wouldn’t be reading this blog, either.
Your business may be large enough that you know right away, without thinking, that “Oh, we can’t afford to be down at all, ever. Whatever the cost, our site has to be up or we lose trust and revenue.”
From an economics side, I can tell you right now that more of you think that is true of your business than is actually the case. No, I’m not trying to be mean; just realistic. I understand the trust factor that’s lost when your system isn’t 100% reliable, but people are more forgiving, on a whole, than we are given credit for. And trust can be maintained with good communication when your site is down.
Even with that comment, it’s still true for some of you that reliability is worth whatever the cost. Such cases include client contracts where you have a minimum service level agreement, which failing to meet could result in the loss of a very lucrative contract. In those cases, you should absolutely spend whatever it takes.
But between those extremes of “whatever the cost” and “redundancy is worthless to me”, there is a whole lot of territory. And you don’t have to (and shouldn’t) be haphazard with what you do with that middle ground as your business grows.
Here’s how to calculate redundancy’s worth to you:
1) Figure out your profit per day from your site / application.
2) Divide your profit per day by 24 (for 24 hours).
3) Divide this number by 60. This is your profit per minute.
These 2 numbers are your first stab at an estimate of your “lowest average downtime expense”, but they still may not tell the full picture. So…
4) Figure out (or estimate, if you’re planning a new business) your peak and valley times, in terms of traffic and revenue
5) Figure out what your peak hour profit is.
6) Divide the peak hour profit by 60 (for 60 minutes)
Now you have the “highest average downtime expense” figure. You’ll have to look at the lowest and highest average numbers you just calculated and figure out how you want to process them. But if your per minute numbers are more than $5-10, then you need to have at least some sort of basic “hot” (fully operational, ready to come online at a moment’s notice) redundant system in place, to stand in for your primary system while you fix it and bring it back online.
The higher those figures, the more you should be willing to spend - because that’s the lost revenue you’ll have for every minute your site is inaccessible.
Demonstration
To demonstrate, let’s apply this calculation to Apple’s iTunes Music Store, since we have some good real-life estimates to work with:
1) Profit per day: 3,000,000 songs per day x $0.25 estimated profit per song = $750,000
2) Profit per hour: $750,000 ÷ 24 = $31,250
3) Profit per minute: $520.83
Since we don’t have any information readily available about the iTunes Music Store’s peak hours, we’ll just stop here.
So for every minute the iTunes Music Store is down, it costs them a minimum average of $521. If it would take 30 minutes to restore a failed server / switch / internet connection, that means they should spend at *least* $30,000 to make sure they have a completely operational server cluster in another location, so if anything happens to their primary one, they can switch over.
(This is an extreme example used to illustrate a principle. It’s worth saying that even before you reach this level, there are multiple options available worth considering that result in a more complicated and cost-effective solution than simply having a duplicate data center in a 2nd, 3rd, or 4th location.)
How much should you spend on redundancy plans? Well, how much insurance do you want? In my case, 2 weeks down in 2004 due to a non-existent backup plan (story here) resulted in an immediate direct cost of $4,000 and ultimately the demise of a $100,000/yr business. Knowing what I know now, a few thousand dollars’ worth of redundancy “insurance” would have been a very wise investment.