Table of Contents
The web hosting industry is increasingly entering a truly unfortunate “profit center” trend. What was once universally standard features have become optional add - on services. Security and backups appear to be most commonly affected. Let’s discuss backups.
There is a term used in finance called “lender of last resort” Wikipedia
In public finance, a lender of last resort (LOLR) is the institution in a financial system that acts as the provider of liquidity to a financial institution which finds itself unable to obtain sufficient liquidity in the interbank lending market when other facilities or such sources have been exhausted.
Service providers face worst case risks every day from security breaches to disasters involving natural and unnatural hazards. These risks can be trivialised for lack of comprehension because the specific event is not of great relevance. The outcome is always the same: complete loss of data.
Customers entrust providers to store their data and it should be expected that most will never download a backup. There is a common misconception that backups are entirely the responsibility of customers but this is not fair. An example to illustrate would involve customer data silently becoming corrupt. Assuming the customer was regularly storing their own backups, is it their fault when this leads to catastrophe?
At an absolute minimum, a backup implementation should involve regular restore points delivered offsite. The term “remote” is often used by providers but it does not have the same meaning. All offsite backups are remote but not all remote backups are offsite. Remote backups are typically housed in the same facility but on different hardware, whereas offsite backups are stored in different geography.
Backups of last resort can be held in long term cold storage. These backups are extremely low cost to maintain but especially expensive to retrieve, which is why they should only ever be used in complete worst case scenarios. Any business should be capable to afford this measure with example below there are 10 000 customer accounts each of size 10GiB with AWS Glacier Deep Archive:
- Total Storage: 98TiB
- Archive Average Object Size: 10GiB (single file archives)
- Request Count: 10 000
Total Monthly cost: $99.85
Peace of mind for 10 000 customers for approximately 1 cent per customer, per month. As shown, it is unbelievably cheap to have a backup of last resort even for established businesses. Try yourself using the AWS Cost CalculatorThe business also indirectly benefits with improved security compliance, a factor which is usually overlooked.
Of course a last resort restore requires that transferred to standard s3 and retrieval will cost thousands of Dollars. Is that not worth it to avoid so many people losing a part of themselves? Even if the business will cease to exist, those affected still deserve their data.
Such an event should be survivable for the business with if there is a culture of automation. It would need time but all lost servers can be rebuilt using Ansible with data restored from last resort.
If you are an engineer reading this and presenting similar solutions, do keep in mind that data might not be regarded sacred by others. I have personally seen discussions with outlined example worst case scenarios immediately dismissed because they seemed an impossibility without understanding the cause is irrelevant but the outcome is always the same. As an engineer, my sympathy is with you.
Consider this a reminder to safely store a backup because your provider might not regard your data of great importance.