Immutability has become such a buzzword due to the sudden increase in ransomware attacks. People are worried – and rightly so – that a ransomware infection might also infect and encrypt their backups as well. Many ransomware stories mention this happening to the victim, forcing them to pay the ransom. It’s important to understand, however, that backup immutability is about more than just ransomware attacks – so much more.
I say this because many vendors' immutability features address only the ransomware risk while ignoring the others I will discuss in this article. One such vendor claims they are the only backup vendor with “true immutability,” because they protect against ransomware infection. This ignores the many other ways their customers’ backups can be damaged. In this blog post, I will give an overview of all of the ways your backups can be corrupted or deleted, explaining how immutability is a much bigger need than you might think.
Immutable should be a binary attribute
What is immutable storage? Immutable simply means that something cannot be changed. It should be a binary attribute, like alive or dead; you’re either immutable or not. Truly immutable backups cannot be deleted, changed, or corrupted by either an unauthorized person or even an authorized person. By contrast, most backup companies that claim immutability – including the aforementioned vendor – have backups that can be deleted by an authorized person, or someone masquerading as authorized.
Historically, if something claimed to be immutable, it meant it was some kind of write-once-read-many (WORM) media, such as WORM tape or optical (e.g. CD-R/DVD-R). A few vendors also developed WORM disk arrays. If you write something to WORM media, it is there forever – even if you change your mind. It was immutable; it could not be changed or deleted even by an authorized person. (That last part is key, as it even protects from a rogue administrator).
Backup immutability is really a spectrum
Sadly, even in the good old days of immutability, if someone can put their hands on your “immutable” storage – its immutability won’t last very long. Set fire to a WORM storage array. Pass a WORM tape across a big enough magnet. Run a WORM DVD through a shredder. Gain admin access to the hypervisor and delete the VMs containing your immutable file system. The “immutable” data on those devices will be gone.
That means that even the best immutable storage devices throughout the years weren’t 100 percent immutable. Nothing is truly immutable; therefore, no vendor can claim to be the only backup vendor with 100 percent immutable backups, since such a thing does not exist.
As much as the grammarian in me hates to say so, some methods of storing data are more immutable than others; what should be a binary attribute is actually a spectrum. A WORM tape stored at a commercial off-site vaulting vendor like Iron Mountain is the most immutable option I can think of. You would need to thwart several levels of physical security to gain access to that tape in order to damage it. On the opposite end of the immutability spectrum, I’ll put MacOS Time Machine backups with the external hard drive still attached. A few mouse clicks and all your backups are gone. All backups fall somewhere between these two extremes.
Risks to your backups
There are essentially only two bad things that can happen to your backup files: corruption or premature deletion. Corruption can happen via encryption, or any process that changes the content of the backup file. Premature deletion can take place within the backup system, or in the operating system of the backup server. This means these two ways your backups can be damaged actually turn into four risks to your backup data.
Encrypting your backups
This is the chief concern of many customers right now, and rightly so. They worry that their backup server might become infected with ransomware, and that ransomware would encrypt their backups. Such a customer would have no choice but to pay the ransom (e.g. gpg -c backup.bak).
Deleting backups outside the backup interface
Modern backup systems typically store their backups on some kind of disk system, and the backup files are typically visible as ordinary files in a filesystem. Those files can become damaged like any other files, primarily by someone accidentally or maliciously deleting them, or by someone reformatting the volume upon which they sit (e.g. rm -f backup.bak).
Corrupting backup files
Backup files can also become corrupted and unusable. This can happen over time via magnetic degradation, also known as bit rot. Someone can also accidentally or maliciously change the files in such a way that they are still there but unusable (e.g. dd if=/dev/random of=backup.bak).
Deleting backups via the backup interface
The backup system’s user interface typically allows a customer to expire (and subsequently delete) backups prior to their originally designated retention period. This is often used to delete backups for a system that is no longer important. Unfortunately, this feature can also be used to accidentally or maliciously delete backups that still have value (e.g. prematurely expire backups in your backup software).
Key takeaways
Every one of these risks above has played out in the news in the last few years. There are dozens of stories of ransomware attacks also encrypting backups. There are also stories of administrators accidentally deleting backups via errant commands. Bit rot is also real, and there’s an entire industry around helping you recover corrupted backups. There are lots of discussions online about whether or not you can recover from backups that you prematurely expired, but they all point to competitor’s sites. (I don’t want to be accused of pointing blame here; suffice it to say it happens).
Hopefully, you get my point, that your backups are at risk and must be protected from all attacks – not just corruption by ransomware. Therefore, all backups should be immutable, and true immutability requires so much more than simply a good backup file-system design.
My next blog post will cover the various attack vectors that allow backup damage to occur, including social engineering, operating system exploits, application exploits, disgruntled employees and contractors, physical exploits, disaster and fires, as well as bit rot. I will then follow that blog post with others discussing how different backup architectures do (or don’t) protect you from these attack vectors. In the meantime, check out this new solution brief to gain more immutability insights.