The job of making backups is usually given to the most junior IT staff because not many want to do the job. As a result, the function and importance of backups are often not given enough attention or totally overlooked. That can no longer be the case as backups, your last line of defense against ransomware, are increasingly exposed to cyber threats.
Tune in to this episode of Ask A CISO to hear:
- The evolution of backups and the implications
- Why it's important to secure your backups, now more than ever
- How you can best do that
- What the killer app of the cloud is, according to our guest
- Whether services like Office 365 should be backed up
About The Guest: W. Curtis Preston
W. Curtis Preston is the Chief Technical Evangelist at Druva, widely acknowledged as an expert in backup & recovery systems; a space he has been working in since 1993.Â
He has written four books on the subject, the fourth of which is Modern Data Protection from O'Reilly, published in May 2021.Â
About The Host: Paul Hadjy
Paul Hadjy is co-founder and CEO of Horangi Cyber Security.Â
Paul leads a team of cybersecurity specialists who create software to solve challenging cybersecurity problems. Horangi brings world-class solutions to provide clients in the Asian market with the right, actionable data to make critical cybersecurity decisions.
Prior to Horangi, Paul worked at Palantir Technologies, where he was instrumental in expanding Palantir’s footprint in the Asia Pacific.Â
He worked across Singapore, Korea, and New Zealand to build Palantir's business in both the commercial and government space and grow its regional teams.Â
He has over a decade of experience and expertise in Anti-Money Laundering, Insider Threats, Cyber Security, Government, and Commercial Banking.Â
Transcript
Mark
Alright! Welcome to Horangi's Ask A CISO podcastAsk A CISO podcast, helping you navigate the rough seas of cybersecurity and information security to get your organizations where they need to go. I'm Mark Fuentes, and I'm sitting in for the boss man, Paul Hadjy, today on our hard-hitting episode so let's get right to it.
Today, we have a really great, great guest - I'd like to present you guys with W Curtis Preston! He is widely known in many circles as Mr. Backup because basically, he's the accepted expert in the field in backup and recovery systems. I think he's been in this space since 1993, quite some time.
Today, he's the Chief Technical Evangelist at Druva, and by today he's written four books on the subject of backup and recovery systems, the latest of which was Modern Data Protection from O'Reilly back in May of last year. He's visiting us today from his own podcast Restore It All, and he is the founder and webmaster of BackupCentral.com.
Welcome, welcome, Preston, how's it going?
Curtis
Glad to be here. Going good.
Mark
Alright, alright, yeah, thanks so much for your time. We appreciate you coming on and you know, sharing some of your hard-earned knowledge with us. So maybe just to get everybody, you know, started, maybe you can tell us a little bit about the Mr. Backup title and how it came about?
Curtis
Everybody else wants to know about that!
The funny thing is I don't even really know how it came to be. It's just that you know, most people ... backup is where a lot of IT people get their start. It's the job that we give to the, you know, to the junior person, which we shouldn't do, we can talk about that, but it is what we do, and the reason we give it to the junior guy is it's the job nobody wants, right?
Mark
Yeah.
Curtis
And so we give it to the junior person and that's how I got my job back in, you know, 1993, at what at that time was the second-largest credit card company. And when I think about it, the fact that I'd been at the company for maybe, I don't know, a couple of weeks, I was a UNIX noob ... I had worked at another company for like two months that just didn't work out, and that was the only UNIX training I had, and now I was being given the root password at a whole bunch of UNIX servers at a 35 billion-dollar company.
Mark
Isn't that the way it used to go? You know, they just, they used to hand out root like it was candy, wasn't it?
Curtis
That is just the way it was years ago right? Yeah, and so what happened I ... you know I've been there a few years I went from being the backup guy to the people in charge of the backup people, and then I went into consulting to actually get out of backup, but through a complete, like nothing having to do with me, I ended up getting put at the headquarters of a large oil and gas company, and their backups were broken, and so it wasn't why I was there but I couldn't help myself and then I just got this idea that maybe I knew something about something that most people didn't, and so I wrote my first article. That was in a magazine called UNIX Review. No one listening to your podcast has ever heard of that magazine but it used to be a big magazine back in the day, and
Mark
I'm definitely old school. I'm one of those people that laments the death of print so I'm right with you, right?
Curtis
Yeah, and that was, I mean I actually ... when that article published I actually got to go down to, we used to have a big chain here called Borders, right?
Mark
Yeah, yeah, I remember that.
Curtis
And that was the last big book store. I actually went to Borders and I ... They had a newstand and my magazine with my name on the cover was on that newsstand, and that was like the coolest moment of my life up to that point, right?
Mark
I can imagine, I can imagine.
Curtis
Yeah, and then, you know, that ... I got the publishing bug from that and shortly after is when I wrote the first book, and, you know, at some point I just realized that, you know, go with what you know, and even though backup is the one of the least sexiest, if not the least sexiest part of, you know, IT. There's a definite need, right? And so ... there you go!
Mark
I mean I would argue and being a GRC person myself, I would argue that probably one of the least sexiest might be ComplianceCompliance. It might be ...
Curtis
We might, we could probably get into a competition!
Mark
I think there's a lot of competition over a lot of things that people consider unsexy being mostly because everyone just wants to be a hacker, right? Everybody wants to be a pen-testerpen-tester so definitely a lot of things we could compete on that so, uh, yeah. I'll leave it there that's I think that's a debate for another episode.
So today you're the Chief Technical Evangelist over at Druva. Maybe a little bit of shameless plugging, maybe you want to tell us about Druva and what's going on over there?
Curtis
Yeah, sure. We are the industry's first and leading SaaS-based data protection system, right? So we are a SaaS service just like Microsoft 365 or Salesforce. Our tech runs in AWSAWS and our customers we back up data centers, SaaS services, cloud services, and endpoints, all, you know, globally deduplicated then backed up to the cloud and, of course, encrypted in transit, encrypted in storage on your behalf, stored in S3S3, and we do all of that as a service without requiring our customers to have any infrastructure. So just you know, depending on what we're backing up, you might just have to authenticate us to a service like 365, you might have to put an agent on a physical machine, or if it's something like VMWare you put an OBA in there to give it, to give us, you know, an end into that world.
But other than that there's no, you know, there's no infrastructure required by our customers.
Mark
Could you ... on that same note, could you maybe extrapolate a little bit about the differences between perhaps somebody managing their own backups and whatnot versus using Druva?
Curtis
Yeah, absolutely. So you know backups have always been ... while they used to almost be easy, what I think back in the day, right? My hardest job was just ... I had, I remember I had like, I don't know, I would make like 50 tapes a day. They were small tapes.
And they would generally all run. Every once in a while a tape drive might break or something but that was relatively easy, but what happened over the years was, actually tape drives got too fast, and everybody thought they were too slow, and so they were changing their designs the wrong way, and I built a whole career out of helping people properly use tape.
And ... but that, but now most everybody, due to that frustration, has moved off of tape, but back then that was our primary problem was let's just get the backups done and get them done, you know, reliably, and what we didn't have to worry about is what I think is the primary concern in backup and recovery environments today. And that's cybersecurity.
No one was attacking the backup systems, right? No one even knew they existed honestly, right?
But now, backup systems are absolutely, you know, a target for ransomware attacks, and exfiltration attacks, and so the backup, yeah, the backup people have to do so much more to their backup system, so basically, in order to have an on-prem backup system or even ... so ... some people, they've taken their on-prem backup VMs and they've moved them into AWS, it's still the same, right? The only thing you don't have to worry about there is the physical infrastructure, but if you just moved your backup into, as a VM in the cloud, you're still dealing with everything I'm about to say. So you need to size it. You need to size the storage, you need to size, you know, the growth of that storage. You typically are buying licenses based on the size of everything, right?
And you're typically doing a large capital purchase which takes you like three to five years, right? It's supposed to take you through three to five years, right, so you have to guess in advance how big you're going to be in three to five years, which is just a ridiculous thing to do, but it's what you have to do. And then once you have guessed that, and you bought all those licenses, you bought all the hardware or virtual, whatever you had to buy, and then you've implemented it. Now you have to secure it against all of these things that, you know, people are going to, you know, to try to do to it, and the problem is, you know I mentioned the tape to disk transformation that happened over the last 20 years, the problem is that move to disk as a primary target for backup is what's created this problem.
Because back in the day, you know, that's what us old guys get to say, back in the day, you made a tape, you handed it to a man in the van, and you had what we called an air gap, right, between the protected system and the protection system.
Now you don't. You have a backup system and behind that backup system it's a disk array, and then you might replicate to another backup system, and behind that backup system is a disk array, and all of this is online.
Mark
It's all networked, yeah it's all networked, right?
Curtis
All of this is tangible.
And so you have to, and then also add to the fact that many of these systems are based on Windows which is the primary attack vector, not ... You know I'm an admitted UNIX bigot. I'll stay it, right? and I won't say that UNIX or Linux is, you know, impervious here. It absolutely isn't, but Windows is still the primary attack vector because that's what the laptops are, right?
So people are out there, they're getting ransomware on their laptops because they kept clicking on the thing we told them not to click on. They get ransomwareransomware then they log into the VPN or come into the building. You remember people used to come into the building?
Mark
I remember I'm one of those back in the day guys too, and I actually miss coming into the building. I miss swiping my card, you know? I miss, I even miss man traps which is weird. Yeah, I miss man traps, you know?
Curtis
I hadn't thought about man traps ... well, I worked at a credit card company, we had real man traps where you had to go in, you had to wear a smock and that had no pockets, and especially when you, what we call plastics which is where they actually made the card, you had to wear you know a thing without pockets and you went in and someone examined you before you got through the other door, right?
Mark
Yeah yeah, definitely like, yeah it's odd I miss those things. I miss those little, you know, now it's just, you know, I get up and I sometimes I put pants on, sometimes I don't. I don't know, you know?
Curtis
Yeah, well, you know you're only being filmed from the waist up, it's all good, you know?
Mark
Yeah yeah, no one knows what's going on under here.
Curtis
Yeah, but the thing is that you as a backup system person, you have to deal with all these things that we just didn't have to deal with back in the day, right?
So that is a reality of a modern-day backup system, a person who's managing a backup system. And many of them are still using backup software that was written 30 years ago and, yes, it's been enhanced over the years but it still often is using you know a design that was inherently flawed, to begin with. You really have to apply, you know, we could talk for a long time about the concepts of least privilegeconcepts of least privilege, role-based administrationrole-based administration, limiting the blast radius of an attack if there is an attack, segregating, you know, there's all sorts of things that a person doing an on-prem backup system must do. It's no longer an option. They must do it and then, or they can use a service like Druva that handles all of that for them, right?
Mark
And that actually, that actually comes to, it brings to mind a question. I had a similar question that brought us into this line, or plan for later, but I'll jump ahead to it. You know I feel like a lot of my clients that I consult with nowadays. When I talk about backup they think of it as like control, like Control-S, like, just, you know hit the save button and we're good.
Right, we oversimplify backup to that process, right? And as you mentioned, you know, there's a lot of cybersecurity that goes into it. There's planning ahead to see how much space you're going to take up, you know, retention rates, there's, you know, practicing the restore process and how complicated restoring is actually, restoring from backup.
So how many, how would you say like what's the percentage of people that understand the complexities of the backup space? Back up and restore space?
Curtis
Tiny, right? For the audio, I'll say I'm holding up two tiny fingers close together.
Again it, I think it's because of, it really goes back to what I said at the beginning that it's not a job anybody wants so if somebody starts getting a, you know, nobody wants to be the person that raises their hand and asks these questions because if you ask these questions, someone's going to say, "Yeah, Curtis, why don't you go look into that?"
Curtis
Yeah now you're the backup expert so, ah, you seem to know a lot about backup, Curtis, why don't you go take a look at the backup system, and so nobody wants to, nobody wants to do the backup job because it's really you're invisible or you're in trouble, right? Nobody remembers the million backups that you got right. They only remember the one restore that you got wrong.
Mark
They only see, they only see you when you fail, right?
Curtis
Yeah. In fact, I, you know, I talk about this job that I first started and I had a major restore failure six weeks into my time there, and luckily that got blamed on poor training and then I had, right towards the end, I managed to push the EPO button in the data center and extended our an outage by several hours because, we know, it was easy to turn off, it wasn't easy to turn back on.
And so I've gone ... that was 25 years ago, and those people, that's literally all they remember to this day. No one remembers all the great things I did to the backups, but they remember that time I pressed the EPO button.
Mark
Remember that time, yeah, I think you remember that time you, that you made it, you made it last six more hours? Remember that time? Oh my gosh ...
Curtis
Yep, yep, yeah, trust me, I remember that time!
So yeah, so nobody wants to be that person and so, as a result, what happens is, is it the backup system, if it, if it is indeed a system or a series of boxes in the corner, it goes completely ignored by so many people, by the cybersecurity person, you know, by sysadmins. It doesn't get patches as quickly as it should.
In my opinion, it should be the first thing that's patched when we're talking about security patches, but again, going back to backup-think, they don't want to patch it because it's the last line of defense. There is nothing more terrifying than upgrading your backup server, right? It's the last line of defense and so no one wants to change it and yet, because of all the cybersecurity stuff that we're talking about, it needs to be changed on a daily basis.
And again going back to a SaaS service, all of that stuff just happens, right? You know we use an Agile development model, we're pushing out code every week or two weeks, and it just, it just gets pushed, nobody's rebooting backup servers anywhere. We actually, on the way the cloud works, you know we're spawning backup servers and containers, and all these types of things when backups need to happen, so there's, this idea of rebooting servers just doesn't exist, right?
Mark
It sounds to me a lot like, I mean, maybe this is an oversimplification but it sounds to me like the AWS of backups basically, right? You don't have to worry about all of those stuff anymore. You just say, hey you guys are my service provider now, and, you know, you guys worry about the nitty-gritty of it all.
Curtis
I would call it the Microsoft 365 of backups, and the reason why I differentiate there is in AWS, you're still responsible for an awful lot, right? They provide you, they provide you the hardware to make a VM, they provide you the storage, they provide you a lot of things but it is still your responsibility to do things with that, right?
Whereas 365, you get the whole kitten caboodle, right? You pay whatever it is, like five bucks a user per month for your email system, and you get magic, right? But the backend system automatically scales up and scales down to meet your needs. All of the ... and just like us, they also need to secure their system and they're going to apply the best, you know, the latest and greatest security systems that they can to protect their customers. The same is true of us, right? So it's just a service that is just super simple to use and it's been designed from the ground up to run in the cloud. We do have some competitors that have taken ...
So that's the beauty of 365 is that you just pay the five dollars a user and you get everything you need, the backend, and this is the real key part, the backend automatically sizes for you, right? So it automatically expands and contracts to meet your needs. You get automatically the amount of storage that you need, no more and no less, right? That saves them money that which gets passed on to you. And then the other thing is that we do have competitors that are running SaaS services in the cloud, but what they've done is adopted a lift-and-shift mentality so what they did was they took, and they just moved VMs up in the cloud and they put a UI in front of it, and there's nothing wrong with that per se.
But when you're, what you're really doing there is you're really just, it's like you're using AWS or Azure as a giant co-lo facility and your costs as a vendor are going to be very different than if you actually program to the way AWS works, right? So
Mark
And you're essentially propagating outdated methods, methodologies, and mechanisms, right? So I ...
Curtis
Agreed.
Mark
When you say there's, I know you're being, uh, I know you're being diplomatic when you say there's nothing wrong with the lift-and-shift approach ...
Curtis
Well, what I'm not gonna say, it's not diplomatic to argue with, it's not diplomatic, there's nothing's wrong with the approach, just not a long-term
approach, right? If you're going to live in the cloud, you need to refactor for the cloud.
Mark
Exactly, and as we move further and further into the cloud, I think lift-and-shift is really not something anyone wants to, not an approach anyone wants to adopt because, like, just first and foremost, it's just not cost-effective, right? And you're not using like you said, you're not using the optimizations that are built into the cloud to your advantage, so I ...
Yeah, but yeah, definitely, definitely I'm right there with you. I actually had a question, I had a question about, my producer put on my list here about what a Chief Technology Evangelist does, but I think I know already because I'm pretty sold, I'm pretty sold, yeah, yeah, I, after I'm sold, I'm sold on this solution. I think it makes sense, you know?
Curtis
I think you do! This is, it is what I do. I sit and I talk all the time.
Mark
That's fantastic. So, yeah, I don't have to ask anymore. I think you sold me. I'm all about the solution at this point, ]: but I want to shift gears really fast, so you know, one, you know one of the hot button issues today is ransomware, obviously, and you know I was just talking about this with a couple of people yesterday and the day before, and we were arguing about how widely known it is that backups are the, like, the best protection against ransomware, and how well people know this, right?
Curtis
Right.
Mark
Is it a widely accepted thing or not. but maybe I wanted you to, you know, in case there are people out there that didn't realize that backups are your main, you know, line of defense against ransomware, which is quite relevant for today? Maybe you could, yeah, maybe you could talk about that a little bit?
Curtis
Yeah, absolutely.
Yeah, I would say it's really your only defense, right? It's your only defense if you've been hit, right? So there are, there are many many things that you must be doing in advance to try to stop the attack, but if you have been attacked, you know, it's the only thing that is going to get you out of hot water without paying the ransom.
And the, you know, it's a difference between your company going out of business or not, and ... but the key here is that it needs to be again, I'll go back in the day, back in the day, so many companies didn't have what I would consider to be a real DR system, right, a Disaster Recovery system. They, we had a box of tapes in a van, right, and we prayed to God we'd never had to use them because we would do, like, you know my, again, we were, I was at this huge bank and we didn't have a real DR system, and we had a box of tapes. Every six months we would test part of the restore but never once, I was there three years, never once did we do a full DR test at all. And even if we did a full DR test, what I know is because the full restore was a true restore, even if it worked, the restore would have taken a ridiculous amount of time that would have cost the company millions and millions of dollars, right?
Mark
Yeah, the proposition is terrifying, it's terrifying.
Curtis
Yeah, absolutely, and so that's why again the cloud comes into play here. The cloud, or DR, and by the way the reason why I'm talking about DR is that's really what, ransomware is another type of disaster, a ransomware attack, right?
Mark
It's a disaster, a disaster, no, yeah, no two ways about it. It's a disaster.
Curtis
It's a particular kind of disaster, will come back to that, right? It's not the same as a hurricane, right? When it, well shoot, I'll just cover it now. When you get hit with a hurricane or, you know, monsoon or whatever, the hurricane hits, the flood happens, the monsoon comes, and then it's over, and then you begin your recovery.
That isn't the case with a ransomware attack. The ransomware attack comes and it's ongoing as you are trying to restore, and there is a group of people who are actively trying to have you not be successful at that restore. So it's, so you know we have to acknowledge that it's a different kind of restore, but it is the only way you're going to really respond well is to have a good, modern-day DR system, and the good news is that DR is the killer app for the cloud because what you need is a whole bunch of infrastructure really quick and you don't want to pay for that infrastructure until you actually need it.
That is the cloud, right?
Mark
Yeah, we've essentially, we've essentially made, you know, hot sites and warm sites obsolete, right?
Curtis
Absolutely, absolutely, and so what you do is, again a modern system, and yes, Druva does this, we're not the only one, but a modern DR system basically pre restores your data into your DR environment, so it's like a standby site but where you're only paying for the disk, the disk part prior to actually testing or declaring a disaster so you do the restore in advance, so you specify what you want your RTO and RPO, your Recovery Time Objective, Recovery Point Objective.
You specify what you want those to be, you specify what needs to be included in a DR, you know, and an executed DR plan, and then it's pre restored on your behalf so that when you go to do disaster recovery, you don't actually have to do the restore part. You might have to do some messing around with it, just a little bit, and you might, and you, and depending on how big you are, your biggest job will be simply booting, you know, a significant number of VMs in a reasonable amount of time, right?
And so what you need is, you need something to orchestrate all of that. So, yeah, backup and DR are your only tools if you've been hit, but they're, they're a little bit like, you know, it's got to be taken you know before, right? It's like, it's like, it's like a, it's like a vaccine, right? Vaccine doesn't help after you've been infected. A vaccine is a great tool on the front end. This is more like medicine, right, so ...
Mark
Yeah, you gotta put in the work upfront.
Curtis
Yeah, but you've got to have decided upfront what's important, decided upfront what, you know, what's going to be included in your disaster, and also where and how you're going to recover that. And hopefully, you have a modern day data protection system that basically does all this in advance for you and then when it's time to do a restore, you literally just push a button and it should, basically, the problem is, the hardest part then is just how long it's gonna take to bring all the VMs online.
Mark
Yeah, and actually, I think you said a keyword there that I think leads me, right, segues me perfectly into my next question. You said modern data protection system which calls back to your latest book, Modern Data Protection, so maybe you know just for those out there, I figure maybe you want to tell us a little bit about the book, you know, who's it for, what are the really good takeaways, a couple of good takeaways from the, obviously I don't wanna spoil it for everybody, right?
Curtis
So, first off, it is an independent book, right? It's not a, it's not a Druva book, it's an O'Reilly book. And you, and what it does is it first lays out here are all the things that need to be backed up which sounds like a really basic, but for example one of the things we talk about are SaaS services like Microsoft 365 and G suite and why, yes, they do need to be backed up, and by the way, what does the Chief Technical Evangelist do? Explain that last thing a lot, right?
Because you have, you do have some people that actively are arguing against that point. So here are all the things that need to be backed up here, you know, and we talk about everything from servers to laptops to SaaS services to containers, and what's unique about them, and VM,s and all of these different things and what's unique about each one. Then we talk about all of the different things that you could back them up to, the different types of backup software out there, I think I identified about 10 different categories of backup software. Then there's also another chapter on backup methods, everything from full and incremental to byte-level incremental to source ID duplication, there's all these different ways to do backup.
And then, and then, you know, I have it, and we also talk about archive, and how archive is not backup, and why it's different. We talk about disaster recovery. We have a chapter on how to explain all this and get money for it, right? How to create, you know, how to get buy-in for a new system. That is really important, right?
And then sort of at the end, we, you know, sum it all up to: here's how to help decide which of all of that is right for you, right? And you know, and you might expect because I work for Druva I'm just going to push Druva in there. I don't. In fact, I tried really hard to give the other sides an even, you know, nicer to the other side to make sure that I wasn't doing that. It was also technically reviewed by people that work at our competitors, so I think I did a pretty good job of being even-handed there, and then, but, you know, really what it comes down to, it's a relatively small book. It's about 350 pages. It is technical but not deep technical, right?
So I'm covering, if I went one layer deeper than I went, the book would have been 20,000 pages long because I'd have to cover every product. I don't cover product names, I just cover product categories, right?
And so I used phrases like traditional backup software, VMWare-centric backup software. There's half a dozen products in that category, the chief of which would be Veeam. Veeam came out and that's what they aimed at. Then there's a couple of companies that have come out that have done scaled out backup appliances. There's a couple of companies that have done that. Rubrik and Cohesity would fall into that category and then there's the SaaS services like Druva, and we're not the only one of those either but, so yeah, so that's so if you're, I will say, if you're worried the book would be too technical for you, I think you'll be fine.
If what you're looking for is a how-to guide on how to, you know, install net backup, this isn't the book for you, right? It would have been 20,000 pages if I did that, but if you're looking to prove a point of like, you know, hey, 365 needs to be backed up, you'll find, you'll find a couple of pages just for you, right? And if you, if you don't know, if you find yourself tasked with backup systems, and you find yourselves in the middle of alphabet soup and you don't know what, all of you know, what is deduplication, what's source side and target side, and you know, you know, it explains what all of those things are to the best of my ability.
Mark
I think, except from what you're saying, it sounds like the book we need, right? The book we need today because you know, there are a billion books about, like you said, the technical side of backup, how it's done, the different concepts, I think we need a book that explains to us at a high level how this stuff works, but more importantly how to get money to fund it, right?
I think that's the thing that's missing in a lot of security programs. Today's personnel who understand how to explain to leadership that, hey, we need to spend money on this, on this stuff, right?
Curtis
Right, and that chapter actually, it's one of two chapters that I actually farmed out to friends, and it's written by a person who that's what he's had to do his whole career, right? He's worked at big corporations where he's had to get money and get buy-in. It's about getting money. It's about getting the rest of your company to buy into not just the money, but the design. The way that you're doing the backups, right?
Because that matters.
Mark
And having them understand the value of that design, right?
Curtis
Right.
Mark
And actually, that actually reaches back to something you said earlier, and you said it and I was very, you know, my ears perked up. You said that there are people who are actively arguing against these things. Could you tell us a little bit about that? Like what's, where, what are the main arguments against this?
Curtis
Yeah, it's ...
The main argument and, by the way, I encounter this most in the Microsoft 365 crowd, which there's some smart people, right? They feel that Microsoft has built enough redundancy and enough native data protection features that you don't need backup. And I say, well, you know, a couple of things, right? My ... I have a couple of sort of silver bullets, right? One of them is the 3-2-1 rule of backup which is a very, you know, it's a very well-acknowledged rule. At least three versions of your stuff, two different media, one of which is off-site.
They have some nice features and they use words like restore in the documentation, you can restore an email, but you're not restoring an email because restore means you went to a backup. A backup by definition is stored somewhere else. It's not stored anywhere else. It's stored in a server, right? I mean I realize, you know, it's a bank of servers, but it's stored in a computer and behind that computer is a database, right, a very specialized database, but it and if something goes wrong, it goes poof. I don't know how to say anything other than that, so that's the three, two, and that's why we have always had backups, right?
The other thing is that, you know, there are stories of people either accidentally or someone maliciously massively corrupting their Microsoft 365 environment, and then if they didn't have a third party backup, they are SOL, right?
And I, you know, I can think of the most famous one, there was one last year where there's a large company called KPMG, and they were a 365 customer. They were trying to, they were, it was in the chats. So they had a user's chat that they wanted to delete they wanted to and, there was never a story as to why this thing needed to be deleted, but it was something bad like Steve should not have said the thing that he said, and we just got to erase all memory of Steve. I don't know what the story is, but they needed to delete Steve's chat history and the history of his chat history, right, because they had the, they had this concept. Microsoft 365 has this concept called retention policies, and you can specify, you can do it globally or you can do it by application. You can say every email, every chat, every file, must be retained for 90 days, you pick the time, right? And you can even push a button that says, and we can't change our mind, so once an email has been received or created, it will be retained for 90 days no matter who says otherwise because you push that button.
And so some people would push this as a, you know, as an alternative to backup. And to which I say, well the problem is there's like a single-digit percentage of Microsoft 365 users that even know they exist let alone use them, but here's what happened. So they had a retention policy for all of their users who said all chats get kept for, again I'm gonna say, 90 days and it, but they wanted to delete this chat right away.
And so what they needed to do is they need to create another policy and then move Steve into that other policy, and another policy with the retention period of zero.
Mark
I can almost see where this is going.
Curtis
And then, and then move in there, yeah, you know where it's going, right? So they did the wrong step. They moved everyone else into the new policy and then, boom, 165,000 users had all of their private chats deleted instantaneously with no backup of them whatsoever. The system did as it was designed to do, right?
And you know, at least, you know, KPMG didn't try 'cause I've seen companies when companies do something stupid like this, they try to sue. They're like, you know, you should have, you know, you should have done, no, at least they knew they, they made a major mistake, and
Mark
You can't really sue for a feature, right?
Curtis
Well, but I mean I've seen it. There was a company that deleted their entire account out of a Google Suite or G suite, right? And that, they were a very cloud company so they stored their entire company's intellectual property in G Drive, and their admin meant to delete a test account deleted the production account, and their entire company ceased to exist. They tried to sue Google and no deal, right?
Mark
We actually, we actually ... I had a customer, same thing. They, their whole stuff was in G suite. They had one, you know they had really poor policies, right? They had one employee that synced the entire thing to his desktop and he decided he didn't want any of that stuff so he just deleted everything and then sync back with their G Suite and the whole company was gone, just gone in a flash.
Curtis
Yeah, and see, this is why we make backups!
Right, now the argument on that side is well if they set up the system properly ...
Mark
But, how many people, but how many people set up stuff properly anyway, right?
Curtis
This is why we make backups and again, and again, I know it sounds like I am pushing it, but this is why I think backups should be as simple as possible. Right, our 365 backups you just authenticate this to the proper user, you specify either the users you want to backup, or just say get everybody and then you're done. That's it, right? And then it's just going to automatically run from that point. So there are people who are actively trying to basically say that Microsoft has built-in enough redundancy, and again remember, redundancy isn't backup.
Redundancy, you know, you have, if you have replication, replication just makes your mistake more effective. Copies of your mistake, right?
And so you know they talk about that there are lagged copies, which is true, but ask Microsoft if you're allowed to use them, and the answer is no. I asked them, where a Microsoft 365 customer, and I asked him a point-blank question if I could use the lagged copy to restore my account if it got horribly corrupted and the answer was no.
Right, you contrast that, I'm aware of our customers get ransomware attacks all the time and I'm aware of a really big one that happened relatively recently. They attacked thousands of desktops and their 365 environments, and we were able to restore them. Easy peasy, right? So the question is when the worst happens, do you want a tool that was never designed to do what you're going to ask it to do, or do you want a tool that was designed for that, right? Because you know, the thing with retention policies, for example, is it will be able to get you, if you used it properly, excuse me, it will be able to get your stuff back, but what it's actually an e-discovery tool, it's not a backup tool, and so it doesn't understand concepts of like folders, for example. It also doesn't know how to set. You can't tell it to put my mailbox the way it looked yesterday, right? I received, you know, 2,000 emails in the last 90 days. I don't want all 2,000 emails. I want the ones that were in my inbox yesterday. That's what a backup tool does and I just don't understand people not wanting that. I get them saying, look we're, you know, we're already paying Microsoft 30 grand a month, we don't want to pay another 30 grand a month for backup. I get it.
I just don't know what to tell you in terms of, you know ...
Mark
You know, I see a lot of that not just for backup but for all kinds of security solutions, security controls, right? That's all. There's always that argument that, well we pay money for this thing that kind of does the same thing, right, but those people are almost immediately, almost all of those people are immediately converted when the stuff hits the fan, right, when something bad happens. like, oh you know what? We should, we should get the enterprise version of the software. You know what? We should probably best in backups, you know?
Curtis
We should, exactly. It's like the old phrase, you've heard this before, but there are no atheists in foxholes, right?
Mark
That's right, exactly.
Curtis
When the bombs are falling, everybody's praying, right?
Mark
Everyone suddenly believes, right?
We're coming, well, we're coming towards the end of our set time so I want to set a little bit of time aside for you to maybe, you know, give us your final thoughts. What are, what, you know, if there was one thing that we all had to walk away from this session today, what is that thing we should be taking away from this session?
Curtis
Well, again, you've got to secure your backup system, right? We all agree we should have one.
Your backup system should be moved to the front in terms of your cybersecurity policies, right? It needs to be not the ignored thing sitting over in the corner. It needs to be in the front because it is the last line of defense, so it needs to be locked up tighter than a drum, and I've written a lot on the kinds of things that you should be doing, and they are in my book, you know, written, I've talked about it on my podcast, Restore It All podcast. We talk a lot about the things that you need to be doing to protect the backup system.
Just make sure you're doing that and again, or you could just use something like Druva and have all of that done for you.
And you know, if what you, if what I'm saying makes, interest you at all, you know you can go to druva.com/podcast and send you a report about it. If you're interested in it, in the book, it's on backup. So I'm sorry, if you're interested in the book, it's on Amazon. Modern Data Protection and my podcast is Restore It All. We even have a theme song that's, it's the name of the podcast, came from a music parody that I wrote. It's a parody of Adele's Rolling In The Deep.
So you know the phrase in the song where she says you could have had it all,
Mark
You could have had it all, right, right.
Curtis
Yeah so mine, so mine is, it's a, it's a song about a girl who got all, her data got deleted, and the backup system didn't work and so she's very angry, and so the phrase is you could restore it all but ...
Mark
You could restore it all, okay.
Curtis
So that's that's where the name came from, and so if you listen to podcasts you get to hear it
Mark
Adele, if you're listening, if you're listening, Adele, we'd like you to record this version it would be fantastic! I think it's a win for both sides!
Curtis
It's a win-win!
Mark
It's a win-win, you could have restored it all, right? That's awesome. That's awesome
Well, just to sum it up then for everyone, Restore It All podcast that's led by Curtis Preston over here who is Mr. Backup. Please check out Druva and also check out Modern Data Protection, the new book out from Preston over here on Amazon.com.
Last words from us: This has been Mark Fuentes with the Ask A CISO podcast. Please stay tuned for more episodes. Thank you so much!
