Absolutely you're right the best way to handle a rootkit is restore from a known-good backup. Just like you practiced, last month when you tested it when found and fixed the problem with backup system.
Unfortunately, 90% of people don't have a proper backup system. Probably over half of systems that are being "backed up" can't actually be restored because the backup media went bad a year ago or whatever.
For the people who don't have a solid backup:
> some IT professional who sells himself to a client by cl
Out of the box, Windows sets you up with OneDrive and points all of your storage stuff to OneDrive. The result is that all your files are backed up.
Out of the box, Apple sets up iCloud and points all your file storage to iCloud. The result is that all your files are backed up.
You can use DropBox or a thousand alternatives if you want.
If you want a better solution, you can use either Windows Backup and Restore or Apple Time Machine which does pretty much the same thing.
If you're a developer, then all your stuff is on Github or similar.
As for applications, Windows Store and App Store makes that pretty quick and simple. Of course, there are some other programs you would install otherwise, but it's not like you can't download them.
Also, if you have a Mac or a Microsoft Surface, you can simply reinstall the OS no matter how bungled it may by simply connecting to the Internet from the UEFI system and recovering from the cloud for example.
You have to be an absolute moron in 2018 to no have access to all your stuff.
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems. I keep most of my stuff on Github. Other than VS Code and.NET Core, I don't really use much more than a simple Linux install anyway. I don't use anything but Raspberry, Orange and Banana Pis for servers anymore. I have 25,000 of them now. When they die, I just throw them away and get more.
> We dragged all his files to an external USB. I determined that there were thousands fewer files on the backup, but we pressed on nevertheless. It turned out that I had not backed up any of his Thunderbird POP3 mailboxes, where all his business-critical data was stored
--If you don't know what you did wrong, you shouldn't be trying to help friends with upgrades. Next time go to Folder Options and Show Hidden Files. And either use Xcopy from CMD window or a modern file copier like Teracopy.
Thank you for post. You've done great job listing things that fool smart, conscientious people into thinking they have a backup. That's why I said a "proper backup", proper being an important word. Those things all LOOK a lot like proper backup, don't they? And yet people who do those things end up asking me to try forensic techniques to recover their data. You seem like you know a few things, so I don't need to tell you exactly how you should do a backup, but let me point out a few common pitfalls to avoi
In the day of half inch 9-track tape the tape format was able to recover multiple single-track errors, be it NRZI or PE. But I have never seen the actual mini computer controller that could do this recovery. No wonder the success rate was near 50 per cent.
Several times I've seen the backup server ran out of space. The ssh key was changed. The list of directories to backup or not backup wasn't up to date. Those are a few things that have broken it after it was setup and running.
All of these can be detected by occasionally doing a test restore, perhaps to a VM, and checking that the important files are there and important functionality works.
> Ransomware reminds us of another requirement - the system being backed up (which may get ransomware) can not have the ability to delete or modify the backups. Sending backups to a network drive just means the ransomware or disgruntled employee will destroy two copies of the data
--ZFS+Snapshots+Samba works pretty well for this. Keeping a ZFS snapshot every (2) hours for a month (as well as changing file permissions) is pretty easy on a Linux server.
Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share. That can be made secure by the backup backup pulling files that are shared by machine to be backed up. So the reverse of the typical model.
> Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share:nods: --Yep, but once you A) isolate the infected machines and B) ssh into the server and do a ZFS rollback (to a pre-infection snapshot time) on the ZFS-backed Samba dataset, all is back to normal again:D Rollback even "deletes" the infected files for you. ZFS snapshots are immutable unless you have basically root on the server.
I had envisioned something a tad different when I read your earlier post.
That's fairly similar to part of what I did on the very cool backup service I used to sell. Except I used LVM snapshots rather than ZFS, which gave us the flexibility to do some other really cool stuff.
Sometimes the integration of ZFS is handy, sometimes it's a major limitation. It's a lot more flexible to use a file system as a file system, a volume manager as a volume manager, and RAID for RAID. ZFS tries to be all three, creating co
Nearly 10 years ago, I suffered from a hard drive crash and I lost a ton of data. Ever since that issue, I’ve been religious about backups. I used Mac OS’s built-in backup software and I copied all my documents and work files to a flash drive daily. I instructed my family to grab the NAS drive on the way out of the house in the event of fire.
I subscribed to Crashplan cloud backup at some point. They went belly-up but I had already switched to Backlaze. It sounded like it would be a hass
Some 15 years ago I worked in some institution. My policy is:
1) Install the new system on new HDD. 2) Copy all work files to the new HDD. 3) Hide the old HDD. 4) When it's known that everything works then save some critical work files somewhere, test and reuse the HDD.
I asked the management that I need a new HDD. The institution head told my boss to supply me with HDD. My boss left the resolution "You don't need a new HDD". I copied the work files, erased the HDD and reinstalled the system. Then it appeared th
> apps will need to be reinstalled but at least on Linux that's fairly easy.
Re-installing the software is REALLY easy if your data includes the output of rpm -qa.
Also sometimes very handy when things go wrong - cat/proc/mdstat, pvdisplay, lvmbackup, and gdisk -l
I'm recovering an old customer's data right now. He no longer has backups with me and someone built a new, wmpty raid on his drives, making it "impossible" to recover his data. However, the old copies of mdstat and the partition layout were still
Most malware doesn't immediately destroy your computer, it cripples it over days or weeks. I can't tell you the number of people who told me "Yeah, I noticed something last week and it's been flaky since then."
Meanwhile, you've been syncing your infection up to the cloud the whole time so now your cloud storage is infected, too. You may get some of it back, but I've also seen people just re-infect themselves, too.
Some cloud storage often at higher tiers will offer some kind of versioning and let you restore pre-infected files, but for most people this isn't the default or isn't even a feature they have.
The only way cloud sync really works as a backup is if you have a spare computer you only bring online periodically that syncs itself and that you then take offline again, but now all you've done is add a complex network transaction to what amounts to a local backup.
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems.
I don't mean to be rude, but you should turn in your geek card. Maintaining good backups is even easier in Linux than any other operating system.
Everything unique will be under/home/username. You can back this up with rsync, cp, tar, or even dd if it is a partition. There is no hand holding, but then, it really shouldn't be necessary when the design itself is so elegantly simple. What is even cooler is that this knowledge of backing up carries across to the various BSDs and other Unix-like operating system
One way to make your old car run better is to look up the price of a new model.
Re: (Score:3)
Data yes, OS and programs, no (Score:2)
Absolutely you're right the best way to handle a rootkit is restore from a known-good backup. Just like you practiced, last month when you tested it when found and fixed the problem with backup system.
Unfortunately, 90% of people don't have a proper backup system. Probably over half of systems that are being "backed up" can't actually be restored because the backup media went bad a year ago or whatever.
For the people who don't have a solid backup:
> some IT professional who sells himself to a client by cl
Re:Data yes, OS and programs, no (Score:3, Interesting)
Out of the box, Windows sets you up with OneDrive and points all of your storage stuff to OneDrive. The result is that all your files are backed up.
Out of the box, Apple sets up iCloud and points all your file storage to iCloud. The result is that all your files are backed up.
You can use DropBox or a thousand alternatives if you want.
If you want a better solution, you can use either Windows Backup and Restore or Apple Time Machine which does pretty much the same thing.
If you're a developer, then all your stuff is on Github or similar.
As for applications, Windows Store and App Store makes that pretty quick and simple. Of course, there are some other programs you would install otherwise, but it's not like you can't download them.
Also, if you have a Mac or a Microsoft Surface, you can simply reinstall the OS no matter how bungled it may by simply connecting to the Internet from the UEFI system and recovering from the cloud for example.
You have to be an absolute moron in 2018 to no have access to all your stuff.
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems. I keep most of my stuff on Github. Other than VS Code and
Re: (Score:1)
Re: (Score:2)
> We dragged all his files to an external USB. I determined that there were thousands fewer files on the backup, but we pressed on nevertheless. It turned out that I had not backed up any of his Thunderbird POP3 mailboxes, where all his business-critical data was stored
--If you don't know what you did wrong, you shouldn't be trying to help friends with upgrades. Next time go to Folder Options and Show Hidden Files. And either use Xcopy from CMD window or a modern file copier like Teracopy.
--Also, grab
Re: (Score:1)
Two infected, fire-burned copies isn't backup (Score:3)
Thank you for post. You've done great job listing things that fool smart, conscientious people into thinking they have a backup. That's why I said a "proper backup", proper being an important word. Those things all LOOK a lot like proper backup, don't they? And yet people who do those things end up asking me to try forensic techniques to recover their data. You seem like you know a few things, so I don't need to tell you exactly how you should do a backup, but let me point out a few common pitfalls to avoi
Re: (Score:2)
I mentioned before backups must be tested regularly. Backups that haven't been recently tested have a failure rate of about 50%, in my experience.
What kinds of failures do you see? In the days of tape, 50% (or probably higher) was pretty common, but most people are using the 'cloud' now.
Re: (Score:2)
In the day of half inch 9-track tape the tape format was able to recover multiple single-track errors, be it NRZI or PE. But I have never seen the actual mini computer controller that could do this recovery. No wonder the success rate was near 50 per cent.
Many different problems. Keys, space, directories (Score:2)
Several times I've seen the backup server ran out of space. The ssh key was changed. The list of directories to backup or not backup wasn't up to date. Those are a few things that have broken it after it was setup and running.
All of these can be detected by occasionally doing a test restore, perhaps to a VM, and checking that the important files are there and important functionality works.
Re: (Score:2)
> Ransomware reminds us of another requirement - the system being backed up (which may get ransomware) can not have the ability to delete or modify the backups. Sending backups to a network drive just means the ransomware or disgruntled employee will destroy two copies of the data
--ZFS+Snapshots+Samba works pretty well for this. Keeping a ZFS snapshot every (2) hours for a month (as well as changing file permissions) is pretty easy on a Linux server.
Pull, not push on that Samba (Score:2)
Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share. That can be made secure by the backup backup pulling files that are shared by machine to be backed up. So the reverse of the typical model.
Re: (Score:2)
> Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share :nods: :D Rollback even "deletes" the infected files for you. ZFS snapshots are immutable unless you have basically root on the server.
--Yep, but once you A) isolate the infected machines and B) ssh into the server and do a ZFS rollback (to a pre-infection snapshot time) on the ZFS-backed Samba dataset, all is back to normal again
https://forums.freebsd.org/thr... [freebsd.org]
https:/ [github.com]
Cool (Score:2)
I had envisioned something a tad different when I read your earlier post.
That's fairly similar to part of what I did on the very cool backup service I used to sell. Except I used LVM snapshots rather than ZFS, which gave us the flexibility to do some other really cool stuff.
Sometimes the integration of ZFS is handy, sometimes it's a major limitation. It's a lot more flexible to use a file system as a file system, a volume manager as a volume manager, and RAID for RAID. ZFS tries to be all three, creating co
Re: (Score:2)
Nearly 10 years ago, I suffered from a hard drive crash and I lost a ton of data. Ever since that issue, I’ve been religious about backups. I used Mac OS’s built-in backup software and I copied all my documents and work files to a flash drive daily. I instructed my family to grab the NAS drive on the way out of the house in the event of fire.
I subscribed to Crashplan cloud backup at some point. They went belly-up but I had already switched to Backlaze. It sounded like it would be a hass
Re: (Score:2)
Some 15 years ago I worked in some institution. My policy is:
1) Install the new system on new HDD.
2) Copy all work files to the new HDD.
3) Hide the old HDD.
4) When it's known that everything works then save some critical work files somewhere, test and reuse the HDD.
I asked the management that I need a new HDD. The institution head told my boss to supply me with HDD. My boss left the resolution "You don't need a new HDD". I copied the work files, erased the HDD and reinstalled the system. Then it appeared th
Re: (Score:2)
SpiderOak is good for Linux. It can only cover your data, apps will need to be reinstalled but at least on Linux that's fairly easy.
On Windows there is Chocolately for installing and updating apps, but I haven't tried it.
Rpm -qa, cat mdstat, gdisk -l (Score:2)
> apps will need to be reinstalled but at least on Linux that's fairly easy.
Re-installing the software is REALLY easy if your data includes the output of rpm -qa.
Also sometimes very handy when things go wrong - /proc/mdstat, pvdisplay, lvmbackup, and gdisk -l
cat
I'm recovering an old customer's data right now. He no longer has backups with me and someone built a new, wmpty raid on his drives, making it "impossible" to recover his data. However, the old copies of mdstat and the partition layout were still
Sync isn't backup (Score:5, Informative)
Sync to OneDrive, et al, isn't backup.
Most malware doesn't immediately destroy your computer, it cripples it over days or weeks. I can't tell you the number of people who told me "Yeah, I noticed something last week and it's been flaky since then."
Meanwhile, you've been syncing your infection up to the cloud the whole time so now your cloud storage is infected, too. You may get some of it back, but I've also seen people just re-infect themselves, too.
Some cloud storage often at higher tiers will offer some kind of versioning and let you restore pre-infected files, but for most people this isn't the default or isn't even a feature they have.
The only way cloud sync really works as a backup is if you have a spare computer you only bring online periodically that syncs itself and that you then take offline again, but now all you've done is add a complex network transaction to what amounts to a local backup.
Re: (Score:2)
>to be honest, I have absolutely no idea how to maintain good backups of my Linux systems
--Tar and fsarchiver. Send me a private email and I can send you my root admin scripts, complete with bare-metal restore ability.
Re: (Score:2)
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems.
I don't mean to be rude, but you should turn in your geek card. Maintaining good backups is even easier in Linux than any other operating system.
Everything unique will be under /home/username. You can back this up with rsync, cp, tar, or even dd if it is a partition. There is no hand holding, but then, it really shouldn't be necessary when the design itself is so elegantly simple. What is even cooler is that this knowledge of backing up carries across to the various BSDs and other Unix-like operating system