Absolutely you're right the best way to handle a rootkit is restore from a known-good backup. Just like you practiced, last month when you tested it when found and fixed the problem with backup system.
Unfortunately, 90% of people don't have a proper backup system. Probably over half of systems that are being "backed up" can't actually be restored because the backup media went bad a year ago or whatever.
For the people who don't have a solid backup:
> some IT professional who sells himself to a client by claiming he can remove this and leave the user's precious data intact?
What you definitely don't do is try to salvage the operating system and programs. Just re-install those. It was time to upgrade anyway. DATA *can* be painstakingly recovered. It's a heck of a lot easier if your data isn't mixed with code - no MS Office macros, etc. If you keep your data separate from executable code, it absolutely can be recovered, though it's very easy to slip up and let a potentially infected file through.
One technique for data sterilization is to convert to a different format. For example, converting a Word document to WordPerfect will make sure there are no macros, I believe. Then convert back. Even better, convert to plain text if possible, and leave it as plain text. JPG to bump, etc.
Sorry, but Perl and Python code is programs. You don't trust them from an infected system. Text files you can usually trust, and html that doesn't use javascript or some such. (Not just javascript. You've also got to be careful about allowing CSS, with simple formatting being safe, but anything else needing to be carefully hand checked.) For spreadsheets you should recover from CSV files, but the CSV files can be stored on the disk that got infected. Etc.
Huh? What operating system are you using? Out of the box, Windows sets you up with OneDrive and points all of your storage stuff to OneDrive. The result is that all your files are backed up.
Out of the box, Apple sets up iCloud and points all your file storage to iCloud. The result is that all your files are backed up.
You can use DropBox or a thousand alternatives if you want.
If you want a better solution, you can use either Windows Backup and Restore or Apple Time Machine which does pretty much the same thin
Encrypted backups take care of the "massive invasion of privacy" especially when one's paying for the service, instead of being the product themselves.
> We dragged all his files to an external USB. I determined that there were thousands fewer files on the backup, but we pressed on nevertheless. It turned out that I had not backed up any of his Thunderbird POP3 mailboxes, where all his business-critical data was stored
--If you don't know what you did wrong, you shouldn't be trying to help friends with upgrades. Next time go to Folder Options and Show Hidden Files. And either use Xcopy from CMD window or a modern file copier like Teracopy.
Wtf? No. His first mistake was not finding out what was necessary to backup and exporting profile from Thunderbird along with contacts, address book, autocomplete, etc. Seriously, how mail, documents and browser settings aren't automatic thoughts tells us that this guy was not the "goto guy" he thought he was. "Contracted"? Wtf?
Thank you for post. You've done great job listing things that fool smart, conscientious people into thinking they have a backup. That's why I said a "proper backup", proper being an important word. Those things all LOOK a lot like proper backup, don't they? And yet people who do those things end up asking me to try forensic techniques to recover their data. You seem like you know a few things, so I don't need to tell you exactly how you should do a backup, but let me point out a few common pitfalls to avoi
In the day of half inch 9-track tape the tape format was able to recover multiple single-track errors, be it NRZI or PE. But I have never seen the actual mini computer controller that could do this recovery. No wonder the success rate was near 50 per cent.
Several times I've seen the backup server ran out of space. The ssh key was changed. The list of directories to backup or not backup wasn't up to date. Those are a few things that have broken it after it was setup and running.
All of these can be detected by occasionally doing a test restore, perhaps to a VM, and checking that the important files are there and important functionality works.
> Ransomware reminds us of another requirement - the system being backed up (which may get ransomware) can not have the ability to delete or modify the backups. Sending backups to a network drive just means the ransomware or disgruntled employee will destroy two copies of the data
--ZFS+Snapshots+Samba works pretty well for this. Keeping a ZFS snapshot every (2) hours for a month (as well as changing file permissions) is pretty easy on a Linux server.
Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share. That can be made secure by the backup backup pulling files that are shared by machine to be backed up. So the reverse of the typical model.
> Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share:nods: --Yep, but once you A) isolate the infected machines and B) ssh into the server and do a ZFS rollback (to a pre-infection snapshot time) on the ZFS-backed Samba dataset, all is back to normal again:D Rollback even "deletes" the infected files for you. ZFS snapshots are immutable unless you have basically root on the server.
I had envisioned something a tad different when I read your earlier post.
That's fairly similar to part of what I did on the very cool backup service I used to sell. Except I used LVM snapshots rather than ZFS, which gave us the flexibility to do some other really cool stuff.
Sometimes the integration of ZFS is handy, sometimes it's a major limitation. It's a lot more flexible to use a file system as a file system, a volume manager as a volume manager, and RAID for RAID. ZFS tries to be all three, creating co
Nearly 10 years ago, I suffered from a hard drive crash and I lost a ton of data. Ever since that issue, I’ve been religious about backups. I used Mac OS’s built-in backup software and I copied all my documents and work files to a flash drive daily. I instructed my family to grab the NAS drive on the way out of the house in the event of fire.
I subscribed to Crashplan cloud backup at some point. They went belly-up but I had already switched to Backlaze. It sounded like it would be a hass
Some 15 years ago I worked in some institution. My policy is:
1) Install the new system on new HDD. 2) Copy all work files to the new HDD. 3) Hide the old HDD. 4) When it's known that everything works then save some critical work files somewhere, test and reuse the HDD.
I asked the management that I need a new HDD. The institution head told my boss to supply me with HDD. My boss left the resolution "You don't need a new HDD". I copied the work files, erased the HDD and reinstalled the system. Then it appeared th
> apps will need to be reinstalled but at least on Linux that's fairly easy.
Re-installing the software is REALLY easy if your data includes the output of rpm -qa.
Also sometimes very handy when things go wrong - cat/proc/mdstat, pvdisplay, lvmbackup, and gdisk -l
I'm recovering an old customer's data right now. He no longer has backups with me and someone built a new, wmpty raid on his drives, making it "impossible" to recover his data. However, the old copies of mdstat and the partition layout were still
Most malware doesn't immediately destroy your computer, it cripples it over days or weeks. I can't tell you the number of people who told me "Yeah, I noticed something last week and it's been flaky since then."
Meanwhile, you've been syncing your infection up to the cloud the whole time so now your cloud storage is infected, too. You may get some of it back, but I've also seen people just re-infect themselves, too.
Some cloud storage often at higher tiers will offer some kind of versioning and let you restore pre-infected files, but for most people this isn't the default or isn't even a feature they have.
The only way cloud sync really works as a backup is if you have a spare computer you only bring online periodically that syncs itself and that you then take offline again, but now all you've done is add a complex network transaction to what amounts to a local backup.
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems.
I don't mean to be rude, but you should turn in your geek card. Maintaining good backups is even easier in Linux than any other operating system.
Everything unique will be under/home/username. You can back this up with rsync, cp, tar, or even dd if it is a partition. There is no hand holding, but then, it really shouldn't be necessary when the design itself is so elegantly simple. What is even cooler is that this knowledge of backing up carries across to the various BSDs and other Unix-like operating system
Copying full and/or incremental filesets to different disks (USB sticks, HDDs, student servers, Cloud, et. al.) has always worked, and always will. For personal files, you should encrypt first, easily and portably done with openssl. Most of this can be automated too, which on Linux is very easy and transparent.
For very big/complex/incremental, it'll depend on requirements, but this is so rare, that when you need it, you'll find something that works, and you should.
Absolutely you're right the best way to handle a rootkit is restore from a known-good backup.
What is a "known-good" backup? A rootkit is here to conceal its existence. You don't really know when the infection started, and which backups are good.
That's a very good question. You can use diff to see what the differences are between different backups. That normally makes it pretty obvious. You pretty much know which files were supposed to change and which weren't. This can even give you good hints as to HOW you got infected.
There are even faster ways to tell because rootkits tend to re-send the same components. I can normally see a rootkit on a Linux system in seconds, without even actively looking for it. I'm not going to post the trick here because
Re: (Score:3)
Data yes, OS and programs, no (Score:2)
Absolutely you're right the best way to handle a rootkit is restore from a known-good backup. Just like you practiced, last month when you tested it when found and fixed the problem with backup system.
Unfortunately, 90% of people don't have a proper backup system. Probably over half of systems that are being "backed up" can't actually be restored because the backup media went bad a year ago or whatever.
For the people who don't have a solid backup:
> some IT professional who sells himself to a client by claiming he can remove this and leave the user's precious data intact?
What you definitely don't do is try to salvage the operating system and programs. Just re-install those. It was time to upgrade anyway. DATA *can* be painstakingly recovered. It's a heck of a lot easier if your data isn't mixed with code - no MS Office macros, etc. If you keep your data separate from executable code, it absolutely can be recovered, though it's very easy to slip up and let a potentially infected file through.
Re: (Score:1)
Format conversion for sterilization. Word - WPS (Score:2)
One technique for data sterilization is to convert to a different format. For example, converting a Word document to WordPerfect will make sure there are no macros, I believe. Then convert back. Even better, convert to plain text if possible, and leave it as plain text. JPG to bump, etc.
Re: (Score:2)
Sorry, but Perl and Python code is programs. You don't trust them from an infected system. Text files you can usually trust, and html that doesn't use javascript or some such. (Not just javascript. You've also got to be careful about allowing CSS, with simple formatting being safe, but anything else needing to be carefully hand checked.) For spreadsheets you should recover from CSV files, but the CSV files can be stored on the disk that got infected. Etc.
But just running code in a virtual machine does
Re: (Score:3, Interesting)
Out of the box, Windows sets you up with OneDrive and points all of your storage stuff to OneDrive. The result is that all your files are backed up.
Out of the box, Apple sets up iCloud and points all your file storage to iCloud. The result is that all your files are backed up.
You can use DropBox or a thousand alternatives if you want.
If you want a better solution, you can use either Windows Backup and Restore or Apple Time Machine which does pretty much the same thin
Re: (Score:1)
BackBlaze: yes, OS and programs, no (Score:0)
Encrypted backups take care of the "massive invasion of privacy" especially when one's paying for the service, instead of being the product themselves.
Re: (Score:2)
> We dragged all his files to an external USB. I determined that there were thousands fewer files on the backup, but we pressed on nevertheless. It turned out that I had not backed up any of his Thunderbird POP3 mailboxes, where all his business-critical data was stored
--If you don't know what you did wrong, you shouldn't be trying to help friends with upgrades. Next time go to Folder Options and Show Hidden Files. And either use Xcopy from CMD window or a modern file copier like Teracopy.
--Also, grab
Re: (Score:1)
Re: Data yes, OS and programs, no (Score:0)
Wtf? No. His first mistake was not finding out what was necessary to backup and exporting profile from Thunderbird along with contacts, address book, autocomplete, etc. Seriously, how mail, documents and browser settings aren't automatic thoughts tells us that this guy was not the "goto guy" he thought he was. "Contracted"? Wtf?
Re: Data yes, OS and programs, no (Score:0)
Linux backups are easy. Rsync for data, clonezilla for systems. And clonezilla is basically just rsync, ssh, gz and some pipe symbols.
Two infected, fire-burned copies isn't backup (Score:3)
Thank you for post. You've done great job listing things that fool smart, conscientious people into thinking they have a backup. That's why I said a "proper backup", proper being an important word. Those things all LOOK a lot like proper backup, don't they? And yet people who do those things end up asking me to try forensic techniques to recover their data. You seem like you know a few things, so I don't need to tell you exactly how you should do a backup, but let me point out a few common pitfalls to avoi
Re: (Score:2)
I mentioned before backups must be tested regularly. Backups that haven't been recently tested have a failure rate of about 50%, in my experience.
What kinds of failures do you see? In the days of tape, 50% (or probably higher) was pretty common, but most people are using the 'cloud' now.
Re: (Score:0)
I imagine 50% = either it worked or it didn't :P
Re: (Score:2)
In the day of half inch 9-track tape the tape format was able to recover multiple single-track errors, be it NRZI or PE. But I have never seen the actual mini computer controller that could do this recovery. No wonder the success rate was near 50 per cent.
Re: Two infected, fire-burned copies isn't backup (Score:0)
That's good to know
Many different problems. Keys, space, directories (Score:2)
Several times I've seen the backup server ran out of space. The ssh key was changed. The list of directories to backup or not backup wasn't up to date. Those are a few things that have broken it after it was setup and running.
All of these can be detected by occasionally doing a test restore, perhaps to a VM, and checking that the important files are there and important functionality works.
Re: (Score:2)
> Ransomware reminds us of another requirement - the system being backed up (which may get ransomware) can not have the ability to delete or modify the backups. Sending backups to a network drive just means the ransomware or disgruntled employee will destroy two copies of the data
--ZFS+Snapshots+Samba works pretty well for this. Keeping a ZFS snapshot every (2) hours for a month (as well as changing file permissions) is pretty easy on a Linux server.
Pull, not push on that Samba (Score:2)
Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share. That can be made secure by the backup backup pulling files that are shared by machine to be backed up. So the reverse of the typical model.
Re: (Score:2)
> Ransomware typically wipes any network drives using the SMB protocol, as Samba does, if the infected machine has access to the share :nods: :D Rollback even "deletes" the infected files for you. ZFS snapshots are immutable unless you have basically root on the server.
--Yep, but once you A) isolate the infected machines and B) ssh into the server and do a ZFS rollback (to a pre-infection snapshot time) on the ZFS-backed Samba dataset, all is back to normal again
https://forums.freebsd.org/thr... [freebsd.org]
https:/ [github.com]
Cool (Score:2)
I had envisioned something a tad different when I read your earlier post.
That's fairly similar to part of what I did on the very cool backup service I used to sell. Except I used LVM snapshots rather than ZFS, which gave us the flexibility to do some other really cool stuff.
Sometimes the integration of ZFS is handy, sometimes it's a major limitation. It's a lot more flexible to use a file system as a file system, a volume manager as a volume manager, and RAID for RAID. ZFS tries to be all three, creating co
Re: (Score:2)
Nearly 10 years ago, I suffered from a hard drive crash and I lost a ton of data. Ever since that issue, I’ve been religious about backups. I used Mac OS’s built-in backup software and I copied all my documents and work files to a flash drive daily. I instructed my family to grab the NAS drive on the way out of the house in the event of fire.
I subscribed to Crashplan cloud backup at some point. They went belly-up but I had already switched to Backlaze. It sounded like it would be a hass
Re: (Score:2)
Some 15 years ago I worked in some institution. My policy is:
1) Install the new system on new HDD.
2) Copy all work files to the new HDD.
3) Hide the old HDD.
4) When it's known that everything works then save some critical work files somewhere, test and reuse the HDD.
I asked the management that I need a new HDD. The institution head told my boss to supply me with HDD. My boss left the resolution "You don't need a new HDD". I copied the work files, erased the HDD and reinstalled the system. Then it appeared th
Re: (Score:2)
SpiderOak is good for Linux. It can only cover your data, apps will need to be reinstalled but at least on Linux that's fairly easy.
On Windows there is Chocolately for installing and updating apps, but I haven't tried it.
Rpm -qa, cat mdstat, gdisk -l (Score:2)
> apps will need to be reinstalled but at least on Linux that's fairly easy.
Re-installing the software is REALLY easy if your data includes the output of rpm -qa.
Also sometimes very handy when things go wrong - /proc/mdstat, pvdisplay, lvmbackup, and gdisk -l
cat
I'm recovering an old customer's data right now. He no longer has backups with me and someone built a new, wmpty raid on his drives, making it "impossible" to recover his data. However, the old copies of mdstat and the partition layout were still
Sync isn't backup (Score:5, Informative)
Sync to OneDrive, et al, isn't backup.
Most malware doesn't immediately destroy your computer, it cripples it over days or weeks. I can't tell you the number of people who told me "Yeah, I noticed something last week and it's been flaky since then."
Meanwhile, you've been syncing your infection up to the cloud the whole time so now your cloud storage is infected, too. You may get some of it back, but I've also seen people just re-infect themselves, too.
Some cloud storage often at higher tiers will offer some kind of versioning and let you restore pre-infected files, but for most people this isn't the default or isn't even a feature they have.
The only way cloud sync really works as a backup is if you have a spare computer you only bring online periodically that syncs itself and that you then take offline again, but now all you've done is add a complex network transaction to what amounts to a local backup.
Re: (Score:2)
>to be honest, I have absolutely no idea how to maintain good backups of my Linux systems
--Tar and fsarchiver. Send me a private email and I can send you my root admin scripts, complete with bare-metal restore ability.
Re: (Score:2)
That said, to be honest, I have absolutely no idea how to maintain good backups of my Linux systems.
I don't mean to be rude, but you should turn in your geek card. Maintaining good backups is even easier in Linux than any other operating system.
Everything unique will be under /home/username. You can back this up with rsync, cp, tar, or even dd if it is a partition. There is no hand holding, but then, it really shouldn't be necessary when the design itself is so elegantly simple. What is even cooler is that this knowledge of backing up carries across to the various BSDs and other Unix-like operating system
Re: (Score:0)
Copying full and/or incremental filesets to different disks (USB sticks, HDDs, student servers, Cloud, et. al.) has always worked, and always will.
For personal files, you should encrypt first, easily and portably done with openssl. Most of this can be automated too, which on Linux is very easy and transparent.
For very big/complex/incremental, it'll depend on requirements, but this is so rare, that when you need it, you'll find something that works, and you should.
Windows can be more brittle on this because
Re: Data yes, OS and programs, no (Score:0)
I guess you don't know about cron and rsync. Btrfs supports snapshots. Maybe it's more your fault than Linux's.
Re: (Score:2)
Absolutely you're right the best way to handle a rootkit is restore from a known-good backup.
What is a "known-good" backup? A rootkit is here to conceal its existence. You don't really know when the infection started, and which backups are good.
Great question. I wish the answer wasn't secret (Score:2)
That's a very good question. You can use diff to see what the differences are between different backups. That normally makes it pretty obvious. You pretty much know which files were supposed to change and which weren't. This can even give you good hints as to HOW you got infected.
There are even faster ways to tell because rootkits tend to re-send the same components. I can normally see a rootkit on a Linux system in seconds, without even actively looking for it. I'm not going to post the trick here because