Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
-wes
Personally, I just do 'dd if=/dev/zero of=/path/to/zero.img bs=1M; rm -f /path/to/zero.img'. It's inelegant, for sure, but it works (note to run it as a normal user or else be careful of how your system reacts to running out of disk space for a moment).
fix-it-with-a-hammer-digimer
On 08/02/16 04:34 PM, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
-wes _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Feb 8, 2016, at 2:37 PM, Digimer lists@alteeve.ca wrote:
Personally, I just do 'dd if=/dev/zero of=/path/to/zero.img bs=1M; rm -f /path/to/zero.img'. It's inelegant, for sure, but it works (note to run it as a normal user or else be careful of how your system reacts to running out of disk space for a moment).
fix-it-with-a-hammer-digimer
Looks like it’s working. Thanks.
On 08/02/16 04:34 PM, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
On Mon, February 8, 2016 3:37 pm, Digimer wrote:
Personally, I just do 'dd if=/dev/zero of=/path/to/zero.img bs=1M; rm -f /path/to/zero.img'. It's inelegant, for sure, but it works (note to run it as a normal user or else be careful of how your system reacts to running out of disk space for a moment).
This definitely does the trick. reallocated bad blocks aside, one path writing zeroes on modern drives is sufficient, according to one nice paper on the subject I remember. Does not comply DoD (and similar) secure data destruction though... As it always is when army is concerned: overkill ;-)
Valeri
fix-it-with-a-hammer-digimer
On 08/02/16 04:34 PM, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but Iâm not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
-wes _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
I don't understand the point of doing this. If you want to sanitize the disk, use dban <dban.org>, which surely approaches industry standard for the open source answer.
Just zeroing random blocks? Why? If you want to wipe a specific file, there's shred.
mark
On 02/08/2016 03:05 PM, m.roth@5-cent.us wrote:
Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
I don't understand the point of doing this.
Wes didn't say the reason he wanted to zero unused blocks, but I always do this in kickstart scripts when constructing VM images as the image size is considerably reduced by doing this...
-Greg
Once upon a time, Greg Bailey gbailey@lxpro.com said:
Wes didn't say the reason he wanted to zero unused blocks, but I always do this in kickstart scripts when constructing VM images as the image size is considerably reduced by doing this...
For that purpose, use something that can TRIM a VM image, like virt-sparsify.
On 02/08/2016 07:04 PM, Chris Adams wrote:
Once upon a time, Greg Bailey gbailey@lxpro.com said:
Wes didn't say the reason he wanted to zero unused blocks, but I always do this in kickstart scripts when constructing VM images as the image size is considerably reduced by doing this...
For that purpose, use something that can TRIM a VM image, like virt-sparsify.
That's doing the same thing.
virt-sparsify works by mounting the filesystem, filling it to capacity with zeros, then performing a copy operation which skips over the all-zero blocks, leaving them unallocated in the sparse destination file.
Once upon a time, Robert Nichols rnicholsNOSPAM@comcast.net said:
On 02/08/2016 07:04 PM, Chris Adams wrote:
For that purpose, use something that can TRIM a VM image, like virt-sparsify.
That's doing the same thing.
virt-sparsify works by mounting the filesystem, filling it to capacity with zeros, then performing a copy operation which skips over the all-zero blocks, leaving them unallocated in the sparse destination file.
Well, that's the slow method that copies the image. There's also the in-place method, which just does a TRIM that pokes holes in the original image (much faster and uses no extra disk space).
hdparm supports ATA secure erase. This is SSD safe, unlike other options. It's faster than writing zeros to both HDD and SSD.
Chris Murphy
On Mon, Feb 8, 2016, 3:06 PM m.roth@5-cent.us wrote:
Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
I don't understand the point of doing this. If you want to sanitize the disk, use dban <dban.org>, which surely approaches industry standard for the open source answer.
Just zeroing random blocks? Why? If you want to wipe a specific file, there's shred.
mark
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
Chris Murphy
Chris Murphy wrote:
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
dban doesn't? What F/OSS does "secure erase"? And does it do what dban's DoD 5220.22-M does?
mark, overkill
On 2/8/2016 2:18 PM, m.roth@5-cent.us wrote:
dban doesn't? What F/OSS does "secure erase"? And does it do what dban's DoD 5220.22-M does?
do you even know what NISP Operating Manual 5220.22-M is? One thing it does NOT have is ANY specifications of methods of data erasure (it mentions data erasure in 2 short paragraphs, out of a 140 page book on security).
The Defense Security Service C&S-M (clearing and sanitization matrix) procedures state that magnetic disks will be degaussed or physically destroyed. http://www.oregon.gov/DAS/OP/docs/policy/state/107-009-005_Exhibit_B.pdf
note that degaussing a hard drive made since the early 80s will erase its servo tracks and render it scrap.
there is no such thing as secure erasure. the whole silly 3 passes of random data followed by zeroing thing has been debunked numerous times. It MIGHT have worked in the days of MFM disks, when block sparing was an OS function, and the drives just provided a stream of 1s and 0s without recognizing 'sectors' (sectoring was done in the MFM disk controller board). With any modern storage device writing a single pass of zeros will do virtually the same thing, and is adequate to remove casual data but by no means good enough for any sort of government mandated security.
On Mon, Feb 8, 2016 at 3:18 PM, m.roth@5-cent.us wrote:
Chris Murphy wrote:
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
dban doesn't? What F/OSS does "secure erase"? And does it do what dban's DoD 5220.22-M does?
That DoD standard is also obsolete per NIST 800-88. There's zero evidence provided that 2 passes makes any difference compared to 1, let alone doing 7.
hdparm --security-help
This takes the form of something like:
hdparm --user-master u --set-security-pass chickens /dev/sdX hdparm --user-master u --security-erase-enhanced chickens /dev/sdX
The 2nd command doesn't return until completion. hdparm -I can give an estimate of how long it will take. For HDDs I've found it slightly overestimates how long it will take, but is generally pretty close. For SSD's it can be way off. It says 8 minutes for my SSD, but the command returns in 5 seconds and the SSD spits back all zeros.
Secure erase is really the only thing to use on SSDs. Writing a pile of zeros just increases wear (minor negative) but also doesn't actually set the cells to the state required to accept a new write, so you've just added a lot more work for the SSD's garbage collector and wear leveling, so it's going to be slower than before you did the zeroing. Secure erase on an SSD erases the cells so they're ready to accept writes.
On Mon, Feb 8, 2016 at 10:54 PM, Chris Murphy lists@colorremedies.com wrote:
Secure erase is really the only thing to use on SSDs.
Oops. It's probably a fairly close approximation to just mkfs.btrfs -f (or xfs) the entire block device for the SSD. If the kernel sees it as non-rotational, it'll issue a whole device trim first, then write out scant amount of metadata (btrfs writes out a tiny amount of metadata at mkfs time, xfs a bit more, ext4 a lot and then even more after mounting).
For most people this is probably a lot easier than the multistep process using hdparm and secure erase.
On 2/8/2016 9:54 PM, Chris Murphy wrote:
Secure erase is really the only thing to use on SSDs. Writing a pile of zeros just increases wear (minor negative) but also doesn't actually set the cells to the state required to accept a new write, so you've just added a lot more work for the SSD's garbage collector and wear leveling, so it's going to be slower than before you did the zeroing. Secure erase on an SSD erases the cells so they're ready to accept writes.
at least one SSD I had, the vendor told me writing a full pass of zeros on it via dd or whatever would completely reset the garbage collection and effectively defrag it.
On Mon, Feb 8, 2016 at 11:18 PM, John R Pierce pierce@hogranch.com wrote:
On 2/8/2016 9:54 PM, Chris Murphy wrote:
Secure erase is really the only thing to use on SSDs. Writing a pile of zeros just increases wear (minor negative) but also doesn't actually set the cells to the state required to accept a new write, so you've just added a lot more work for the SSD's garbage collector and wear leveling, so it's going to be slower than before you did the zeroing. Secure erase on an SSD erases the cells so they're ready to accept writes.
at least one SSD I had, the vendor told me writing a full pass of zeros on it via dd or whatever would completely reset the garbage collection and effectively defrag it.
Yes it'd be "defragged" in that it has no file system at all to be fragmented in the first place (easier done with a mkfs). But a huge percent of the available cells on the drive (the portion not overprivisioned) would contain valid data (zeros) as far as the drive firmware is concerned, and those cells storing zeros are not in a state to accept writes. So unless it's a very good SSD that's so overprovisioned that it can perform well without the benefit of trim, and some can, this is odd advice. It's much simpler to just do a full device mkfs, which will do a whole device trim, and if you want you can then use wipefs to remove that filesystem's signature.
On 02/08/16 23:54, Chris Murphy wrote: <<>>
hdparm --user-master u --set-security-pass chickens /dev/sdX
centos 6.7, hdparm v- 9.43;
hdparm --user-master u --security-set-pass chickens /dev/sdX
On 2/8/2016 2:14 PM, Chris Murphy wrote:
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
John R Pierce wrote:
On 2/8/2016 2:14 PM, Chris Murphy wrote:
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
Well, no. DeGaussing works, also. Plus, with current storage techniques, I've heard that one pass of whatever will make it unreadable. That was why I jokingly referred to DoD 5220.22-M as overkill, since it does 7 passes.
mark
On Mon, February 8, 2016 4:22 pm, John R Pierce wrote:
On 2/8/2016 2:14 PM, Chris Murphy wrote:
DBAN is obsolete. NIST 800-88 for some time now says to use secure erase or enhanced security erase or crypto erase if supported.
Other options do not erase data in remapped sectors.
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
Without any intent to contradict... This article I found to be very instructive reading:
https://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html
As far as hard drives are concerned, it changed my mind about magnetic media (hard drives): from "you never will be able to securely destroy data" to "one pass of writing zeroes is sufficient for modern drives. Bringing platters over 1000 times deep into hysteresis back and forth is enough to destroy even residual magnetization related to magnetic domain aging... On modern drives though... No, I decided to not spoil it for those who decides to read that article. One thing I learned from there: DRAM had more persistent imprint of information that was sitting in it, which appears much harder to destroy than information on hard drive. I hope I intrigued you enough to go and read that article.
Valeri
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 2/8/2016 3:33 PM, Valeri Galtsev wrote:
DRAM had more persistent imprint of information that was sitting in it, which appears much harder to destroy than information on hard drive.
well aware of that. 30 years ago a friend and I built a specialized video card for a consulting project.... discovered on the prototype that we could power it down, go home, come back the next day, and power it up, and sufficient image remained in the DRAM to be totally recognizable. sure, it had 'noise' bits all over it, but whatever was on the screen the day before was still visible.
On Mon, February 8, 2016 5:45 pm, John R Pierce wrote:
On 2/8/2016 3:33 PM, Valeri Galtsev wrote:
DRAM had more persistent imprint of information that was sitting in it, which appears much harder to destroy than information on hard drive.
well aware of that. 30 years ago a friend and I built a specialized video card for a consulting project.... discovered on the prototype that we could power it down, go home, come back the next day, and power it up, and sufficient image remained in the DRAM to be totally recognizable. sure, it had 'noise' bits all over it, but whatever was on the screen the day before was still visible.
This is amazing! That is what distinguishes experts from us mortals: what we just learned by reading they saw themselves (sometimes quite a while ago)! On the other hand, why I'm so surprised? This is why they can answer any of our questions on this list in a split second.
Still, I'm very impressed!
Valeri
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On Mon, 2016-02-08 at 14:22 -0800, John R Pierce wrote:
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
I can't afford a machine that grinds everything into dust particles.
On 02/08/2016 06:38 PM, Always Learning wrote:
On Mon, 2016-02-08 at 14:22 -0800, John R Pierce wrote:
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
I can't afford a machine that grinds everything into dust particles.
I have a sledge hammer. It's good exercise and fun.
On 02/08/2016 07:38 PM, Always Learning wrote:
On Mon, 2016-02-08 at 14:22 -0800, John R Pierce wrote:
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
I can't afford a machine that grinds everything into dust particles.
I have seen a guy in the I.T. department where I work take a hard drive....remove as many stickers from it (to make it porous as possible) and then soak it in a "brine" of ammonia, hydrogen peroxide, bleach, and salted water...for like a week or two. Afterwards? total useless junk.
EGO II
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of EGO-II.1 Sent: den 9 februari 2016 09:00 To: CentOS mailing list Subject: Re: [CentOS] Utility to zero unused blocks on disk
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
I can't afford a machine that grinds everything into dust particles.
I have seen a guy in the I.T. department where I work take a hard drive....remove as many stickers from it (to make it porous as possible) and then soak it in a "brine" of ammonia, hydrogen peroxide, bleach, and salted water...for like a week or two. Afterwards? total useless junk.
I often find harddrives with several 10 mm drill-holes in them, in the electronics disposal room at work.
I'd say that'd be enough as well. 8-)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 09/02/16 08:28, Sorin Srbu wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of EGO-II.1 Sent: den 9 februari 2016 09:00 To: CentOS mailing list Subject: Re: [CentOS] Utility to zero unused blocks on disk
the only truly safe way to destroy data on magnetic media is to grind the media up into filings or melt it down in a furnace.
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
I can't afford a machine that grinds everything into dust particles.
I have seen a guy in the I.T. department where I work take a hard drive....remove as many stickers from it (to make it porous as possible) and then soak it in a "brine" of ammonia, hydrogen peroxide, bleach, and salted water...for like a week or two. Afterwards? total useless junk.
I often find harddrives with several 10 mm drill-holes in them, in the electronics disposal room at work.
I'd say that'd be enough as well. 8-)
At home I just collect up any old disks, memory sticks, credit cards and anything else with storage from the family and keep them until I am burning garden rubbish. Once there is a nice orange- or yellow- hot firebed in the incinerator I throw them in. Sometimes the aluminium runs out of the bottom, all I find in the ashes is the odd bit of steel. I'd like to see the CIA read that lot!
On 02/08/2016 07:38 PM, Always Learning wrote:
I unscrew the casing, extract the disk platter(s), slide a very strong magnet over both sides of the platter surface then bend the platter in half.
How secure is that ?
Actually, while there is some good security to that it's not for the reason you might suspect. Just taking the platters out is enough, for most drives, unless you have the proper precision jigs necessary to get them properly realigned.
But a static permanent magnet can leave a lot more than you think; degaussing is almost always done with alternating magnetic fields and exponential decay. The exponential decay is the key, and a proper set of passes with a permanent magnet taken at roughly exponentially-decaying heights will do the job (screwdriver demagnetizers have steps in their casing to do this). An AC degausser needs to be kept on while removing from the media to fully degauss (or you end up with more residual magnetism (hysteresis) than can be overcome by the heads).
Piranha solution does a number on all known magnetic media (and skin), but bringing the media up to transition temperature will also effectively degauss. So melting it down is guaranteed, and fun (but remove the printed circuit board and all other plastics first......).
On 02/08/16 15:34, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
. a comment on replies to your post.
i find it interesting that "Subject:" is and you ask for info to
*zero unused blocks on disk*
and all the replies for wiping the _entire_ disk.
why is so hard to understand that _blocks_ does not mean _disk_. ((GBWG))
On 09/02/16 12:08 AM, g wrote:
On 02/08/16 15:34, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
. a comment on replies to your post.
i find it interesting that "Subject:" is and you ask for info to
*zero unused blocks on disk*
and all the replies for wiping the _entire_ disk.
why is so hard to understand that _blocks_ does not mean _disk_. ((GBWG))
Not all of them.
On 02/08/16 23:10, Digimer wrote:
On 09/02/16 12:08 AM, g wrote:
On 02/08/16 15:34, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
. a comment on replies to your post.
i find it interesting that "Subject:" is and you ask for info to
*zero unused blocks on disk*
and all the replies for wiping the _entire_ disk.
why is so hard to understand that _blocks_ does not mean _disk_. ((GBWG))
Not all of them.
. this is true.
also true is that i phrased with;
and all the replies for
not,
with all replies for
because, yes, i did read you reply with all the rest. ;-)
fell better? :-P
On 09/02/16 12:24 AM, g wrote:
On 02/08/16 23:10, Digimer wrote:
On 09/02/16 12:08 AM, g wrote:
On 02/08/16 15:34, Wes James wrote:
Is there a utility to zero unused blocks on a disk?
CentOS 6.7/Ext4
I saw zerofree, but I’m not sure it would work on Ext4 or even work on this version of CentOS.
thanks,
. a comment on replies to your post.
i find it interesting that "Subject:" is and you ask for info to
*zero unused blocks on disk*
and all the replies for wiping the _entire_ disk.
why is so hard to understand that _blocks_ does not mean _disk_. ((GBWG))
Not all of them.
. this is true.
also true is that i phrased with;
and all the replies for
not,
with all replies for
because, yes, i did read you reply with all the rest. ;-)
fell better? :-P
I need ice cream to feel better. ;)
On 02/08/16 23:39, Digimer wrote:
On 09/02/16 12:24 AM, g wrote:
<<>>
fell better? :-P
I need ice cream to feel better. ;)
. Baskin-Robbins 32 flavors + special of month.
On 2/9/2016 12:42 AM, g wrote:
On 02/08/16 23:39, Digimer wrote:
I need ice cream to feel better.;)
. Baskin-Robbins 32 flavors + special of month.
After seeing Aaron Neville tonight (wow, what a show, 3rd row center seats in a small theater, we were 15' from the stage), we stopped at The Penny Ice Creamery, and I had a small cone of Meyers Lemon Sorbet. very yum, intensely fresh lemon.
On 02/09/16 02:51, John R Pierce wrote:
On 2/9/2016 12:42 AM, g wrote:
On 02/08/16 23:39, Digimer wrote:
I need ice cream to feel better.;)
. Baskin-Robbins 32 flavors + special of month.
After seeing Aaron Neville tonight (wow, what a show, 3rd row center seats in a small theater, we were 15' from the stage).
"Tell It Like It Is". a great song by a great singer and group.
i meet the neville brothers in new orleans, la back in mid 60's just after the second hurricane pushed thru.
we stopped at The Penny Ice Creamery, and I had a small cone of Meyers Lemon Sorbet. very yum, intensely fresh lemon.
. sorbet is good, but it is not ice cream, it is ice water.
strawberry and raspberry are my 2 favorite.
-
Always Learning -
Chris Adams -
Chris Murphy -
Digimer -
EGO-II.1 -
g -
Greg Bailey -
J Martin Rushton -
John R Pierce -
Lamar Owen -
m.roth＠5-cent.us -
Robert Nichols -
Sorin Srbu -
SternData -
Valeri Galtsev -
Wes James