<html><head><style type='text/css'>body { font-family: 'Arial'; font-size: 12pt; color: #000000}</style></head><body><br>----- Original Message -----<br>From: rsivak@istandfor.com<br>To: "CentOS Mailing list" <centos@centos.org><br>Sent: Thursday, December 6, 2007 11:18:16 AM (GMT+1000) Australia/Brisbane<br>Subject: [CentOS] Filesystem that doesn't store duplicate data<br><br>Is there such a filesystem available? It seems like it wouldn't be too hard to implement... Basically do things on a block by block basis. Store md5 of a block in the table, and when writing a new block, check if the md5 already exists and then point the new block to the old block. Since md5 is not guaranteed unique, might need to do a diff between the 2 blocks and if the blocks are indeed different, handle it somehow. <br><br>When modifying an existing block that has multiple pointers, copy the block and modify the new block. <br><br>I know I'm oversimplifying things a lot, but something like this could work, no? Would be a great filesystem to store backups on, or things like vmware volumes... <br><br>Russ<br>Sent from my Verizon Wireless BlackBerry<br>_______________________________________________<br>CentOS mailing list<br>CentOS@centos.org<br>http://lists.centos.org/mailman/listinfo/centos<br><br>-- <br>This message has been scanned for viruses and<br>dangerous content by MailScanner, and is<br>believed to be clean.<br><br><br>You are describing what I understand to be 'Data De-duplication". It is all the rage for backups as it has the potential to decrease backup times and volumes by significant amounts. I went to a presentation by Avamar (a partner of EMC ?) regarding this technology and it seemed really nice for your typical windows file server. I suppose it effectively turns your data into 'single-instance' which is no bad thing. I suppose it could be useful for large database backups as well.<br><br>You'd think that using this technology on a live filesystem could incur a significant performance penalty due to all those calculations (fuse module anyone ?). Imagine a hardware optimized data de-duplication disk controller, similar to raid XOR optimized cpus. Now that would be cool. All it would need to store was meta-data when it had already seen the exact same block. I think fundamentally it is similar in result to on the fly disk compression. <br><br>Let us know when you have a beta to test !<br><br>8^)<br><br><br><br><br><br></body><br />--
<br />This message has been scanned for viruses and
<br />dangerous content by
<a href="http://www.mailscanner.info/"><b>MailScanner</b></a>, and is
<br />believed to be clean.
</html>