On 07/03/2014 09:48 PM, Lists wrote:
On 07/03/2014 12:19 PM, John R Pierce wrote:
you do realize, adding/removing or even changing the length of a single line in a block of that pg_dump file will change every block after it as the data will be offset ?
Yes. And I guess this is probably where the conversation should end. I'm used to the capabilities of Mercurial DVCS as well as ZFS snapshots, and was thinking/hoping that this type of capability might exist in a file system. Perhaps it just doesn't belong there.
On 07/03/2014 12:23 PM, Les Mikesell wrote:
But, since this is about postgresql, the right way is probably just to set up replication and let it send the changes itself instead of doing frequent dumps.
Whatever we do, we need the ability to create a point-in-time history. We commonly use our archival dumps for audit, testing, and debugging purposes. I don't think PG + WAL provides this type of capability. So at the moment we're down to:
A) run PG on a ZFS partition and snapshot ZFS. B) Keep making dumps (as now) and use lots of disk space. C) Cook something new and magical using diff, rdiff-backup, or related tools.
Check out 7z from p7zip package. I use command:
7za a -t7z $YearNum-$MonthNum.7z -i@include.lst -mx$CompressionMetod -mmt$ThreadNumber -mtc=on
for compressing a lot of similar files from sysinfo and kernel.log, files backuped every hour that do not change much. And I find out that it reuses already existing blocks/hashes/whatever and I guess just reference them with a pointer instead of storing them again.
So, 742 files that uncompressed have 179 MB, compressed ocupy only 452 KB, which is only 0.2% of original size, 442 TIMES smaller :
Listing archive: 2014-03.7z
-- Path = 2014-03.7z Type = 7z Method = LZMA Solid = + Blocks = 1 Physical Size = 426210 Headers Size = 9231
Date Time Attr Size Compressed Name ------------------- ----- ------------ ------------ ------------------------ 2014-03-01 00:02:07 ....A 259517 416979 Silos-Srbobran-l5-kernlog-2014-03-01-00-02-07.txt 2014-03-01 01:01:52 ....A 259529 Silos-Srbobran-l5-kernlog-2014-03-01-01-01-52.txt ............................................... ............................................... ............................................... 2014-03-31 22:01:33 ....A 259502 Silos-Srbobran-l5-kernlog-2014-03-31-22-01-33.txt 2014-03-31 23:01:33 ....A 259485 Silos-Srbobran-l5-kernlog-2014-03-31-23-01-32.txt ------------------- ----- ------------ ------------ ------------------------ 184553028 416979 742 files, 0 folders
Maybe, if you compress them into the same file (afterwords?) you can get the similar result.