Strahil, I am using about 1012MB for the first ISO. I believe it's because of compression. From there vdostats --hu reports 5.0G usage and 12% in percentage. With savings of 89% for original + 9 copies of the same ISO. On Sun, May 3, 2020 at 1:17 AM Strahil Nikolov <hunter86_bg at yahoo.com> wrote: > On May 3, 2020 8:33:33 AM GMT+03:00, Erick Perez - Quadrian Enterprises < > eperez at quadrianweb.com> wrote: > >sorry corrections: > >For this test I created a 40GB lvm volume group with /dev/sdb and > >/dev/sdc > >then a 40GB LV > >then a 60GB VDO vol (for testing purposes) > > > >vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > >output from just created vdoas > > > >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 > >'saving > >percent' > >physical blocks : 10483712 > > logical blocks : 15728640 > > 1K-blocks : 41934848 > > 1K-blocks used : 4212024 > > 1K-blocks available : 37722824 > > used percent : 10 > > saving percent : 99 > >[root at localhost ~]# > > > >FIRST copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas from source > >outside vdo volume > >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 > >'saving > >percent' > > 1K-blocks used : 4721348 > > 1K-blocks available : 37213500 > > used percent : 11 > > saving percent : 9 > > > >SECOND copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas form > >source > >outside vdo volume > >#cp /root/CentOS-7-x86_64-Minimal-2003.iso > >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > > 1K-blocks used : 5239012 > > 1K-blocks available : 36695836 > > used percent : 12 > > saving percent : 52 > > > >THIRD copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to > >vdoas form inside vdo volume to inside vdo volume > > 1K-blocks used : 5248060 > > 1K-blocks available : 36686788 > > used percent : 12 > > saving percent : 67 > > > >Then I did this a total of 9 more times to have 10 ISOs copied. Total > >data > >copied 10.6GB. > > > > > >Do note this: > >When using DF, it will show the VDO size, in my case 60G > >when using vdostats it will show the size of the LV, in my case 40G > >Remeber dedupe AND compression are enabled. > > > >The df -hT output shows the logical space occupied by these iso files > >as > >seen by the filesystem on the VDO volume. > >Since VDO manages a logical to physical block map, df sees logical > >space > >consumed according to the file system that resides on top of the VDO > >volume. > >vdostats --hu is viewing the physical block device as managed by VDO. > >Physically a single .ISO image is residing on the disk, but logically > >the > >file system thinks there are 10 copies, occupying 10.6GB. > > > >So at the end I have 10 .ISOs of 1086 1MB blocks (total 10860 1MB > >blocks) > >that yield these results: > > 1K-blocks used : 5248212 > > 1K-blocks available : 36686636 > > used percent : 12 > > saving percent : 89 > > > >So at the end it is using 5248212 1K blocks minus 4212024 initial > >used 1K > >blocks, gives (5248212 - 4212024) = 1036188 1K blocks / 1024 = about > >1012MB > >total. > > > >Hope this helps understanding where the space goes. > > > >BTW: Testing system is CentOS Linux release 7.8.2003 stock. with only > >"yum > >install vdo kmod-kvdo" > > > >History of commands: > >[root at localhost vdomounts]# history > > 2 pvcreate /dev/sdb > > 3 pvcreate /dev/sdc > > 8 vgcreate -v -A y vgvol01 /dev/sdb /dev/sdc > > 9 vgdisplay > > 13 lvcreate -l 100%FREE -n lvvdo01 vgvol01 > > 14 yum install vdo kmod-kvdo > > 18 vdo create --name=vdoas --device=/dev/vgvol01/lvvdo01 > >--vdoLogicalSize=60G --writePolicy=async > > 19 mkfs.xfs -K /dev/mapper/vdoas > > 20 ls /mnt > > 21 mkdir /mnt/vdomounts > > 22 mount /dev/mapper/vdoas /mnt//vdomounts/ > > 26 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 28 cp /root/CentOS-7-x86_64-Minimal-2003.iso /mnt/vdomounts/ -vvv > > 29 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 30 cp /root/CentOS-7-x86_64-Minimal-2003.iso > >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > > 31 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 33 cd /mnt/vdomounts/ > > 35 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version3.iso > > 36 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 37 df > > 39 vdostats --hu > > 40 ls -l --block-size=1MB /root/CentOS-7-x86_64-Minimal-2003.iso > > 41 df -hT > > 42 vdo status | grep Dedupl > > 43 vdostats --hu > > 44 vdostats > > 48 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version4.iso > > 49 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version5.iso > > 50 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version6.iso > > 51 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version7.iso > > 52 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version8.iso > > 53 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version9.iso > > 54 df -hT > > 55 ls -l --block-size=1MB > > 56 vdostats --hu > > 57 df -hT > > 58 df > > 59 vdostats --hu > > 60 vdostats > > 61 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 62 cat /etc/centos-release > > 63 history > >[root at localhost vdomounts]# > > > > > > > > > > > >On Sat, May 2, 2020 at 10:07 PM Erick Perez - Quadrian Enterprises < > >eperez at quadrianweb.com> wrote: > > > >> My two cents: > >> 1- Do you have an encrypted filesystem on top of VDO? If yes, you > >will see > >> no benefit from dedupe. > >> 2- can you post the stats of vdostats –verbose /dev/mapper/xxxxx > >(replace > >> with your device) > >> > >> you can do something like: "vdostats -verbose /dev/mapper/xxxxxxxx | > >grep > >> -B6 'save percentage' > >> > >> > >> > >> > >> On Sat, May 2, 2020 at 9:54 PM david <david at daku.org> wrote: > >> > >>> Folks > >>> > >>> I'm looking for a solution for backups because ZFS has failed on me > >>> too many times. In my environment, I have a large amount of data > >>> (around 2tb) that I periodically back up. I keep the last 5 > >>> "snapshots". I use rsync so that when I overwrite the oldest > >backup, > >>> most of the data is already there and the backup completes quickly, > >>> because only a small number of files have actually changed. > >>> > >>> Because of this low change rate, I have used ZFS with its > >>> deduplication feature to store the data. I started using a Centos-6 > >>> installation, and upgraded years ago to Centos7. Centos 8 is on my > >>> agenda. However, I've had several data-loss events with ZFS where > >>> because of a combination of errors and/or mistakes, the entire store > >>> was lost. I've also noticed that ZFS is maintained separately from > >>> Centos. At this moment, the Centos 8 update causes ZFS to > >>> fail. Looking for an alternate, I'm trying VDO. > >>> > >>> In the VDO installation, I created a logical volume containing two > >>> hard-drives, and defined VDO on top of that logical volume. It > >>> appears to be running, yet I find the deduplication numbers don't > >>> pass the smell test. I would expect that if the logical volume > >>> contains three copies of essentially identical data, I should see > >>> deduplication numbers close to 3.00, but instead I'm seeing numbers > >>> like 1.15. I compute the compression number as follows: > >>> Use df and extract the value for "1k blocks used" from the third > >column > >>> use vdostats --verbose and extract the number titled "1K-blocks > >used" > >>> > >>> Divide the first by the second. > >>> > >>> Can you provide any advice on my use of ZFS or VDO without telling > >me > >>> that I should be doing backups differently? > >>> > >>> Thanks > >>> > >>> David > >>> > >>> _______________________________________________ > >>> CentOS mailing list > >>> CentOS at centos.org > >>> https://lists.centos.org/mailman/listinfo/centos > >>> > >> > >> > >> -- > >> > >> --------------------- > >> Erick Perez > >> > >> > > > >-- > > > >--------------------- > >Erick Perez > >Quadrian Enterprises S.A. - Panama, Republica de Panama > >Skype chat: eaperezh > >WhatsApp IM: +507-6675-5083 > >--------------------- > >_______________________________________________ > >CentOS mailing list > >CentOS at centos.org > >https://lists.centos.org/mailman/listinfo/centos > > Hi Erick, > > In the end how much space is used as per vdostats ? > Keep in mind that VDO keeps one slab for itself (default is 2GB) and also > some metadata is written : > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-qs-requirements#vdo-ig-storage-reqts > > Best Regards, > Strahil Nikolov > -- --------------------- Erick Perez Quadrian Enterprises S.A. - Panama, Republica de Panama Skype chat: eaperezh WhatsApp IM: +507-6675-5083 ---------------------