[CentOS] drbd

Sun Oct 12 06:30:53 UTC 2014
John R Pierce <pierce at hogranch.com>

so I've had a drbd replica running for a while of a 16TB raid thats used 
as a backuppc repository.

when I have rebooted the backuppc server, the replica doesn't seem to 
auto-restart til I do it manually, and the backupc /data file system on 
this 16TB LUN doesn't seem to automount, either.

I've rebooted this thing a few times in the 18 months or so its been 
running...   not always cleanly...

anyways, I'm started a drbd verify (from the slave) about 10 hours ago, 
it has 15 hours more to run, and so far it's logged...

Oct 11 13:58:26 sg2 kernel: block drbd0: Starting Online Verify from 
sector 3534084704
Oct 11 14:00:23 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 14:00:29 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967294
Oct 11 14:00:35 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967293
Oct 11 14:00:41 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967292
Oct 11 14:01:16 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 14:02:05 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 14:02:11 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967294
Oct 11 14:02:17 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967293
Oct 11 14:33:41 sg2 kernel: block drbd0: Out of sync: start=3932979480, 
size=8 (sectors)
Oct 11 14:34:46 sg2 kernel: block drbd0: Out of sync: start=3946056120, 
size=8 (sectors)
Oct 11 15:37:07 sg2 kernel: block drbd0: Out of sync: start=4696809024, 
size=8 (sectors)
Oct 11 17:08:15 sg2 kernel: block drbd0: Out of sync: start=6084949528, 
size=8 (sectors)
Oct 11 17:30:53 sg2 kernel: block drbd0: Out of sync: start=6567543472, 
size=8 (sectors)
Oct 11 17:59:04 sg2 kernel: block drbd0: Out of sync: start=7169767896, 
size=8 (sectors)
Oct 11 20:00:50 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 20:01:09 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 20:01:15 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967294
Oct 11 20:01:29 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 20:29:18 sg2 kernel: block drbd0: Out of sync: start=10362907296, 
size=8 (sectors)
Oct 11 20:29:54 sg2 kernel: block drbd0: Out of sync: start=10375790488, 
size=8 (sectors)
Oct 11 21:01:51 sg2 kernel: block drbd0: [drbd0_worker/2197] 
sock_sendmsg time expired, ko = 4294967295
Oct 11 21:42:15 sg2 kernel: block drbd0: Out of sync: start=11907921096, 
size=8 (sectors)
Oct 11 21:43:38 sg2 kernel: block drbd0: Out of sync: start=11937086248, 
size=8 (sectors)
Oct 11 21:44:00 sg2 kernel: block drbd0: Out of sync: start=11944705032, 
size=8 (sectors)
Oct 11 21:49:26 sg2 kernel: block drbd0: Out of sync: start=12062270432, 
size=8 (sectors)
Oct 11 22:07:10 sg2 kernel: block drbd0: Out of sync: start=12440235128, 
size=8 (sectors)
Oct 11 22:58:54 sg2 kernel: block drbd0: Out of sync: start=13548501984, 
size=8 (sectors)
Oct 11 23:23:17 sg2 kernel: block drbd0: Out of sync: start=14072873320, 
size=8 (sectors)
$ date
Sat Oct 11 23:28:11 PDT 2014

its 35% done at this point...   15 4K blocks out wrong of 1/3rd of 16TB 
isn't a lot, but its still more than I like to see.

$ cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by 
phil at Build64R6, 2012-12-20 20:09:51
  0: cs:VerifyS ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
     ns:0 nr:105707 dw:187685496 dr:654444832 al:0 bm:1 lo:107 pe:2104 
ua:435 ap:0 ep:1 wo:f oos:60
         [=====>..............] verified: 34.6% (9846140/15051076)M
         finish: 14:55:27 speed: 187,648 (155,708) want: 204,800 K/sec




really, if I let this complete, then disconnect/reconnect the replica, 
it will repair these glitches ?   I'm gathering I shoudl schedule these 
verifies weekly or something.




-- 
john r pierce                                      37N 122W
somewhere on the middle of the left coast