Discuss

discuss@lists.centos.org

5 participants
34752 discussions

NFS/RDMA connection closed
by admin＠genome.arizona.edu 02 Aug '18

02 Aug '18

Hi, we are having a problem with NFS using RDMA protocol over our FDR10 Infiniband network. I previously wrote to NFS mailing list about this, so you may find our discussion there. I have taken some load off the server which was using NFS for backups, and converted it to use SSH, but we are still having critical problems with NFS clients losing connection to the server, causing the clients to hang and needing a reboot. I wanted to check in here before filing a bug with CentOS. Our setup is a cluster with one head node (NFS server) and 9 compute nodes (NFS clients). All the machines are running CentOS 6.9 2.6.32-696.30.1.el6.x86_64 and using the "Inbox"/CentOS RDMA implementation/drivers (not Mellanox OFED). (We also have other NFS clients but they are using 1GbE for NFS connection and, while they will still hang with messages like "NFS server not responding, retrying" or "timed out", they will eventually recover and don't need a reboot.) On the server (which is named pac) I will see messages like this: Jul 30 18:19:38 pac kernel: svcrdma: failed to send reply chunks, rc=-5 Jul 30 18:19:38 pac kernel: svcrdma: failed to send write chunks, rc=-5 Jul 31 15:03:05 pac kernel: svcrdma: failed to send write chunks, rc=-5 Jul 31 15:09:06 pac kernel: svcrdma: failed to send write chunks, rc=-5 Jul 31 15:16:09 pac kernel: svcrdma: failed to send write chunks, rc=-5 Jul 31 15:23:31 pac kernel: svcrdma: Error -107 posting RDMA_READ Jul 31 15:53:55 pac kernel: svcrdma: failed to send write chunks, rc=-5 Jul 31 16:09:19 pac kernel: svcrdma: failed to send reply chunks, rc=-5 Jul 31 16:09:19 pac kernel: svcrdma: failed to send reply chunks, rc=-5 Previously I had also seen messages like "Jul 11 21:09:56 pac kernel: nfsd: peername failed (err 107)!" however have not seen that in this latest hangup. And on the clients (named n001-n009) I will see messages like this: Jul 30 18:17:26 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810674024c0 (stale): WR flushed Jul 30 18:17:26 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff88106638a640 (stale): WR flushed Jul 30 18:19:26 n001 kernel: nfs: server 10.10.11.100 not responding, still trying Jul 30 18:19:36 n001 kernel: nfs: server 10.10.10.100 not responding, timed out Jul 30 18:19:38 n001 kernel: rpcrdma: connection to 10.10.11.100:20049 on mlx4_0, memreg 5 slots 32 ird 16 Jul 30 18:19:38 n001 kernel: nfs: server 10.10.11.100 OK Jul 31 14:42:08 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810671f02c0 (stale): WR flushed Jul 31 14:42:08 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810677bda40 (stale): WR flushed Jul 31 14:42:08 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810677bd940 (stale): WR flushed Jul 31 14:42:08 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810671f0240 (stale): WR flushed Jul 31 14:43:35 n001 kernel: rpcrdma: connection to 10.10.11.100:20049 on mlx4_0, memreg 5 slots 32 ird 16 Jul 31 15:01:53 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff881065133140 (stale): WR flushed Jul 31 15:01:53 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810666e3f00 (stale): WR flushed Jul 31 15:01:53 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff881063ea0dc0 (stale): WR flushed Jul 31 15:01:53 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810677bdb40 (stale): WR flushed Jul 31 15:03:05 n001 kernel: rpcrdma: connection to 10.10.11.100:20049 on mlx4_0, memreg 5 slots 32 ird 16 Jul 31 15:07:07 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff881060e59d40 (stale): WR flushed Jul 31 15:07:07 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810677efac0 (stale): WR flushed Jul 31 15:07:07 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff88106638a640 (stale): WR flushed Jul 31 15:07:07 n001 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810671f03c0 (stale): WR flushed Jul 31 15:09:06 n001 kernel: rpcrdma: connection to 10.10.11.100:20049 on mlx4_0, memreg 5 slots 32 ird 16 Jul 31 15:16:09 n001 kernel: rpcrdma: connection to 10.10.11.100:20049 closed (-103) Jul 31 15:53:32 n001 kernel: nfs: server 10.10.10.100 not responding, timed out Jul 31 16:08:56 n001 kernel: nfs: server 10.10.10.100 not responding, timed out Jul 30 18:17:26 n002 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff881064461500 (stale): WR flushed Jul 30 18:17:26 n002 kernel: RPC: rpcrdma_sendcq_process_wc: frmr ffff8810604b2600 (stale): WR flushed Jul 30 18:19:26 n002 kernel: nfs: server 10.10.11.100 not responding, still trying Jul 30 18:19:38 n002 kernel: rpcrdma: connection to 10.10.11.100:20049 on mlx4_0, memreg 5 slots 32 ird 16 Jul 30 18:19:38 n002 kernel: nfs: server 10.10.11.100 OK Jul 31 14:43:35 n002 kernel: rpcrdma: connection to 10.10.11.100:20049 closed (-103) Jul 31 16:08:56 n002 kernel: nfs: server 10.10.10.100 not responding, timed out Similar messages show up on the other clients n003-n009. After these messages on the clients, their load will continually go up (viewable through Ganglia) (I would guess since they are waiting for NFS mount to re-appear). They aren't reachable any longer through SSH and neither can root log in through console via IPMI web applet (just hangs after entering password, may get to prompt eventually but system load is so high), they need to be rebooted through IPMI interface. Here is /etc/fstab on the server, UUID=f15df051-ffb8-408c-8ad2-1987b6f082a2 / ext3 defaults 0 1 UUID=c854ee27-32cf-445d-8308-4e6f1a87d364 /boot ext3 defaults 0 2 UUID=b92a100f-2521-408b-9b15-93671c6ae056 swap swap defaults 0 0 UUID=a8a7b737-25ed-43a7-ae4b-391c71aa8c08 /data xfs defaults 0 2 UUID=d5692ec2-d5dc-4bb8-98d4-a4fb2ff54748 /projects xfs defaults 0 2 /dev/drbd0 /newwing xfs noauto 0 0 UUID=a305f309-d997-43ec-8e4f-78e26b07652f /working xfs defaults 0 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 I read that adding "inode64,nobarrier" for the xfs mount options may help? That is something I can try once the server can be rebooted. Here is current mounts on the server, /dev/sda3 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext3 (rw) /dev/sdc1 on /data type xfs (rw) /dev/sdb1 on /projects type xfs (rw) /dev/sde1 on /working type xfs (rw,nobarrier) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /dev/drbd0 on /newwing type xfs (rw) Here is /etc/exports on the server, /data 10.10.10.0/24(rw,no_root_squash,async) /data 10.10.11.0/24(rw,no_root_squash,async) /data 150.x.x.192/27(rw,no_root_squash,async) /data 150.x.x.64/26(rw,no_root_squash,async) /home 10.10.10.0/24(rw,no_root_squash,async) /home 10.10.11.0/24(rw,no_root_squash,async) /opt 10.10.10.0/24(rw,no_root_squash,async) /opt 10.10.11.0/24(rw,no_root_squash,async) /projects 10.10.10.0/24(rw,no_root_squash,async) /projects 10.10.11.0/24(rw,no_root_squash,async) /projects 150.x.x.192/27(rw,no_root_squash,async) /projects 150.x.x.64/26(rw,no_root_squash,async) /tools 10.10.10.0/24(rw,no_root_squash,async) /tools 10.10.11.0/24(rw,no_root_squash,async) /usr/share/gridengine 10.10.10.10/24(rw,no_root_squash,async) /usr/share/gridengine 10.10.11.10/24(rw,no_root_squash,async) /usr/local 10.10.10.10/24(rw,no_root_squash,async) /usr/local 10.10.11.10/24(rw,no_root_squash,async) /working 10.10.10.0/24(rw,no_root_squash,async) /working 10.10.11.0/24(rw,no_root_squash,async) /working 150.x.x.192/27(rw,no_root_squash,async) /working 150.x.x.64/26(rw,no_root_squash,async) /newwing 10.10.10.0/24(rw,no_root_squash,async) /newwing 10.10.11.0/24(rw,no_root_squash,async) /newwing 150.x.x.192/27(rw,no_root_squash,async) /newwing 150.x.x.64/26(rw,no_root_squash,async) The 10.10.10.0/24 network is 1GbE and the 10.10.11.0/24 is the Infiniband. The other networks are also 1GbE. Our cluster nodes will normally mount all of these using the Infiniband with RDMA and the computation jobs will normally be using /working which will see the most reading/writing but /newwing, /projects, and /data are also used. Here is an /etc/fstab from the nodes, #NFS/RDMA #10.10.11.100:/opt /opt nfs rdma,port=20049 0 0 #10.10.11.100:/data /data nfs rdma,port=20049 0 0 #10.10.11.100:/tools /tools nfs rdma,port=20049 0 0 #10.10.11.100:/home /home nfs rdma,port=20049 0 0 #10.10.11.100:/usr/local /usr/local nfs rdma,port=20049 0 0 #10.10.11.100:/usr/share/gridengine /usr/share/gridengine nfs rdma,port=20049 0 0 #10.10.11.100:/projects /projects nfs rdma,port=20049 0 0 #10.10.11.100:/working /working nfs rdma,port=20049 0 0 #10.10.11.100:/newwing /newwing nfs rdma,port=20049 0 0 #NFS/IPoIB 10.10.11.100:/opt /opt nfs tcp 0 0 10.10.11.100:/data /data nfs tcp 0 0 10.10.11.100:/tools /tools nfs tcp 0 0 10.10.11.100:/home /home nfs tcp 0 0 10.10.11.100:/usr/local /usr/local nfs tcp 0 0 10.10.11.100:/usr/share/gridengine /usr/share/gridengine nfs tcp 0 0 10.10.11.100:/projects /projects nfs tcp 0 0 10.10.11.100:/working /working nfs tcp 0 0 10.10.11.100:/newwing /newwing nfs tcp 0 0 #NFS/TCP #10.10.10.100:/opt /opt nfs defaults 0 0 #10.10.10.100:/data /data nfs defaults 0 0 #10.10.10.100:/tools /tools nfs defaults 0 0 #10.10.10.100:/home /home nfs defaults 0 0 #10.10.10.100:/usr/local /usr/local nfs defaults 0 0 #10.10.10.100:/usr/share/gridengine /usr/share/gridengine nfs defaults 0 0 #10.10.10.100:/projects /projects nfs defaults 0 0 #10.10.10.100:/working /working nfs defaults 0 0 #10.10.10.100:/newwing /newwing nfs defaults 0 0 Here I can switch between different interfaces/protocols for the NFS mounts. Currently we are trying the IPoIB. We haven't started a cluster job yet so not sure how it will perform. With the NFS/TCP over 1GbE the server/nodes would hang from time to time but still did not crash at least, however was of course slow being limited by 1GbE. We haven't had this problem until recently. I upgraded our cluster to add the two additional nodes (n008 and n009) and we also added more storage to the server (/newwing and /working). The new nodes are AMD EPYC platform whereas the server and the nodes n001-n007 are Intel Xeon platform, not sure if that would cause such a crash. The new nodes were cloned from n001 and only kernel command line and network parameters were changed. The jobs are submitted to the cluster via Sun Grid Engine, and in total there are about 61 jobs that may start at once and open connections to the NFS server... it sounds like it is a system overload, although the load on the server remains low, under 10%, even as it hangs the load may increase to 80%. The server is a few years old but still has 2x 6-core Intel Xeon E5-2620 v2 @ 2.10GHz with 128GB of RAM. Would appreciate your assistance to troubleshoot this critical problem and, if needed, gather the required information to submit a bug to the tracker! Thanks, -- Chandler Arizona Genomics Institute www.genome.arizona.edu

1 1

Desktop lag with CentOS7/Mate and high CPU usage
by James Pearson 02 Aug '18

02 Aug '18

We have noticed significant desktop/UI lag on CentOS7 workstations using Mate when the CPU usage is high - i.e. the mouse pointer lags and moving windows (e.g. terminal windows) become jumpy (not smooth) We didn't see (or notice) this issue with CentOS6/Gnome2 This can easily be shown by running something like 'cpuburn' (https://patrickmn.com/projects/cpuburn/) and moving a mate-terminal window around the screen We can make things 'better' by running CPU intensive apps via 'nice' (without any noticeable performance hit to the CPU intensive app) - but this is a rather messy way of fixing the issue We can also make things better by renice'ing X to a negative nice level However, both these approaches seem wrong to me - as I'm probably missing something straightforward here Does anyone know of any CentOS7/Mate settings that could be used to improve desktop/UI responsiveness with high CPU usage ? Thanks James Pearson

5 5

Screensavers on Centos-7
by Fred Smith 01 Aug '18

01 Aug '18

Hi all! I keep having problems with screensaver(s) on Centos-7. I have in the past liked Xscreensaver, so I've installed it and tried to suppress whatever other screensaver is on the system. Xscreensaver works fine for a day or 20 then quits working. After a lengthy battle I gave up and removed it. I'm now trying mate-screensaver, which does the same thing! when I say "quits working" I mean just that. one night it goes to a black screen (as configured) then a minute or two later the power manager shuts off the screen. The next night it does neither, and forever after. Sometimes after a reboot it'll work again for a while then it starts misbehaving again. First question: Anybody have any similar issues? Anybody have any ideas on what's going on here and what I can do about it? So, not being in a position to reboot, last night I killed mate-screensaver and mate-powermanager, with the hopes the daemons would restart themselves. No such luck. so I tried to figure out what programs I could run to get them started again. So far I'm too stupid to be able to figure that out, as well. Second question: Anybody able to clue me in on how the screensavers and power managers are managed/started? there isn't a systemctl module for it, as far as I can see. if I run mate-screensaver from the commandline (as me, not root) I get no prompt back, so I assume it's running silently, but it also doesn't blank the screen at the specified time. So, thanks in advance for any assistance! -- ---- Fred Smith -- fredex(a)fcshome.stoneham.ma.us ----------------------------- "Not everyone who says to me, 'Lord, Lord,' will enter the kingdom of heaven, but only he who does the will of my Father who is in heaven." ------------------------------ Matthew 7:21 (niv) -----------------------------

1 0

C7 Xfce group
by Robert Moskowitz 01 Aug '18

01 Aug '18

I am working with Centos-arm which right now only has the gnome and kde desktops. I am spoiled with using Xfce on all my Fedora systems (MUCH longer battery life on my notebook, for example). I like its simplicity. I see from messages here that mainline C7 does have the Xfce desktop. Can someone give me a list of rpms that make up the group, so I can try starting with the minimal image and see if I can build an Xfce desktop? Meanwhile I am going to put in a request for Xfce on the centos-arm list. thanks

2 3

Finding user's files
by mark 01 Aug '18

01 Aug '18

This is among the things we need to do when a user leaves, and it's a larger question than it sounds. Our Office has many servers, with a good number of fileservers for projects, with large filesystems (i.e. 10's of TB). Can anyone think of a way *other* than running what's probably a many-hour long find / -user on all our systems, which is really intensive, to find all the files own by a given user? Locate would be great, but from the man pages and what I can find online, it only stores filenames and paths. mark

4 4

Strange vncserver behavior
by Robert Moskowitz 01 Aug '18

01 Aug '18

I got vncserver working per: https://www.tecmint.com/install-and-configure-vnc-server-in-centos-7/ Then I went to set up a second instance, and have that for logging on to root: ========== cp /lib/systemd/system/vncserver@.service /etc/systemd/system/vncserver@:2.service vi /etc/systemd/system/vncserver@\:2.service replace <USER> with root systemctl daemon-reload systemctl start vncserver@:2 ========== I got the message: Job for vncserver@:2.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:2.service" and "journalctl -xe" for details. #systemctl -l status vncserver@:2.service ● vncserver@:2.service - Remote desktop service (VNC) Loaded: loaded (/etc/systemd/system/vncserver@:2.service; disabled; vendor preset: disabled) Active: failed (Result: resources) since Wed 2018-08-01 08:45:19 EDT; 1min 8s ago Process: 2267 ExecStart=/usr/sbin/runuser -l root -c /usr/bin/vncserver %i (code=exited, status=0/SUCCESS) Process: 2260 ExecStartPre=/bin/sh -c /usr/bin/vncserver -kill %i > /dev/null 2>&1 || : (code=exited, status=0/SUCCESS) Aug 01 08:45:14 localhost systemd[1]: Starting Remote desktop service (VNC)... Aug 01 08:45:19 localhost systemd[1]: PID file /home/root/.vnc/localhost:2.pid not readable (yet?) after start. Aug 01 08:45:19 localhost systemd[1]: Failed to start Remote desktop service (VNC). Aug 01 08:45:19 localhost systemd[1]: Unit vncserver@:2.service entered failed state. Aug 01 08:45:19 localhost systemd[1]: vncserver@:2.service failed. =========== And yet, I am able to connect to 5902! It is running. cat /root/.vnc/localhost:2.log shows: Xvnc TigerVNC 1.8.0 - built Apr 11 2018 21:04:25 Copyright (C) 1999-2017 TigerVNC Team and many others (see README.txt) See http://www.tigervnc.org for information on TigerVNC. Underlying X server release 11903000, The X.Org Foundation Wed Aug 1 08:45:16 2018 vncext: VNC extension running! vncext: Listening for VNC connections on all interface(s), port 5902 vncext: created VNC server for screen 0 Wed Aug 1 08:45:23 2018 ComparingUpdateTracker: 0 pixels in / 0 pixels out ComparingUpdateTracker: (1:nan ratio) Wed Aug 1 08:51:01 2018 Connections: accepted: 192.168.160.20::53318 SConnection: Client needs protocol version 3.8 SConnection: Client requests security type VeNCrypt(19) SVeNCrypt: Client requests security type TLSVnc (258) etc. ======== what is going on with systemctl? thanks

1 0

how to update python 2.7.5 to python 2.7.15 on centos 7.4
by qw 01 Aug '18

01 Aug '18

Hi, I have one question about python update. How to update python 2.7.5 to python 2.7.15 on centos 7.4? Thanks! Regards Andrew

2 1

CVE-2018-0495 Libgcrypt Vulnerability Fix for CentOS 6?
by nschehovin＠yahoo.com 01 Aug '18

01 Aug '18

Is there a schedule for fixing CVE-2018-0495 Libgcrypt Vulnerability for CentOS 6? I can not find a patch for this CVE. Please point me in the right direction if I am missing something. Thanks, Ed

2 1

Re: [CentOS] (EXT) CentOS Digest, Vol 162, Issue 29
by Selvam, Joshua 01 Aug '18

01 Aug '18

Send CentOS mailing list submissions to centos(a)centos.org -----Original Message----- From: CentOS <centos-bounces(a)centos.org> On Behalf Of centos-request(a)centos.org Sent: Tuesday, July 31, 2018 5:30 PM To: centos(a)centos.org Subject: (EXT) CentOS Digest, Vol 162, Issue 29 Send CentOS mailing list submissions to centos(a)centos.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.centos.org/mailman/listinfo/centos or, via email, send a message with subject or body 'help' to centos-request(a)centos.org You can reach the person managing the list at centos-owner(a)centos.org When replying, please edit your Subject line so it is more specific than "Re: Contents of CentOS digest..." Today's Topics: 1. Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Yannis Milios) 2. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Simon Matter) 3. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Yannis Milios) 4. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (mark) 5. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Yannis Milios) 6. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (mark) 7. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Yannis Milios) 8. Re: Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Simon Matter) 9. Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] (Yannis Milios) 10. Increase Disk Space with LVM (Felix K?lzow) ---------------------------------------------------------------------- Message: 1 Date: Mon, 30 Jul 2018 16:26:43 +0100 From: Yannis Milios <yannis.milios(a)gmail.com> To: centos(a)centos.org Subject: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <CAFiF2OpdpWuVLyZq7_4n1MxtMr2ypJkgKhLt+03ayapexgNuqA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hello, I'm having a strange problem booting a new centos7 installation. Below some background on this. [I have attached the tech details at the bottom of this message] I started a new CentOS7 installation on a VM, so far all good, o/s boots fine. Then I decided to increase VM disk size (initially was 10G) to 13G. Powered off the VM, increased the vhd via the hypervisor, booted from CentOS livecd, selected "recover my centos installation". Then I used the following sequence of commands to make the new vhd size "visible" to the o/s ... - ran fdisk /dev/sda and deleted partition 2. (This is the LVM partition where the o/s is stored. The first partition is /boot [xfs] partition *non lvm*.) - created a new partition with same starting sector as before deleting it, but a different ending sector (to reflect to the increased space). - set partition type to 8e (lvm), saved settings and exit. - ran pvresize and lvresize to make new space visible to the o/s.All good, I can see the space increase on 'lvdisplay centos/root'. - ran 'mount /dev/mapper/centos-root /mnt/root' to temporarily mount the o/s partition. - ran 'xfs_growfs /mnt/root' to resize the o/s fs to the new size. It was successful and I could actually chroot to the o/s and verify new disk size. - rebooted and tried to boot from hdd this time. Grub menu shows up and loads default kernel (3.10.0-862.9.1.el7.x86_64). - after initial kernel boot up process, booting continues to initrd and then it's where the problem starts... - looks like dracut has issues locating/enabling /dev/mapper/centos-root (lv) and as a result it cannot boot to the 'real' root fs (/). - while in dracut shell, I execute the following command sequence: 1. lvm lvchange -ay /dev/centos/root 2. lvm lvchange -ay /dev/centos/swap 3. ln -s /dev/mapper/centos-root /dev/root 4. exit ...and the o/s boots fine...so looks like the pv,vg,lv is detected properly while in initrd, but somehow dracut has difficulties enabling the root,swap LVs ? - While in o/s, I rebuild initrd by using: 'dracut -f -v -a lvm' Note, I have to use '-a lvm' as for some reason, if don't, the lvm utils (lvm,lvm_scan) are not being included to the initrd, not sure why this happens. After rebooting, same thing happens, I have to manually boot the system via dracut shell. I'm a bit stuck at this point, any clues what I'm doing wrong in here ? As I previously said, this is a new installation, so I could simply reinstall the whole thing, but I would rather try to find out some answer to why is this happening ... :-) [curiosity] Some additional details: ================== I've attached /run/initramfs/rdsosreport.txt file [generated while in inird] here: https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jMxiUxq… o/s ver=CentOS Linux release 7.5.1804 (Core) [root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 13G 0 disk ??sda1 8:1 0 500M 0 part /boot ??sda2 8:2 0 12.5G 0 part ??centos-root 253:0 0 11.5G 0 lvm / ??centos-swap 253:1 0 1G 0 lvm [SWAP] [root@localhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rd.shell rhgb" GRUB_DISABLE_RECOVERY="true" ------------------------------ Message: 2 Date: Mon, 30 Jul 2018 17:53:56 +0200 From: "Simon Matter" <simon.matter(a)invoca.ch> To: "CentOS mailing list" <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <09679ddf131cfabaaa9a38ee3a874541.squirrel(a)webmail.bi.invoca.ch> Content-Type: text/plain;charset=utf-8 Hi, > Hello, > > I'm having a strange problem booting a new centos7 installation. Below > some background on this. [I have attached the tech details at the > bottom of this message] > > I started a new CentOS7 installation on a VM, so far all good, o/s > boots fine. Then I decided to increase VM disk size (initially was 10G) to 13G. > Powered off the VM, increased the vhd via the hypervisor, booted from > CentOS livecd, selected "recover my centos installation". Then I used > the following sequence of commands to make the new vhd size "visible" > to the o/s ... > > - ran fdisk /dev/sda and deleted partition 2. (This is the LVM > partition where the o/s is stored. The first partition is /boot [xfs] > partition *non > lvm*.) > - created a new partition with same starting sector as before deleting > it, but a different ending sector (to reflect to the increased space). > - set partition type to 8e (lvm), saved settings and exit. At this point I usually reboot and do the following steps while the system is running. I don't say how you do it is wrong, it's just easier/faster to do things while the system is online if downtime matters. I don't really know what causes your issues. The reason may be that I have more and more problems understanding the whole boot process and all the variants we used to have over more than two decades... But, what happens if you let the kernel post install scripts do the work or setting up the initrd things? If you reinstall the latest kernel, does it change anything? Regards, Simon > - ran pvresize and lvresize to make new space visible to the o/s.All > good, I can see the space increase on 'lvdisplay centos/root'. > - ran 'mount /dev/mapper/centos-root /mnt/root' to temporarily mount > the o/s partition. > - ran 'xfs_growfs /mnt/root' to resize the o/s fs to the new size. It > was successful and I could actually chroot to the o/s and verify new > disk size. > - rebooted and tried to boot from hdd this time. Grub menu shows up > and loads default kernel (3.10.0-862.9.1.el7.x86_64). > - after initial kernel boot up process, booting continues to initrd > and then it's where the problem starts... > - looks like dracut has issues locating/enabling > /dev/mapper/centos-root > (lv) and as a result it cannot boot to the 'real' root fs (/). > - while in dracut shell, I execute the following command sequence: > 1. lvm lvchange -ay /dev/centos/root > 2. lvm lvchange -ay /dev/centos/swap > 3. ln -s /dev/mapper/centos-root /dev/root > 4. exit > ...and the o/s boots fine...so looks like the pv,vg,lv is detected > properly while in initrd, but somehow dracut has difficulties enabling > the root,swap LVs ? > > - While in o/s, I rebuild initrd by using: 'dracut -f -v -a lvm' > Note, I have to use '-a lvm' as for some reason, if don't, the lvm > utils > (lvm,lvm_scan) are not being included to the initrd, not sure why this > happens. > After rebooting, same thing happens, I have to manually boot the > system via dracut shell. > > I'm a bit stuck at this point, any clues what I'm doing wrong in here ? > > As I previously said, this is a new installation, so I could simply > reinstall the whole thing, but I would rather try to find out some > answer to why is this happening ... :-) [curiosity] > > Some additional details: > ================== > I've attached /run/initramfs/rdsosreport.txt file [generated while in > inird] here: > https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jM > xiUxq5DWhTVbWc= > > o/s ver=CentOS Linux release 7.5.1804 (Core) > > [root@localhost ~]# lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 13G 0 disk > ??sda1 8:1 0 500M 0 part /boot > ??sda2 8:2 0 12.5G 0 part > ??centos-root 253:0 0 11.5G 0 lvm / > ??centos-swap 253:1 0 1G 0 lvm [SWAP] > > [root@localhost ~]# cat /etc/default/grub > GRUB_TIMEOUT=5 > GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" > GRUB_DEFAULT=saved > GRUB_DISABLE_SUBMENU=true > GRUB_TERMINAL_OUTPUT="console" > GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap > rd.shell rhgb" > GRUB_DISABLE_RECOVERY="true" > _______________________________________________ > CentOS mailing list > CentOS(a)centos.org > https://lists.centos.org/mailman/listinfo/centos > ------------------------------ Message: 3 Date: Mon, 30 Jul 2018 17:32:53 +0100 From: Yannis Milios <yannis.milios(a)gmail.com> To: centos(a)centos.org Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <CAFiF2Or=a_2b+e+jj9AeUufnZg4N+rHvS8KJHF+LoBDJ+QcRNg(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" > > But, what happens if you let the kernel post install scripts do the > work or setting up the initrd things? Initially, I just rebooted from LiveCD and left Grub,kernel and dracut do their job with the defaults, but unfortunately, boot process stops at initrd (dracut) stage with the following error: [ 197.768159] localhost.localdomain dracut-initqueue[252]: Warning: Could > not boot. > [ 197.821409] localhost.localdomain systemd[1]: Received SIGRTMIN+20 > from PID 254 (plymouthd). > [ 197.822328] localhost.localdomain dracut-initqueue[252]: Warning: > /dev/centos/root does not exist > [ 197.823044] localhost.localdomain dracut-initqueue[252]: Warning: > /dev/centos/swap does not exist > [ 197.846807] localhost.localdomain systemd[1]: Starting Dracut > Emergency Shell... > [ 197.870983] localhost.localdomain systemd[1]: Received SIGRTMIN+21 > from PID 254 (plymouthd). Then, I'm dropped to dracut shell, where I boot the system by using the lvm commands as described to my first post. Once in O/S, I re-create initrd by using 'dracut -f -v -a lvm' command and reboot again the system. But still I get the same error during boot ... If you reinstall the latest kernel, does > it change anything? > This is actually the latest kernel (3.10.0-862.9.1.el7.x86_64) but I tried to boot with the second (older) kernel (vmlinuz-3.10.0-327.36.3.el7.x86_64), where I got exactly the same error during boot. Clearly, this is not a kernel issue, perhaps not even a dracut (initrd) issue, since everything was working fine, before resizing the PV/LV. I suspect that something went wrong during that step, but what? As I can successfully boot the system manually via dracut, normally everything should be ok? no? What can prevent dracut from activating the 2 LVs (centos/root and centos/swap) during initial boot phase ? Thanks, Yannis ------------------------------ Message: 4 Date: Mon, 30 Jul 2018 15:13:08 -0400 From: "mark" <m.roth(a)5-cent.us> To: "CentOS mailing list" <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <e9796451bea645de4bc4fbe210591182.squirrel(a)host290.hostmonster.com> Content-Type: text/plain;charset=utf-8 Yannis Milios wrote: >> But, what happens if you let the kernel post install scripts do the >> work or setting up the initrd things? > > Initially, I just rebooted from LiveCD and left Grub,kernel and dracut > do their job with the defaults, but unfortunately, boot process stops > at initrd (dracut) stage with the following error: > > [ 197.768159] localhost.localdomain dracut-initqueue[252]: Warning: > Could > >> not boot. [ 197.821409] localhost.localdomain systemd[1]: Received >> SIGRTMIN+20 from >> PID 254 (plymouthd). >> [ 197.822328] localhost.localdomain dracut-initqueue[252]: Warning: >> /dev/centos/root does not exist >> [ 197.823044] localhost.localdomain dracut-initqueue[252]: Warning: >> /dev/centos/swap does not exist >> [ 197.846807] localhost.localdomain systemd[1]: Starting Dracut >> Emergency Shell... >> [ 197.870983] localhost.localdomain systemd[1]: Received SIGRTMIN+21 >> from PID 254 (plymouthd). >> > > > Then, I'm dropped to dracut shell, where I boot the system by using > the lvm commands as described to my first post. Once in O/S, I > re-create initrd by using 'dracut -f -v -a lvm' command and reboot > again the system. But still I get the same error during boot ... > > If you reinstall the latest kernel, does > >> it change anything? >> > > This is actually the latest kernel (3.10.0-862.9.1.el7.x86_64) but I > tried to boot with the second (older) kernel > (vmlinuz-3.10.0-327.36.3.el7.x86_64), where I got exactly the same > error during boot. > > Clearly, this is not a kernel issue, perhaps not even a dracut > (initrd) issue, since everything was working fine, before resizing the > PV/LV. I suspect that something went wrong during that step, but what? > As I can successfully boot the system manually via dracut, normally > everything should be ok? no? > > What can prevent dracut from activating the 2 LVs (centos/root and > centos/swap) during initial boot phase ? > Suggestion: once it's up, rebuild the initramfs. mark ------------------------------ Message: 5 Date: Mon, 30 Jul 2018 22:14:44 +0100 From: Yannis Milios <yannis.milios(a)gmail.com> To: CentOS mailing list <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <CAFiF2OrJUiNh35M6PBxe=CQrtF1dRYb14b5H4pewTX-EsSm-TA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Suggestion: once it's up, rebuild the initramfs. > > I tried that already, but still the same problem. Aparently dracut does not want to activate the LVs required to boot to the root filesystem, for some reason ... Yannis -- Sent from Gmail Mobile ------------------------------ Message: 6 Date: Mon, 30 Jul 2018 17:16:51 -0400 From: "mark" <m.roth(a)5-cent.us> To: "CentOS mailing list" <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <ce2b2fbed114eda6c7159b25098d48d1.squirrel(a)host290.hostmonster.com> Content-Type: text/plain;charset=utf-8 Yannis Milios wrote: > Suggestion: once it's up, rebuild the initramfs. >> > I tried that already, but still the same problem. > Aparently dracut does not want to activate the LVs required to boot to the > root filesystem, for some reason ... > At this point, I'd start wondering about the grub2 defaults, and the kernel command line. mark ------------------------------ Message: 7 Date: Mon, 30 Jul 2018 23:06:15 +0100 From: Yannis Milios <yannis.milios(a)gmail.com> To: CentOS mailing list <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <CAFiF2Or5Ave-rhSCMmcD-h7X-sbhGsAGRG6psc0=jxdma7rN5g(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" On Monday, July 30, 2018, mark <m.roth(a)5-cent.us> > > At this point, I'd start wondering about the grub2 defaults, and the > kernel command line. > > This is Grub config.. [root@localhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rd.shell rhgb" GRUB_DISABLE_RECOVERY="true" -- Sent from Gmail Mobile ------------------------------ Message: 8 Date: Tue, 31 Jul 2018 08:22:12 +0200 From: "Simon Matter" <simon.matter(a)invoca.ch> To: "CentOS mailing list" <centos(a)centos.org> Subject: Re: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <970e648731344f68e1a1837a94b71d41.squirrel(a)webmail.bi.invoca.ch> Content-Type: text/plain;charset=utf-8 > Yannis Milios wrote: >>> But, what happens if you let the kernel post install scripts do the >>> work or setting up the initrd things? >> >> Initially, I just rebooted from LiveCD and left Grub,kernel and dracut >> do >> their job with the defaults, but unfortunately, boot process stops at >> initrd (dracut) stage with the following error: >> >> [ 197.768159] localhost.localdomain dracut-initqueue[252]: Warning: >> Could >> >>> not boot. [ 197.821409] localhost.localdomain systemd[1]: Received >>> SIGRTMIN+20 from >>> PID 254 (plymouthd). >>> [ 197.822328] localhost.localdomain dracut-initqueue[252]: Warning: >>> /dev/centos/root does not exist >>> [ 197.823044] localhost.localdomain dracut-initqueue[252]: Warning: >>> /dev/centos/swap does not exist >>> [ 197.846807] localhost.localdomain systemd[1]: Starting Dracut >>> Emergency >>> Shell... >>> [ 197.870983] localhost.localdomain systemd[1]: Received SIGRTMIN+21 >>> from PID 254 (plymouthd). >>> >> >> >> Then, I'm dropped to dracut shell, where I boot the system by using the >> lvm commands as described to my first post. Once in O/S, I re-create >> initrd >> by using 'dracut -f -v -a lvm' command and reboot again the system. But >> still I get the same error during boot ... >> >> If you reinstall the latest kernel, does >> >>> it change anything? >>> >> >> This is actually the latest kernel (3.10.0-862.9.1.el7.x86_64) but I >> tried to boot with the second (older) kernel >> (vmlinuz-3.10.0-327.36.3.el7.x86_64), where I got exactly the same error >> during boot. >> >> Clearly, this is not a kernel issue, perhaps not even a dracut (initrd) >> issue, since everything was working fine, before resizing the PV/LV. I >> suspect that something went wrong during that step, but what? As I can >> successfully boot the system manually via dracut, normally everything >> should be ok? no? >> >> What can prevent dracut from activating the 2 LVs (centos/root and >> centos/swap) during initial boot phase ? >> > Suggestion: once it's up, rebuild the initramfs. And when you did so, did you remove the "rd.shell" from GRUB_CMDLINE_LINUX before? Regards, Simon ------------------------------ Message: 9 Date: Tue, 31 Jul 2018 08:40:52 +0100 From: Yannis Milios <yannis.milios(a)gmail.com> To: CentOS mailing list <centos(a)centos.org> Subject: [CentOS] Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs] Message-ID: <CAFiF2Or3nxaJcY44mWjosiTbwb==w2hU95iMpMC4=sTY7Wvn2A(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" And when you did so, did you remove the "rd.shell" from GRUB_CMDLINE_LINUX > before? > > Yes, basically that option was not there by default, I added it during the troubleshootig steps. Normally, rd.lvm.lv=centos/root and rd.lvm.lv=centos/swap are the kernel parameters which are parsed to dracut for enablig the two LVs, but somehow dracut does not. I've copied the log from dracut in here, can this be helpful to locate the problem? https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jMxiUxq… -- Sent from Gmail Mobile ------------------------------ Message: 10 Date: Tue, 31 Jul 2018 13:31:15 +0200 From: Felix K?lzow <felix.koelzow(a)gmx.de> To: centos(a)centos.org Subject: [CentOS] Increase Disk Space with LVM Message-ID: <1dd31e6b-ee9e-4a00-1bf2-22c0c2f0e699(a)gmx.de> Content-Type: text/plain; charset=utf-8; format=flowed Dear CentOS-Community, we have a server with four hard drives that are configured as raid10 (/dev/sda). Now, /home and /root are almost full. Therefore, we decided to buy four additional hard drives that should configured also as raid 10 (/dev/sdb). I want to use LVM to extend disk space for root and home. My (successful) test procedure in a virtual environment looks like this: 1. devide /dev/sdb into /dev/sdb1 for root and /dev/sdb2 for home using parted 2. Convert disk to physical volume: pvcreate /dev/sdb1 3. add physical volume to volume group (called centos): vgextend centos /dev/sdb1 4. Allocate physical volume to a logical volume:lvextend -l +100$FREE /dev/centos/root 5. resize2fs /dev/centos/root or xfs_grows /dev/centos/root depending on file system used 6. repeat steps 2-6 for /home and sdb2 The mentioned instruction I've got from this page: https://askubuntu.com/questions/458476/adding-disks-with-lvm#459176 Now my question: Is this procedure really safe or is something missing or are there some steps which I overlooked actually? Kind Regards Felix K?lzow ------------------------------ Subject: Digest Footer _______________________________________________ CentOS mailing list CentOS(a)centos.org https://lists.centos.org/mailman/listinfo/centos ------------------------------ End of CentOS Digest, Vol 162, Issue 29 ***************************************

1 0

ecryptfs (from centosplus) and libvirt lxc quest on it - quest upon start shoots host in the head
by lejeczek 31 Jul '18

31 Jul '18

hi guys I have a lxc guest on Dell's R815(AMD Opterons) which I just moved to an ecryptfs mounted folder(using centosplus' kernel) Copying data to that folder seems to work just fine, but I as soon as I start the quest the host gets shot in the head, hard reboot, no dump. I also have another, virtually identical guest but on Intel platform and that one does not do anything but runs okey. Any of you guys has experience such gross abnormality? many thanks, L.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss