In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var
To do so, I mounted the new drive as /var2, cp -R (in hindsight should had rsync to preserve attributes), deleted the original /var to free up space, edited fstab and rebooted... unsurprisingly to a fubar'd server.
The thing is it still boots, I can get into single user mode but a full init get stuck at starting the syslogger. What is the best way to rescue the server now from my own stupidity and keeping all the existing configuration and data?
2012/8/3 Emmanuel Noobadmin centos.admin@gmail.com:
In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var
To do so, I mounted the new drive as /var2, cp -R (in hindsight should had rsync to preserve attributes), deleted the original /var to free up space, edited fstab and rebooted... unsurprisingly to a fubar'd server.
The thing is it still boots, I can get into single user mode but a full init get stuck at starting the syslogger. What is the best way to rescue the server now from my own stupidity and keeping all the existing configuration and data? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Did you rewrite the selinux policy on /var or have you tried disabling selinux if you haven't do so already?
On 8/3/12, Darod Zyree darodzyree@gmail.com wrote:
Did you rewrite the selinux policy on /var or have you tried disabling selinux if you haven't do so already?
Thank you so much!
Turning off selinux allowed me get the system running. However, after running fixfiles to restore the context for /var, I still cannot boot to init 5, with the choke point now NFS statd.
I'll probably have to slowly hunt down the relevant selinux context one by one when nobody's screaming about the server being down.
Hello Emmanuel,
On Fri, 2012-08-03 at 18:52 +0800, Emmanuel Noobadmin wrote:
Turning off selinux allowed me get the system running. However, after running fixfiles to restore the context for /var, I still cannot boot to init 5, with the choke point now NFS statd.
If you copied a live /var over to the new partition you might have issues from stale lock files that prevent services from starting. /var/lock/subsys/nfsd comes to mind in this particular case. If the service ain't running that file shouldn't be there.
I don't think stale pid files are as troublesome but it doesn't hurt to have a look at /var/run as well.
Regards, Leonard.
On 08/03/2012 11:52 AM, Emmanuel Noobadmin wrote:
I'll probably have to slowly hunt down the relevant selinux context one by one when nobody's screaming about the server being down.
Would restorecon not help get this bootrapped ? and then with selinux in permissive mode, watch the audit log like a hawk.
- KB
On 8/3/12, Karanbir Singh mail-lists@karan.org wrote:
On 08/03/2012 11:52 AM, Emmanuel Noobadmin wrote:
I'll probably have to slowly hunt down the relevant selinux context one by one when nobody's screaming about the server being down.
Would restorecon not help get this bootrapped ? and then with selinux in permissive mode, watch the audit log like a hawk.
fixfiles/restorecon managed to get init 5 past syslogger but it got stuck still at NFS statd which locks up the entire server.
But with setenforce to permissive, the system appears to work fine and yes I would be doing that watch the audit log thing during the next scheduled down time.
On Friday, August 03, 2012 06:24:46 AM Emmanuel Noobadmin wrote:
In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var
...
This sort of things pops up from time to time.... from a thread back in April.....
On Wednesday, April 11, 2012 05:38:13 PM Jason Pyeron wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Alexander Dalloz Sent: Wednesday, April 11, 2012 17:12 To: CentOS mailing list Subject: Re: [CentOS] How to fix a chown oops...
Am 11.04.2012 23:02, schrieb Jason Pyeron:
chown -R 7.0 /sbin/ chown -R 98.98 //
Is there a rpm way fix all the permissions of files managed by rpms?
http://wiki.centos.org/TipsAndTricks/YumAndRPM#head-20a3ecce3d 0762b9cdd3307ef2632e0c274a2bfd
rpm -qa | while read line; do echo $line && rpm --setperms $line; done
...
By extension:
rpm -qa | while read line; do echo $line && rpm --setugids $line; done
should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
Hi,
On 08/03/2012 04:25 PM, Lamar Owen wrote:
rpm -qa | while read line; do echo $line && rpm --setugids $line; done should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
maybe add --setperms as well
On Friday, August 03, 2012 12:03:01 PM Karanbir Singh wrote:
Hi,
On 08/03/2012 04:25 PM, Lamar Owen wrote:
rpm -qa | while read line; do echo $line && rpm --setugids $line; done should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
maybe add --setperms as well
Hmm, I thought by including that in my quoted text that it was implied that one should do both.... My bad. It was a long day....
I'm not sure if both can be used on a single command line, since both --setperms and --setugids are implemented via popt aliases; I reserve the right to be wrong, of course.
I also forgot to specify in my reply that these commands would only repair files owned by packages; any non-package-owned files in /var won't be helped by either --setperms or --setugids; but it will give the OP a good start.
On 8/3/12, Lamar Owen lowen@pari.edu wrote:
On Friday, August 03, 2012 06:24:46 AM Emmanuel Noobadmin wrote:
In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var
...
This sort of things pops up from time to time.... from a thread back in April.....
Lesson learnt, never try to fix things I'm not familiar with when feeling pressured by relentless error messages, especially if nobody else is complaining yet.
rpm -qa | while read line; do echo $line && rpm --setperms $line; done
By extension:
rpm -qa | while read line; do echo $line && rpm --setugids $line; done
should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
Thanks for this tip, I'll try it and then see if there is anything else in audit log that needs attention.
On 08/03/12 3:24 AM, Emmanuel Noobadmin wrote:
In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var
To do so, I mounted the new drive as /var2, cp -R (in hindsight should had rsync to preserve attributes), deleted the original /var to free up space, edited fstab and rebooted... unsurprisingly to a fubar'd server.
The thing is it still boots, I can get into single user mode but a full init get stuck at starting the syslogger. What is the best way to rescue the server now from my own stupidity and keeping all the existing configuration and data?
if you had any database servers like postgresql or mysql, and their data files were in the default locations under /var, your databases are undoubtably corrupted, unless you stopped the DB server(s) before doing this copy.
On 8/3/12, John R Pierce pierce@hogranch.com wrote:
if you had any database servers like postgresql or mysql, and their data files were in the default locations under /var, your databases are undoubtably corrupted, unless you stopped the DB server(s) before doing this copy.
I think the fortunate thing is that everything else important on the server was running in their own VMs with their own LVM partitions. So luckily there doesn't seem to be anything important affected by my stupidity, the most important I wanted saved were the LVM configuration and VM configs which fortunately were in standalone XML files.
I *think* I probably could had quickly reinstall a bare minimum C6 to fix it, but after realizing my epic foolishness with replacing /var, I didn't want to take any chances of anaconda wiping all the LVM partitions.
-
Darod Zyree -
Emmanuel Noobadmin -
John R Pierce -
Karanbir Singh -
Lamar Owen -
Leonard den Ottolander