I'm under a bit of a crunch here, so any immediate help would be appreciated. We kickstart our customer boxes and have started migrating to CentOS 7. We're running Radiator 4.6 1 (I know, but bear with me) and we just deployed our first radius box to a customer to be turned up today. (I know, I know. I had no idea it wasn't being tested sooner than this.)
I was brought in because the strict.pm perl module was missing and causing compilation errors. It turns out nearly the entire /usr/share/perl5/ directory was pretty much empty. I ended up having to copy that directory over from another C7 server which was intact.
yum whatprovides /usr/share/perl5/strict.pm tells me it's the base perl-5.16.x package, which is installed on this box.
Any idea what happened?
On Wed, 2017-11-29 at 12:28 -0500, Mark Haney wrote:
I'm under a bit of a crunch here, so any immediate help would be appreciated. We kickstart our customer boxes and have started migrating to CentOS 7. We're running Radiator 4.6 1 (I know, but bear with me) and we just deployed our first radius box to a customer to be turned up today. (I know, I know. I had no idea it wasn't being tested sooner than this.)
I was brought in because the strict.pm perl module was missing and causing compilation errors. It turns out nearly the entire /usr/share/perl5/ directory was pretty much empty. I ended up having to copy that directory over from another C7 server which was intact.
yum whatprovides /usr/share/perl5/strict.pm tells me it's the base perl-5.16.x package, which is installed on this box.
Any idea what happened?
Obviously something has removed everything - perhaps some over jealous script. Could you not just do 'yum reinstall perl'.
P.
Mark Haney wrote:
I'm under a bit of a crunch here, so any immediate help would be appreciated. We kickstart our customer boxes and have started migrating to CentOS 7. We're running Radiator 4.6 1 (I know, but bear with me) and we just deployed our first radius box to a customer to be turned up today. (I know, I know. I had no idea it wasn't being tested sooner than this.)
I was brought in because the strict.pm perl module was missing and causing compilation errors. It turns out nearly the entire /usr/share/perl5/ directory was pretty much empty. I ended up having to copy that directory over from another C7 server which was intact.
yum whatprovides /usr/share/perl5/strict.pm tells me it's the base perl-5.16.x package, which is installed on this box.
Any idea what happened?
No idea what could have happened, but if it were me, I wouldn't copy anything - I'd yum reinstall instantly. You have no idea what *else* is missing.
Thinking about it... you might consider verifying the entire system. Since something's missing from initscripts, I'd worry a *lot*.
mark
On 11/29/2017 01:43 PM, m.roth@5-cent.us wrote:
Mark Haney wrote:
Any idea what happened?
No idea what could have happened, but if it were me, I wouldn't copy anything - I'd yum reinstall instantly. You have no idea what *else* is missing.
Thinking about it... you might consider verifying the entire system. Since something's missing from initscripts, I'd worry a *lot*.
mark
Believe me, I am. Unfortunately and unbeknownst to me, this box has been in production on the customer side for a couple of weeks now. I've checked every other box that's been kickstarted for the last month and none show the same problems. It's really bizarre.
And as far as the /etc/init.d/functions file goes, C7 doesn't place it there, it's in /etc/rc.d/init.d/functions, so symlinking to it from /etc/init.d/ fixed that particular problem.
The weird issue with /usr/share/perl5/ is that there was some files and directories there, just not everything, so it wasn't completely empty. I have no real answer to that, though.
But, right now, the box is stable for what it will be doing, and I've got a production MySQL server to troubleshoot why it's imploded twice the last two nights after being up for 400 days without trouble.
The joys of dealing with multiple dumpster fires at a time is why I love (and hate) IT.
On 30/11/17 08:40, Mark Haney wrote:
On 11/29/2017 01:43 PM, m.roth@5-cent.us wrote:
Mark Haney wrote:
Any idea what happened?
I'd say at best (and likely) a botched install, at worst something more malicious.
No idea what could have happened, but if it were me, I wouldn't copy anything - I'd yum reinstall instantly. You have no idea what *else* is missing.
Thinking about it... you might consider verifying the entire system. Since something's missing from initscripts, I'd worry a *lot*.
I'd run "rpm -Va" and examine the output. Anything it shows in the output is something that differs from what should be installed and what actually is. See the rpm man page for details.
Run yum reinstall for any package that appears to have issues.
Believe me, I am. Unfortunately and unbeknownst to me, this box has been in production on the customer side for a couple of weeks now.
It should still be fine to run yum reinstall on a package, unless the customer has actually done something to rely on the brokenness of the system.
And as far as the /etc/init.d/functions file goes, C7 doesn't place it there, it's in /etc/rc.d/init.d/functions, so symlinking to it from /etc/init.d/ fixed that particular problem.
/etc/init.d is supposed to be a symbolic link to /etc/rc.d/init.d, if they're not then you need to fix that, and before you fix that you'll need to move any scripts you put inside it to /etc/rc.d.init.d. The damage you're seeing here is because they have somehow become separate directories on the server.
But, right now, the box is stable for what it will be doing,
I would not call that stable, I can just about guarantee that it will have problems down the road if you keep running it in this state.
Peter
Mark Haney wrote:
On 11/29/2017 01:43 PM, m.roth@5-cent.us wrote:
Mark Haney wrote:
Any idea what happened?
No idea what could have happened, but if it were me, I wouldn't copy anything - I'd yum reinstall instantly. You have no idea what *else* is missing.
Thinking about it... you might consider verifying the entire system. Since something's missing from initscripts, I'd worry a *lot*.
Believe me, I am. Unfortunately and unbeknownst to me, this box has been in production on the customer side for a couple of weeks now. I've checked every other box that's been kickstarted for the last month and none show the same problems. It's really bizarre.
Some admin ran find / -exec rm -f {} ;, when they meant to run find /somepathorother.
And as far as the /etc/init.d/functions file goes, C7 doesn't place it there, it's in /etc/rc.d/init.d/functions, so symlinking to it from /etc/init.d/ fixed that particular problem.
The weird issue with /usr/share/perl5/ is that there was some files and directories there, just not everything, so it wasn't completely empty. I have no real answer to that, though.
But, right now, the box is stable for what it will be doing, and I've got a production MySQL server to troubleshoot why it's imploded twice the last two nights after being up for 400 days without trouble.
The joys of dealing with multiple dumpster fires at a time is why I love (and hate) IT.
Best of luck.
mark
-
m.roth＠5-cent.us -
Mark Haney -
Pete Biggs -
Peter