We started updating via CR over a week ago, before 6.7 was official, and just today identified an issue.... For (alleged) security, the agency I work as a contractor for runs SiteMinder, from CA.
#insert rant_against_CA.h
Anyway, starting late last week, we found issues - as in, its process, which runs under, and is started by, apache, was suddenly pegging a CPU or so. Trying to stop httpd, that worked... but this idiot process never did (and it's ugly to clean up after).
What we just this morning found out to be the problem is that some package seems to change the permissions on /var/log/httpd to 700 from 770. The result was that this ...thing... couldn't write to its own logs, running as apache:root, while /var/log/httpd was root:root.
I just did rpm -q httpd --scripts, and that doesn't show anything, so as I don't know what package did it.... If anyone knows, I'd like to know.
mark
------------ Original Message ------------
Date: Tuesday, August 11, 2015 11:43:21 -0400 From: m.roth@5-cent.us
We started updating via CR over a week ago, before 6.7 was official, and just today identified an issue.... For (alleged) security, the agency I work as a contractor for runs SiteMinder, from CA.
# insert rant_against_CA.h
Anyway, starting late last week, we found issues - as in, its process, which runs under, and is started by, apache, was suddenly pegging a CPU or so. Trying to stop httpd, that worked... but this idiot process never did (and it's ugly to clean up after).
What we just this morning found out to be the problem is that some package seems to change the permissions on /var/log/httpd to 700 from 770. The result was that this ...thing... couldn't write to its own logs, running as apache:root, while /var/log/httpd was root:root.
I just did rpm -q httpd --scripts, and that doesn't show anything, so as I don't know what package did it.... If anyone knows, I'd like to know.
mark
I didn't try poking at the rpm too much, but just checked and found that the httpd-2.2.15-45 rpm, that's part of the (regular) 6.7 update, will change the permissions on the /var/log/httpd directory (but not the files in it) to 700 and the ownership (again, of the directory, not the included files) to root.root from whatever you may have set them to. Those are the same ownerships/permissions that are the default in 6.6.
I.e., it appears that someone/thing modified the /var/log/httpd directory permissions and ownerships from the default and the updated httpd put them back.
Isn't there a bit of a security issue in your (modified) setup with those files being able to be written to by the apache user?
Richard wrote:
From: m.roth@5-cent.us
<snip>
Anyway, starting late last week, we found issues - as in, its process, which runs under, and is started by, apache, was suddenly pegging a CPU or so. Trying to stop httpd, that worked... but this idiot process never did (and it's ugly to clean up after).
What we just this morning found out to be the problem is that some package seems to change the permissions on /var/log/httpd to 700 from 770. The result was that this ...thing... couldn't write to its own logs, running as apache:root, while /var/log/httpd was root:root.
I just did rpm -q httpd --scripts, and that doesn't show anything, so as I don't know what package did it.... If anyone knows, I'd like to know.
I didn't try poking at the rpm too much, but just checked and found that the httpd-2.2.15-45 rpm, that's part of the (regular) 6.7 update, will change the permissions on the /var/log/httpd directory (but not the files in it) to 700 and the ownership (again, of the directory, not the included files) to root.root from whatever you may have set them to. Those are the same ownerships/permissions that are the default in 6.6.
Really! Ok, how did you see that? When I ran rpm -q httpd --scripts, I got preinstall scriptlet (using /bin/sh): # Add the "apache" user getent group apache >/dev/null || groupadd -g 48 -r apache getent passwd apache >/dev/null || \ useradd -r -u 48 -g apache -s /sbin/nologin \ -d /var/www -c "Apache" apache exit 0 postinstall scriptlet (using /bin/sh): # Register the httpd service /sbin/chkconfig --add httpd /sbin/chkconfig --add htcacheclean preuninstall scriptlet (using /bin/sh): if [ $1 = 0 ]; then /sbin/service httpd stop > /dev/null 2>&1 /sbin/chkconfig --del httpd /sbin/service htcacheclean stop > /dev/null 2>&1 /sbin/chkconfig --del htcacheclean fi posttrans scriptlet (using /bin/sh): test -f /etc/sysconfig/httpd-disable-posttrans || \ /sbin/service httpd condrestart >/dev/null 2>&1 || :
And there's no reference to /var/log/httpd.
I.e., it appears that someone/thing modified the /var/log/httpd directory permissions and ownerships from the default and the updated httpd put them back.
Isn't there a bit of a security issue in your (modified) setup with those files being able to be written to by the apache user?
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Or does it expect that httpd run as apache:apache, and then /var/log/httpd should be apache:apache?
Certainly, httpd shouldn't be running as root....
mark
Date: Tuesday, August 11, 2015 12:59:58 -0400 From: m.roth@5-cent.us
Richard wrote:
From: m.roth@5-cent.us
<snip> >> Anyway, starting late last week, we found issues - as in, its >> process, which runs under, and is started by, apache, was >> suddenly pegging a CPU or so. Trying to stop httpd, that >> worked... but this idiot process never did (and it's ugly to >> clean up after). >> >> What we just this morning found out to be the problem is that >> some package seems to change the permissions on /var/log/httpd >> to 700 from 770. The result was that this ...thing... couldn't >> write to its own logs, running as apache:root, while >> /var/log/httpd was root:root. >> >> I just did rpm -q httpd --scripts, and that doesn't show >> anything, so as I don't know what package did it.... If anyone >> knows, I'd like to know. > > I didn't try poking at the rpm too much, but just checked and > found that the httpd-2.2.15-45 rpm, that's part of the (regular) > 6.7 update, will change the permissions on the /var/log/httpd > directory (but not the files in it) to 700 and the ownership > (again, of the directory, not the included files) to root.root > from whatever you may have set them to. Those are the same > ownerships/permissions that are the default in 6.6.
Really! Ok, how did you see that? When I ran rpm -q httpd --scripts, I got preinstall scriptlet (using /bin/sh): # Add the "apache" user getent group apache >/dev/null || groupadd -g 48 -r apache getent passwd apache >/dev/null || \ useradd -r -u 48 -g apache -s /sbin/nologin \ -d /var/www -c "Apache" apache exit 0 postinstall scriptlet (using /bin/sh): # Register the httpd service /sbin/chkconfig --add httpd /sbin/chkconfig --add htcacheclean preuninstall scriptlet (using /bin/sh): if [ $1 = 0 ]; then /sbin/service httpd stop > /dev/null 2>&1 /sbin/chkconfig --del httpd /sbin/service htcacheclean stop > /dev/null 2>&1 /sbin/chkconfig --del htcacheclean fi posttrans scriptlet (using /bin/sh): test -f /etc/sysconfig/httpd-disable-posttrans || \ /sbin/service httpd condrestart >/dev/null 2>&1 || :
And there's no reference to /var/log/httpd.
I.e., it appears that someone/thing modified the /var/log/httpd directory permissions and ownerships from the default and the updated httpd put them back.
Isn't there a bit of a security issue in your (modified) setup with those files being able to be written to by the apache user?
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Or does it expect that httpd run as apache:apache, and then /var/log/httpd should be apache:apache?
Certainly, httpd shouldn't be running as root....
mark
I simply mucked with the permissions and ownerships of the /var/log/httpd directory/files on a 6.7 system and then did "yum reinstall" of the httpd-2.2.15-45 rpm that's part of 6.7 and noted what happened. I happen to also have a 6.6 system and compared things between the two, so can say that the 6.7 and 6.6 /var/log/httpd directory defaults are the same -- 700 / root.root.
The default is that httpd starts as root and then spawns its worker tasks as the user set in the "User" directive in the httpd.conf.
I believe that the security view is that if there is a hole in httpd (directly, or more likely a (php/perl/...) script that someone is using) you don't want apache (or whatever user the httpd worker tasks are running as) to have write access to anything. That's why it is generally considered bad form to have documentroot directories/files owned by "apache" (or writeable by the apache group) in the default setup.
Given that, I found it slightly amusing that your "security tool", seemingly running as "apache", had write access to (and ownership control of?) the httpd log directory and files.
Richard wrote:
From: m.roth@5-cent.us Richard wrote:
From: m.roth@5-cent.us
<snip> >> Anyway, starting late last week, we found issues - as in, its >> process, which runs under, and is started by, apache, was >> suddenly pegging a CPU or so. Trying to stop httpd, that >> worked... but this idiot process never did (and it's ugly to >> clean up after). >> >> What we just this morning found out to be the problem is that >> some package seems to change the permissions on /var/log/httpd >> to 700 from 770. The result was that this ...thing... couldn't >> write to its own logs, running as apache:root, while >> /var/log/httpd was root:root. >> >> I just did rpm -q httpd --scripts, and that doesn't show >> anything, so as I don't know what package did it.... If anyone >> knows, I'd like to know. > > I didn't try poking at the rpm too much, but just checked and > found that the httpd-2.2.15-45 rpm, that's part of the (regular) > 6.7 update, will change the permissions on the /var/log/httpd > directory (but not the files in it) to 700 and the ownership > (again, of the directory, not the included files) to root.root > from whatever you may have set them to. Those are the same > ownerships/permissions that are the default in 6.6.
<snip>
And there's no reference to /var/log/httpd.
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Or does it expect that httpd run as apache:apache, and then /var/log/httpd should be apache:apache?
Certainly, httpd shouldn't be running as root....
I simply mucked with the permissions and ownerships of the /var/log/httpd directory/files on a 6.7 system and then did "yum reinstall" of the httpd-2.2.15-45 rpm that's part of 6.7 and noted what happened. I happen to also have a 6.6 system and compared things between the two, so can say that the 6.7 and 6.6 /var/log/httpd directory defaults are the same -- 700 / root.root.
Right. I can't do that. I don't have a system to set it up on that way.
The default is that httpd starts as root and then spawns its worker tasks as the user set in the "User" directive in the httpd.conf.
Ahhh! I did know that, but had forgotten it. <snip>
Given that, I found it slightly amusing that your "security tool", seemingly running as "apache", had write access to (and ownership control of?) the httpd log directory and files.
It ain't mine. It's a required thing (and note that the division that mandates this stuff is very heavily <blinder>WINDOWS!!!<blinder>
SiteMinder is put out by Computer Associates, a huge company that was putting out very expensive and popular mainframe software decades ago, and hasn't gotten any smaller.... And I do not believe they've ever wrapped their heads around Unix, much less Linux. Actually, I was talking to someone from our internal SiteMinder support, and asked them to *please* put an enhancement request into CA to add an selinux policy, so I don't have to fight it with every release... and he says they've got two requests in now....
mark
On Tue, 2015-08-11 at 12:59 -0400, m.roth@5-cent.us wrote:
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
On my C5.11 and C6.7 systems I get:-
: .l /var/log/|.g htt 24:drwx------ 2 root root 4096 Sep 16 2014 httpd
On Tue, Aug 11, 2015 at 12:59:58PM -0400, m.roth@5-cent.us wrote:
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Just a data point:
$ rpm -qp --qf='[%-11{filemodes:perms} %-8{fileusername} %-8{filegroupname} %{filenames}\n]' httpd-2.2.15-45.el6.x86_64.rpm | grep /var/log/httpd
drwx------ root root /var/log/httpd
Am 12.08.2015 um 14:27 schrieb Jonathan Billings billings@negate.org:
On Tue, Aug 11, 2015 at 12:59:58PM -0400, m.roth@5-cent.us wrote:
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Just a data point:
$ rpm -qp --qf='[%-11{filemodes:perms} %-8{fileusername} %-8{filegroupname} %{filenames}\n]' httpd-2.2.15-45.el6.x86_64.rpm | grep /var/log/httpd
drwx------ root root /var/log/httpd
httpd do not need access as apache user. the logging is done as "root" and for the inet interfaces privileges are dropped to apache:
# LANG=C ls -la /var/log/httpd/ total 1380 drwx------ 2 root root 4096 Aug 9 04:02 . drwxr-xr-x 6 root root 4096 Aug 12 04:02 .. -rw-r--r-- 1 root root 149617 Aug 12 14:36 access_log -rw-r--r-- 1 root root 294039 Aug 9 04:00 access_log.1 -rw-r--r-- 1 root root 298025 Aug 2 04:00 access_log.2 -rw-r--r-- 1 root root 284032 Jul 26 03:59 access_log.3 -rw-r--r-- 1 root root 289967 Jul 19 03:59 access_log.4 -rw-r--r-- 1 root root 302 Aug 9 04:02 error_log -rw-r--r-- 1 root root 11430 Aug 9 04:02 error_log.1 -rw-r--r-- 1 root root 1898 Aug 2 04:02 error_log.2 -rw-r--r-- 1 root root 31327 Jul 26 04:02 error_log.3 -rw-r--r-- 1 root root 1896 Jul 19 04:02 error_log.4
-- LF
Jonathan Billings wrote:
On Tue, Aug 11, 2015 at 12:59:58PM -0400, m.roth@5-cent.us wrote:
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Just a data point:
$ rpm -qp --qf='[%-11{filemodes:perms} %-8{fileusername} %-8{filegroupname} %{filenames}\n]' httpd-2.2.15-45.el6.x86_64.rpm | grep /var/log/httpd
drwx------ root root /var/log/httpd
Yeah, well, SiteMinder runs as a child of the httpd started by service start, so it runs as apache.
Ask me how much I think of SiteMinder... offlist, if you want the rant....
mark !@#$@!#$!@#~!@
Date: Wednesday, August 12, 2015 09:28:59 -0400 From: m.roth@5-cent.us
Jonathan Billings wrote:
On Tue, Aug 11, 2015 at 12:59:58PM -0400, m.roth@5-cent.us wrote:
So, since I haven't yet found where /var/log/httpd is created, what would a default package make the ownership of the directory? Does it expect it to be apache:root?
Just a data point:
$ rpm -qp --qf='[%-11{filemodes:perms} %-8{fileusername} %-8{filegroupname} %{filenames}\n]' httpd-2.2.15-45.el6.x86_64.rpm | grep /var/log/httpd
drwx------ root root /var/log/httpd
Yeah, well, SiteMinder runs as a child of the httpd started by service start, so it runs as apache.
Ask me how much I think of SiteMinder... offlist, if you want the rant....
mark !@#$@!#$!@#~!@
That's "fine" (within context), but then it shouldn't be able to write to files in the /var/log/httpd directory. [from something you posted I got the sense that it owned that directory, which is even worse (especially for a "security tool"), if that was correct.]
Assuming any ability to configure things, change it's logging to an application-specific directory.
The long-and-short is that at some point someone/thing changed the permissions (and maybe ownerships) on /var/log/httpd from the defaults. [something that i would have assumed would have gone into your change-management system.]
On 08/12/2015 09:21 AM, Richard wrote:
The long-and-short is that at some point someone/thing changed the permissions (and maybe ownerships) on /var/log/httpd from the defaults. [something that i would have assumed would have gone into your change-management system.]
I'm willing to bet a fairly substantial sum that it was the SiteMinder installation script.
-
Always Learning -
Ian Pilcher -
Jonathan Billings -
Leon Fauster -
m.roth＠5-cent.us -
Richard