After reboot of web-server accessing website shows "Forbidden", restarting httpd all is fine
Hi.
I am stuck with this one and I do not know where and how to search for this problem nor do I know how to fix it.
When I reboot one of our servers (CentOS 6.7, selinux target, yum fully updated) the http server loads fine (no erros) but when accessing one of the server's websites it displays "Forbidden", restarting the httpd server (command line) will give full access and all is fine.
What troubles me that a simple restart of the daemon fixes everything but it does not come up on reboot.
[Sat Nov 07 13:02:44 2015] [notice] caught SIGTERM, shutting down [Sat Nov 07 13:02:45 2015] [notice] SELinux policy enabled; httpd running as context unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [Sat Nov 07 13:02:45 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sat Nov 07 13:02:45 2015] [notice] Digest: generating secret for digest authentication ... [Sat Nov 07 13:02:45 2015] [notice] Digest: done [Sat Nov 07 13:02:45 2015] [notice] Apache/2.2.15 (Unix) PHP/5.4.45 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
I started it with an strace but I cant find anything that looks suspicious ...
Putting selinux into permissive mode starts the server right from boot. Looking at all the logs I cannot see anything.
Any ideas?
Jobst
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
What troubles me that a simple restart of the daemon fixes everything but it does not come up on reboot.
Running the service script manually may not give you the same selinux context as on boot. Services should be started using "run_init" to ensure they get the correct context.
I think this is legitimately the most confusing aspect of SELinux, and it's one of the things that systemd fixed properly.
On Fri, Nov 06, 2015 at 07:23:59PM -0800, Gordon Messmer wrote:
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
What troubles me that a simple restart of the daemon fixes everything but it does not come up on reboot.
Running the service script manually may not give you the same selinux context as on boot. Services should be started using "run_init" to ensure they get the correct context.
How long has this been the case? I have never heard of this before, it seems a very well-kept secret!
I think this is legitimately the most confusing aspect of SELinux, and it's one of the things that systemd fixed properly.
On Sat, 7 Nov 2015, Fred Smith wrote:
On Fri, Nov 06, 2015 at 07:23:59PM -0800, Gordon Messmer wrote:
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
What troubles me that a simple restart of the daemon fixes everything but it does not come up on reboot.
Running the service script manually may not give you the same selinux context as on boot. Services should be started using "run_init" to ensure they get the correct context.
Yet it isn't really documented anywhere that you can or should use it, certainly not in the RHEL Administration or SELinux documentation, not even as a footnote. Only in a few bug reports and errata notices can you even find mention of the command.
How long has this been the case? I have never heard of this before, it seems a very well-kept secret!
Always, i.e., since SELinux appeared in RHEL and thus CentOS.
The service command has never done what is expected of it -- properly (re)start a service. This is noted here and there in the documentation. It usually does well enough for the other management tasks, reloading, stopping and providing status. But start and restart is almost totally wrong.
It allows (even forces) a "dirty" environment to be provided to the service (which is seldom wanted or expected), does not ensure that the current tty cannot be the controlling tty for the service (which sometimes matters) and leaves the CWD unchanged instead of ensuring / is used (which sometimes matters). No revision of the service command took place to cope with context when SELinux appeared and so the service inherits the current context, usually unconfined (which is wildly wrong). Sometimes doing it this way is useful, but not often and when it is one can invoke the service's init script directly.
So even before SELinux (or with it disabled) a mere "/sbin/service whichever start" does not always suffice. But building a boot environment is a bit tedious, you'd almost want some command to take care of that for you -- alas none exists. However the boot environment is usually very simple, which can be approximated with something like:
( cd / && setsid env -i /sbin/service whichever start )
With SELinux it is critical to run it in a more useful context:
( cd / && /usr/sbin/run_init setsid env -i /sbin/service whichever start )
If run_init isn't installed the following gives a similar result:
( cd / && runcon -u system_u setsid env -i /sbin/service whichever start )
Also for restart, condrestart, try-restart or --full-restart. And for some services reload or force-reload too, thus easiest to always use one of these for all service management needs. (To be fair, I don't usually use it for status or stop, but I was bitten once.) Alas the latter two fail if SELinux is disabled at which point you trim it back to the first.
Patching the service command locally is a hassle -- it is a shell script so not at all difficult to change, merely difficult to remember to re-patch after system updates lest you go back to being surprised (though a custom RPM with a trigger can do it) or go elsewhere where it hasn't ever been patched. Or produce your own script that does all that, so that only when it isn't available do you need to type a long line to (re)start a service. I leave the service command unpatched and didn't write my own script, instead I have the subshell things as muscle memory.
With systemd (CentOS 7) this changes. If you know systemd handles the service -- which is most of the time -- you can use the bare service command (or systemctl) to (re)start the service, in an environment that is the same as at boot, i.e., as expected. But not everything is controlled by systemd, and there you are back where this started.
/mark
On Mon, Nov 09, 2015 at 06:08:11AM -0800, Mark Milhollan wrote:
It allows (even forces) a "dirty" environment to be provided to the service (which is seldom wanted or expected), does not ensure that the current tty cannot be the controlling tty for the service (which sometimes matters) and leaves the CWD unchanged instead of ensuring / is used (which sometimes matters).
A quick viewing of /sbin/service on C6 makes me think you might be mistaken here. There's clearly a 'cd /' and an 'env -i' there. It does preserve $PATH though (also $TERM), which I view as a dirty environment.
No revision of the service command took place to cope with context when SELinux appeared and so the service inherits the current context, usually unconfined (which is wildly wrong). Sometimes doing it this way is useful, but not often and when it is one can invoke the service's init script directly.
I'm pretty sure that what happens is that service runs the service scripts in /etc/rc.d/init.d/, which all have labels on them that indicate what entrypoint type they run under, which is by default 'initrc_exec_t' but I see several have their own special label, such as sshd having sshd_initrc_exec_t. If 'service' were just sourcing the init.d files instead of executing them, it would be different, but it does execute them, and since the init scripts have an entrypoint type to transition to the appropriate initrc domain.
Try to set your service to enable to running automatically on boot. I hope it work's On Nov 9, 2015 9:33 PM, "Jonathan Billings" billings@negate.org wrote:
On Mon, Nov 09, 2015 at 06:08:11AM -0800, Mark Milhollan wrote:
It allows (even forces) a "dirty" environment to be provided to the service (which is seldom wanted or expected), does not ensure that the current tty cannot be the controlling tty for the service (which sometimes matters) and leaves the CWD unchanged instead of ensuring / is used (which sometimes matters).
A quick viewing of /sbin/service on C6 makes me think you might be mistaken here. There's clearly a 'cd /' and an 'env -i' there. It does preserve $PATH though (also $TERM), which I view as a dirty environment.
No revision of the service command took place to cope with context when SELinux appeared and so the service inherits the current context, usually unconfined (which is wildly wrong). Sometimes doing it this way is useful, but not often and when it is one can invoke the service's init script directly.
I'm pretty sure that what happens is that service runs the service scripts in /etc/rc.d/init.d/, which all have labels on them that indicate what entrypoint type they run under, which is by default 'initrc_exec_t' but I see several have their own special label, such as sshd having sshd_initrc_exec_t. If 'service' were just sourcing the init.d files instead of executing them, it would be different, but it does execute them, and since the init scripts have an entrypoint type to transition to the appropriate initrc domain.
-- Jonathan Billings billings@negate.org _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
sorry late reply. thanks, will look into it.
On Fri, Nov 06, 2015 at 07:23:59PM -0800, Gordon Messmer (gordon.messmer@gmail.com) wrote:
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
What troubles me that a simple restart of the daemon fixes everything but it does not come up on reboot.
Running the service script manually may not give you the same selinux context as on boot. Services should be started using "run_init" to ensure they get the correct context.
I think this is legitimately the most confusing aspect of SELinux, and it's one of the things that systemd fixed properly. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
Putting selinux into permissive mode starts the server right from boot. Looking at all the logs I cannot see anything.
Which logs? You should see AVC denies logged in /var/log/audit/audit.log, unless you've disabled audit logging.
The AVCs should indicate which files are labeled incorrectly, and what their current label is. You probably need to fix the tree from which you're serving files.
Could be just "restorecon -r -v /var/www" if you're using the default paths. Otherwise, you should use semanage to fix whatever paths you're using:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/htm...
sorry, late reply. thanks, one part was a path.
jobst
On Mon, Nov 09, 2015 at 09:40:44AM -0800, Gordon Messmer (gordon.messmer@gmail.com) wrote:
On 11/06/2015 06:30 PM, Jobst Schmalenbach wrote:
Putting selinux into permissive mode starts the server right from boot. Looking at all the logs I cannot see anything.
Which logs? You should see AVC denies logged in /var/log/audit/audit.log, unless you've disabled audit logging.
The AVCs should indicate which files are labeled incorrectly, and what their current label is. You probably need to fix the tree from which you're serving files.
Could be just "restorecon -r -v /var/www" if you're using the default paths. Otherwise, you should use semanage to fix whatever paths you're using:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/htm... _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Adhi Tj -
Fred Smith -
Gordon Messmer -
Jobst Schmalenbach -
Jonathan Billings -
Mark Milhollan