-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 17/02/16 14:08, Michael H wrote:

On 17/02/16 13:01, Johnny Hughes wrote:

...

I normally just let the daily announce post to this list show what is available for updates, but there is a CVE (CVE-2015-7547) that needs a bit more attention which will be on today's announce list of updates.

We released a new glibc yesterday for CentOS-6 and CentOS-7 .. it is VERY important that all users update to these versions: This update is rated as Critical by Red Hat, meaning that it is remotely exploitable under some circumstances. Make sure this update works in your environments and update as soon as you can.

CentOS-7: https://lists.centos.org/pipermail/centos-announce/2016-February/021672.html

https://rhn.redhat.com/errata/RHSA-2016-0176.html

...

CentOS-6: https://lists.centos.org/pipermail/centos-announce/2016-February/021668.html

https://rhn.redhat.com/errata/RHSA-2016-0175.html

...

These mitigate CVE-2015-7547: https://access.redhat.com/security/cve/CVE-2015-7547

https://bugzilla.redhat.com/show_bug.cgi?id=1293532

Can't stress how important this update is .. here are a couple stories:

http://arstechnica.com/security/2016/02/extremely-severe-bug-leaves-dizzying...

http://www.theregister.co.uk/2016/02/16/glibc_linux_dns_vulernability/

Please note that the ONLY way this is tested to work is with ALL

...

updates from CentOS-6 or CentOS-7 applied along with the glibc updates. So a yum update with base and updates repo enabled is the ONLY tested scenario. Did I say *ONLY* enough?

Thanks, Johnny Hughes

Hi Johnny,

Thank you as always, Should I be rebooting servers to ensure that all services are using the new glibc?

sorry for the rookie question, just need some clarification.

thanks

Michael

It depends on your environment : it's adviced to restart the node, but if you can't, you can list the service[s] that depend on libc and (selectively) restart those (like sshd/httpd/postfix/...) on public facing nodes :

lsof +c0 -d DEL | awk 'NR==1 || /libc-/ {print $2,$1,$4,$NF}' | column -t

Source : https://access.redhat.com/articles/2161461

- -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

The easy answer is yes .. glibc requires so many things to be restarted, that is the best bet. Or certainly the easiest.

Note: in CentOS 7, there is also a kernel update which is rated as Important .. so you should boot to that anyway: https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html

Here is a good link to figure out what to restart if you don't want to reboot:

https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-a...

and there is this thread: http://markmail.org/message/dodinyrhwgey35mh

But generalyl, after a glibc update or a kernel update .. rebooting is easiest and it ensures everything is protected.

Wow, so, I updated my server (yum update -y) which applied a new kernel and the new glibc among other things, After the update completed it knocked my master postgresql database offline.

Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter "max_stack_depth": 16384 Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed 7680kB. Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth limit via "ulimit -s" or local equivalent. Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file "/var/lib/pgsql/data/postgresql.conf" contains errors Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server Feb 17 13:46:16 db1 pg_ctl: Examine the log output. Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, code=exited status=1 Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. Feb 17 13:46:16 db1 systemd: postgresql.service failed.

I have kernel parameters specified in /etc/sysctl.conf

vm.swappiness=0 vm.overcommit_memory=2 vm.overcommit_ratio=90 kernel.shmmax=35433480192 kernel.shmall=8650752

After the update my postgresql service could not start because these parameters had been reset, I promptly rebooted to server to re-apply them.

Has something changed?!? after a reboot the service still complained that my max_stack_depth was too high because kernel shmmax and shmall were too low with the same error shown above.

[root@db1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514616 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514616 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

confirms that my entries in /etc/sysctl.conf were ignored.

Why would these not work anymore?

Are the parameters specified elsewhere now?

any information would be very helpful!

Thanks

Michael (slightly more grey now)

On 02/17/2016 08:39 AM, Johnny Hughes wrote:

On 02/17/2016 08:10 AM, Michael H wrote:

...

...

The easy answer is yes .. glibc requires so many things to be restarted, that is the best bet. Or certainly the easiest.

Note: in CentOS 7, there is also a kernel update which is rated as Important .. so you should boot to that anyway: https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html

Here is a good link to figure out what to restart if you don't want to reboot:

https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-a...

and there is this thread: http://markmail.org/message/dodinyrhwgey35mh

But generalyl, after a glibc update or a kernel update .. rebooting is easiest and it ensures everything is protected.

Wow, so, I updated my server (yum update -y) which applied a new kernel and the new glibc among other things, After the update completed it knocked my master postgresql database offline.

Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter "max_stack_depth": 16384 Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed 7680kB. Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth limit via "ulimit -s" or local equivalent. Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file "/var/lib/pgsql/data/postgresql.conf" contains errors Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server Feb 17 13:46:16 db1 pg_ctl: Examine the log output. Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, code=exited status=1 Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. Feb 17 13:46:16 db1 systemd: postgresql.service failed.

I have kernel parameters specified in /etc/sysctl.conf

vm.swappiness=0 vm.overcommit_memory=2 vm.overcommit_ratio=90 kernel.shmmax=35433480192 kernel.shmall=8650752

After the update my postgresql service could not start because these parameters had been reset, I promptly rebooted to server to re-apply them.

Has something changed?!? after a reboot the service still complained that my max_stack_depth was too high because kernel shmmax and shmall were too low with the same error shown above.

[root@db1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514616 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514616 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

confirms that my entries in /etc/sysctl.conf were ignored.

Why would these not work anymore?

Are the parameters specified elsewhere now?

any information would be very helpful!

Thanks

Michael (slightly more grey now)

Since you are talking about SystemD .. I assume c7.

In c7 .. there is a symlink to /etc/sysctl.d/99-sysctl.conf to /etc/sysctl.conf

Have you verified your sysctl.conf actually contains those settings still.

Your best bet on CentOS-7 is to create a new file in /etc/sysctl.d/ called something like 99-postgres.conf and put youjr mods in there. That way it will never change.

Also .. verify all the files in /etc/sysctl.d/ and /etc/sysctl.conf are set to this label for selinux:

unconfined_u:object_r:etc_t:s0

See this for labeling: red.ht/1ooTpiI

But, /etc/sysctl.conf should still work in centos-7.

Looks like that is working (it seems to be reading your /etc/sysctl.conf file) based on your other post in this thread.

On Wed, 17 Feb 2016, Michael H wrote:

ulimit -s is definitely not showing the correct parameter that I specified in /etc/sysctl.conf.

Are you not confusing two unrelated things?

man systemd.exec / LimitSTACK no?

https://bugzilla.redhat.com/show_bug.cgi?id=750923 may point you further.

jh

On 17/02/16 14:39, Johnny Hughes wrote:

On 02/17/2016 08:10 AM, Michael H wrote:

...

...

The easy answer is yes .. glibc requires so many things to be restarted, that is the best bet. Or certainly the easiest.

Note: in CentOS 7, there is also a kernel update which is rated as Important .. so you should boot to that anyway: https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html

Here is a good link to figure out what to restart if you don't want to

...

...

reboot:

https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-a...

and there is this thread:

...

...

http://markmail.org/message/dodinyrhwgey35mh

But generalyl, after a glibc update or a kernel update .. rebooting is easiest and it ensures everything is protected.

Wow, so, I updated my server (yum update -y) which applied a new kernel and the new glibc among other things, After the update completed it knocked my master postgresql database offline.

Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter "max_stack_depth": 16384 Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed 7680kB. Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth limit via "ulimit -s" or local equivalent. Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file "/var/lib/pgsql/data/postgresql.conf" contains errors Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server Feb 17 13:46:16 db1 pg_ctl: Examine the log output. Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, code=exited status=1 Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. Feb 17 13:46:16 db1 systemd: postgresql.service failed.

I have kernel parameters specified in /etc/sysctl.conf

vm.swappiness=0 vm.overcommit_memory=2 vm.overcommit_ratio=90 kernel.shmmax=35433480192 kernel.shmall=8650752

After the update my postgresql service could not start because these parameters had been reset, I promptly rebooted to server to re-apply them.

Has something changed?!? after a reboot the service still complained that my max_stack_depth was too high because kernel shmmax and shmall were too low with the same error shown above.

[root@db1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514616 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514616 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

confirms that my entries in /etc/sysctl.conf were ignored.

Why would these not work anymore?

Are the parameters specified elsewhere now?

any information would be very helpful!

Thanks

Michael (slightly more grey now)

Since you are talking about SystemD .. I assume c7.

In c7 .. there is a symlink to /etc/sysctl.d/99-sysctl.conf to /etc/sysctl.conf

Have you verified your sysctl.conf actually contains those settings still.

Contents are still in tact.

Your best bet on CentOS-7 is to create a new file in /etc/sysctl.d/ called something like 99-postgres.conf and put youjr mods in there. That way it will never change.

Also .. verify all the files in /etc/sysctl.d/ and /etc/sysctl.conf are set to this label for selinux:

unconfined_u:object_r:etc_t:s0

# ll -dZ /etc/sysctl.d drwxr-xr-x. root root system_u:object_r:etc_t:s0 /etc/sysctl.d

# ll -Z /etc/sysctl.conf -rw-r--r--. root root system_u:object_r:system_conf_t:s0 /etc/sysctl.conf

I tried restorecon -Frv /etc/sysctl* to no avail.

Should I manually re-label these or is this an issue with the selinux-policy package having the incorrect defaults?

See this for labeling: red.ht/1ooTpiI

But, /etc/sysctl.conf should still work in centos-7.

Thanks,

Michael

On 17/02/16 14:32, Michael H wrote:

Hi, re-posting this with a more appropriate subject for my reply;

...

The easy answer is yes .. glibc requires so many things to be restarted, that is the best bet. Or certainly the easiest.

Note: in CentOS 7, there is also a kernel update which is rated as Important .. so you should boot to that anyway: https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html

Here is a good link to figure out what to restart if you don't want to reboot:

https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-a...

and there is this thread: http://markmail.org/message/dodinyrhwgey35mh

But generalyl, after a glibc update or a kernel update .. rebooting is easiest and it ensures everything is protected.

Wow, so, I updated my server (yum update -y) which applied a new kernel and the new glibc among other things, After the update completed it knocked my master postgresql database offline.

Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter "max_stack_depth": 16384 Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed 7680kB. Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth limit via "ulimit -s" or local equivalent. Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file "/var/lib/pgsql/data/postgresql.conf" contains errors Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server Feb 17 13:46:16 db1 pg_ctl: Examine the log output. Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, code=exited status=1 Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. Feb 17 13:46:16 db1 systemd: postgresql.service failed.

I have kernel parameters specified in /etc/sysctl.conf

vm.swappiness=0 vm.overcommit_memory=2 vm.overcommit_ratio=90 kernel.shmmax=35433480192 kernel.shmall=8650752

After the update my postgresql service could not start because these parameters had been reset, I promptly rebooted to server to re-apply them.

Has something changed?!? after a reboot the service still complained that my max_stack_depth was too high because kernel shmmax and shmall were too low with the same error shown above.

[root@db1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514616 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514616 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

confirms that my entries in /etc/sysctl.conf were ignored.

Why would these not work anymore?

Are the parameters specified elsewhere now?

any information would be very helpful!

Some additional information;

sysctl -a | grep kernel.shm kernel.shmall = 8650752 kernel.shmmax = 35433480192 kernel.shmmni = 4096

which corresponds to my /etc/sysctl.conf kernel.shmmax=35433480192 kernel.shmall=8650752

but contradicts; ulimit -a [...] stack size (kbytes, -s) 8192 [...]

Any suggestions as to why this has happened?

thanks

Michael

On 17/02/16 19:55, John R Pierce wrote:

On 2/17/2016 6:39 AM, Michael H wrote:

...

Some additional information;

sysctl -a | grep kernel.shm kernel.shmall = 8650752 kernel.shmmax = 35433480192 kernel.shmmni = 4096

which corresponds to my /etc/sysctl.conf kernel.shmmax=35433480192 kernel.shmall=8650752

but contradicts; ulimit -a [...] stack size (kbytes, -s) 8192

SysV Shared Memory has nothing to do with stack size.

note, btw, the latest releases of postgres (I think as of 9.3?) no longer need large values of shmall,shmmax as they now use a different method of allocating the shared_buffers ...

Hi John,

I dived into the issue in a panic, trying to fix something that was completely unrelated. Turns out my service file was overwritten and lost my stack setting. I've resolved it now with a drop-in snippet.

I like the look of the new features in postgresql but we are using postgresql-server.x86_64 9.2.14-1.el7_1.

Thanks for the information,

Michael

On 02/17/2016 07:40 AM, Corey Johnson wrote:

On 2/17/2016 8:01 AM, Johnny Hughes wrote:

...

I normally just let the daily announce post to this list show what is available for updates, but there is a CVE (CVE-2015-7547) that needs a bit more attention which will be on today's announce list of updates.

We released a new glibc yesterday for CentOS-6 and CentOS-7 .. it is VERY important that all users update to these versions: This update is rated as Critical by Red Hat, meaning that it is remotely exploitable under some circumstances. Make sure this update works in your environments and update as soon as you can.

CentOS-7: https://lists.centos.org/pipermail/centos-announce/2016-February/021672.html

https://rhn.redhat.com/errata/RHSA-2016-0176.html

CentOS-6: https://lists.centos.org/pipermail/centos-announce/2016-February/021668.html

https://rhn.redhat.com/errata/RHSA-2016-0175.html

These mitigate CVE-2015-7547: https://access.redhat.com/security/cve/CVE-2015-7547

https://bugzilla.redhat.com/show_bug.cgi?id=1293532

Can't stress how important this update is .. here are a couple stories:

http://arstechnica.com/security/2016/02/extremely-severe-bug-leaves-dizzying...

http://www.theregister.co.uk/2016/02/16/glibc_linux_dns_vulernability/

Please note that the ONLY way this is tested to work is with ALL updates from CentOS-6 or CentOS-7 applied along with the glibc updates. So a yum update with base and updates repo enabled is the ONLY tested scenario. Did I say *ONLY* enough?

I am trying to find conclusive info on whether pre glibc version 2.9 needs to be of concern. I have some older CentOS-5 machines running some older software, and they currently have glibc 2.5-123 installed. Some technical info i have read on this vulnerability states that the issue was introduced in version 2.9. But other less technical articles mention that older version "could" be vulnerable. Would appreciate any comments from the community on this.

Red Hat says no: https://access.redhat.com/security/cve/CVE-2015-7547

Is it possible they are wrong .. I guess, anything is possible.

You can test with this:

https://github.com/fjserna/CVE-2015-7547