Hi Leon,
Thanks a lot for taking a look at my test and the suggestion.
We have tested this suggestion
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Unfortunately, the Ram usage was the same with this option changed as well.
Just to give some additional context to the original question to all,
We see 10-25 percent more RSS on CentOS 8 ( as compared to CentOS 7) for the same Java vm process that we are benchmarking . And in that case , the RSS growth for the executable and the shared library of our application is considerably higher.
Perhaps, for the big server use cases the rss growth did not matter, but for us, we are trying to use it in an embedded scenario and this is a major issue.
We will be glad to hear any further suggestions/ tests that we could check to further understand the differences between the 2 versions.
Regards, -Anand
On Wed, Jul 7, 2021, 5:30 PM centos-request@centos.org wrote:
Send CentOS mailing list submissions to centos@centos.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.centos.org/mailman/listinfo/centos or, via email, send a message with subject or body 'help' to centos-request@centos.org
You can reach the person managing the list at centos-owner@centos.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of CentOS digest..."
Today's Topics:
- RSS usage on centos8.4 is higher for user-application compared to centos7.9 (Anand Babu)
- Re: RSS usage on centos8.4 is higher for user-application compared to centos7.9 (Leon Fauster)
- Re: Centos versions in the future? (Nikolaos Milas)
- Re: Centos versions in the future? (J Martin Rushton)
- Re: Centos versions in the future? (Nikolaos Milas)
- Re: Centos versions in the future? (Leon Fauster)
Message: 1 Date: Tue, 6 Jul 2021 15:02:49 +0200 From: Anand Babu mail2anand@gmail.com To: centos@centos.org Subject: [CentOS] RSS usage on centos8.4 is higher for user-application compared to centos7.9 Message-ID: <CAB5++c4-Z= 7FheHD02RW6YLP6d-trwCqqQqvDdAKsoBZF4tbUg@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
Hi Centos Community,
This is my first time here and I apologize in advance if I made a mistake here and hope you will correct me , if i made any.
With that said,
This is a native example that was written to narrow down the higher RAM usage that we see when we are using Centos8 as opposed to Centos7.
The c-code is very small and looks like this :
#include<stdio.h>
#include<unistd.h>
int main()
{
pause();
return 0;
}
I compile and run the binary on centos7 and memory backed section of pmap output looks like below:
pmap -X $(pidof sleep) | head -n -2 | awk '{ if (NR > 2 && $5 > 0 ) printf "%12s %8s %8s %4s %s\n", $1, $6, $7, $2,$13}' 00400000 4 4 r-xp sleep 00600000 4 4 r--p sleep 00601000 4 4 rw-p sleep*7f54514f9000 1808 204 r-xp libc-2.17.so http://libc-2.17.so *7f54516bd000 2044 0 ---p libc-2.17.so 7f54518bc000 16 16 r--p libc-2.17.so 7f54518c0000 8 8 rw-p libc-2.17.so 7f54518c7000 136 108 r-xp ld-2.17.so 7f5451ae8000 4 4 r--p ld-2.17.so 7f5451ae9000 4 4 rw-p ld-2.17.so
and ps says the following:
** ps -o rss= pidof sleep 352**
Running the same binary on centos8 leads to
** ps -o rss= pidof pause 784 **
and the pmap output looks like below:
00400000 4 4 r-xp pause 00600000 4 4 r--p pause 00601000 4 4 rw-p pause* 7f24029a8000 1776788 r-xp libc-2.28.so http://libc-2.28.so * 7f2402b64000 2044 0 ---p libc-2.28.so 7f2402d63000 16 16 r--p libc-2.28.so 7f2402d67000 8 8 rw-p libc-2.28.so 7f2402d6d000 176 176 r-xp ld-2.28.so 7f2402f99000 4 4 r--p ld-2.28.so 7f2402f9a000 8 8 rw-p ld-2.28.so
For running the same executable, the libc.so takes 788KB(204KB on centos7).
Note:
This is not the only library that is showing this behavior, but we see the same behavior for other shared-library as well as executables that were compiled on centos7. Running the same executables/shared objects take a higher amount of pages on centos8 than on centos7. 2.
Since the glibc version on Centos8 was 2.28 , i have compiled 2.17 version and then used patchelf to patch the centos7 built binary to make use of 2.17 glibc on centos8 host like patchelf --set-interpreter
/home/babu/RSS_measurement/2_17_downloaded/glibc_home/lib/ld-linux-x86-64.so.2 --set-rpath /home/babu/RSS_measurement/2_17_downloaded/glibc_home/lib /tmp/pause
and then run it under LD_DEBUG=libs and verified the 2.17 version of libc libraries were used and still the memory usage on centos8 was higher than on centos7 (about the same numbers as above).
What could be the reason for the higher RAM usage on centos8 vs centos7? I can imagine some defaults have changed on centos8 and that has meant this impact and i could change this default and could reproduce the same RSS numbers on centos8 as well?
I am posting my environment details in the P.S below.
Regards,
-Anand
P.S:
cat /etc/centos-release
CentOS Linux release 8.4.2105
[root@localhost RSS_measurement]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
I am running both the centos versions under Virtualbox.
Message: 2 Date: Tue, 6 Jul 2021 17:59:11 +0200 From: Leon Fauster leonfauster@googlemail.com To: centos@centos.org Subject: Re: [CentOS] RSS usage on centos8.4 is higher for user-application compared to centos7.9 Message-ID: 4e84ea30-9d8b-2e9d-a898-e469303a537c@gmail.com Content-Type: text/plain; charset=utf-8; format=flowed
On 06.07.21 15:02, Anand Babu wrote:
Hi Centos Community,
This is my first time here and I apologize in advance if I made a mistake here and hope you will correct me , if i made any.
With that said,
This is a native example that was written to narrow down the higher RAM usage that we see when we are using Centos8 as opposed to Centos7.
The c-code is very small and looks like this :
#include<stdio.h>
#include<unistd.h>
int main()
{
pause();
return 0;
}
I compile and run the binary on centos7 and memory backed section of pmap output looks like below:
pmap -X $(pidof sleep) | head -n -2 | awk '{ if (NR > 2 && $5 > 0 ) printf "%12s %8s %8s %4s %s\n", $1, $6, $7, $2,$13}' 00400000 4 4 r-xp sleep 00600000 4 4 r--p sleep 00601000 4 4 rw-p sleep*7f54514f9000 1808 204 r-xp libc-2.17.so http://libc-2.17.so *7f54516bd000 2044 0 ---p libc-2.17.so 7f54518bc000 16 16 r--p libc-2.17.so 7f54518c0000 8 8 rw-p libc-2.17.so 7f54518c7000 136 108 r-xp ld-2.17.so 7f5451ae8000 4 4 r--p ld-2.17.so 7f5451ae9000 4 4 rw-p ld-2.17.so
and ps says the following:
** ps -o rss= pidof sleep 352**
Running the same binary on centos8 leads to
** ps -o rss= pidof pause 784 **
and the pmap output looks like below:
00400000 4 4 r-xp pause 00600000 4 4 r--p pause 00601000 4 4 rw-p pause* 7f24029a8000 1776788 r-xp libc-2.28.so http://libc-2.28.so * 7f2402b64000 2044 0 ---p libc-2.28.so 7f2402d63000 16 16 r--p libc-2.28.so 7f2402d67000 8 8 rw-p libc-2.28.so 7f2402d6d000 176 176 r-xp ld-2.28.so 7f2402f99000 4 4 r--p ld-2.28.so 7f2402f9a000 8 8 rw-p ld-2.28.so
For running the same executable, the libc.so takes 788KB(204KB on
centos7).
Note:
1. This is not the only library that is showing this behavior, but wesee
the same behavior for other shared-library as well as executablesthat were
compiled on centos7. Running the same executables/shared objectstake a
higher amount of pages on centos8 than on centos7. 2. Since the glibc version on Centos8 was 2.28 , i have compiled 2.17 version and then used patchelf to patch the centos7 built binary tomake
use of 2.17 glibc on centos8 host like patchelf --set-interpreter/home/babu/RSS_measurement/2_17_downloaded/glibc_home/lib/ld-linux-x86-64.so.2
--set-rpath /home/babu/RSS_measurement/2_17_downloaded/glibc_home/lib /tmp/pause
and then run it under LD_DEBUG=libs and verified the 2.17 version of libc libraries were used and still the memory usage on centos8 was higher than on centos7 (about the same numbers as above).
What could be the reason for the higher RAM usage on centos8 vs centos7?
I
can imagine some defaults have changed on centos8 and that has meant this impact and i could change this default and could reproduce the same RSS numbers on centos8 as well?
How does the results look like after doing:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
-- Leon
Message: 3 Date: Wed, 7 Jul 2021 12:44:15 +0300 From: Nikolaos Milas nmilas@noa.gr To: CentOS mailing list centos@centos.org Subject: Re: [CentOS] Centos versions in the future? Message-ID: 6b3c136f-5aae-1b0a-e65c-868377875f7a@noa.gr Content-Type: text/plain; charset=utf-8; format=flowed
On 30/4/2021 7:27 ?.?., Gionatan Danti wrote:
The correct answer is to buy RH: fine. But do not let Stream touch anything which require a kABI compatible modules. As said above, the Stream move is squarely addresses *cloud* vendor requests and needs. Again, fine. But please leave apart the RH comparison, this is not going to help Stream.
Again, don't let me wrong: I wishes the best to Stream, and I will use it where appropriate. But "where" is much smaller today than yesterday. But this aside, I really thank you all CentOS maintainer for your monumental work, and I really hope Stream will be a success.
I re-visit this thread, since it is crucial for CentOS 8 users.
RESF / Rocky Linux is gaining worldwide recognition and sets itself as the primary organization / platform to become the CentOS 8 heir / successor in the future.
Google and Microsoft become RESF sponsors/partners:
https://rockylinux.org/news/community-update-june-2021/And so IBM/RH lose the tremendous advantage they had by owning the CentOS project, which - it seems - never evaluated correctly.
From now on, it is clear that hundreds of thousands of CentOS installations will be migrating to Rocky Linux.
I also wish the best of success to CentOS Stream, but this is not what the CentOS community expected.
My 2c. Nick
Message: 4 Date: Wed, 7 Jul 2021 10:47:16 +0100 From: J Martin Rushton martinrushton56@btinternet.com To: centos@centos.org Subject: Re: [CentOS] Centos versions in the future? Message-ID: e6298010-bece-df14-98d0-9f8b5e04e8e2@btinternet.com Content-Type: text/plain; charset=utf-8; format=flowed
There's also Alma, which is where I've gone after being with CentOS since 5.3
On 07/07/2021 10:44, Nikolaos Milas wrote:
On 30/4/2021 7:27 ?.?., Gionatan Danti wrote:
The correct answer is to buy RH: fine. But do not let Stream touch anything which require a kABI compatible modules. As said above, the Stream move is squarely addresses *cloud* vendor requests and needs. Again, fine. But please leave apart the RH comparison, this is not going to help Stream.
Again, don't let me wrong: I wishes the best to Stream, and I will use it where appropriate. But "where" is much smaller today than yesterday. But this aside, I really thank you all CentOS maintainer for your monumental work, and I really hope Stream will be a success.
I re-visit this thread, since it is crucial for CentOS 8 users.
RESF / Rocky Linux is gaining worldwide recognition and sets itself as the primary organization / platform to become the CentOS 8 heir / successor in the future.
Google and Microsoft become RESF sponsors/partners:
?? https://rockylinux.org/news/community-update-june-2021/
And so IBM/RH lose the tremendous advantage they had by owning the CentOS project, which - it seems - never evaluated correctly.
From now on, it is clear that hundreds of thousands of CentOS installations will be migrating to Rocky Linux.
I also wish the best of success to CentOS Stream, but this is not what the CentOS community expected.
My 2c. Nick
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-- J Martin Rushton MBCS
Message: 5 Date: Wed, 7 Jul 2021 13:07:12 +0300 From: Nikolaos Milas nmilas@noa.gr To: CentOS mailing list centos@centos.org Subject: Re: [CentOS] Centos versions in the future? Message-ID: ce7e72f7-7159-78ee-4576-e75e806025f7@noa.gr Content-Type: text/plain; charset=utf-8; format=flowed
On 7/7/2021 12:47 ?.?., J Martin Rushton via CentOS wrote:
There's also Alma, which is where I've gone after being with CentOS since 5.3
AlmaLinux is a great project too, IMHO, but things show that the new industry standard (replacing CentOS) will probably be Rocky Linux.
(Yes, RHEL **AND** CentOS have indeed been industry standards - the point of reference -, IMHO, and this is what IBM/RHEL have failed to realize: You don't alter a point of reference.)
It is interesting to see what Service Providers will do with their (huge numbers of) CentOS installations, when they migrate...
From the users/admins' perspective it is to their interest to have robust and healthy alternatives.
In our org, I am now using Rocky Linux on new installations (without issues) and will be migrating several CentOS 8 boxes to Rocky Linux as well.
Cheers, Nick
Message: 6 Date: Wed, 7 Jul 2021 12:28:58 +0200 From: Leon Fauster leonfauster@googlemail.com To: centos@centos.org Subject: Re: [CentOS] Centos versions in the future? Message-ID: 9d2d8304-6779-a452-1a8b-62c58d7722e1@gmail.com Content-Type: text/plain; charset=utf-8; format=flowed
On 07.07.21 12:07, Nikolaos Milas wrote:
On 7/7/2021 12:47 ?.?., J Martin Rushton via CentOS wrote:
There's also Alma, which is where I've gone after being with CentOS since 5.3
AlmaLinux is a great project too, IMHO, but things show that the new industry standard (replacing CentOS) will probably be Rocky Linux.
(Yes, RHEL **AND** CentOS have indeed been industry standards - the point of reference -, IMHO, and this is what IBM/RHEL have failed to realize: You don't alter a point of reference.)
It should not be forgotten that Rocky Linux will be a 1:1 rebuild, also in the future. So, to shape this future everyone is invited to participate at CentOS Stream. This is a great future or not?
It is interesting to see what Service Providers will do with their (huge numbers of) CentOS installations, when they migrate...
From the users/admins' perspective it is to their interest to have robust and healthy alternatives.
In our org, I am now using Rocky Linux on new installations (without issues) and will be migrating several CentOS 8 boxes to Rocky Linux as well.
Cheers, Nick
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Subject: Digest Footer
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
End of CentOS Digest, Vol 198, Issue 6