[CentOS-devel] RabbitMQ eats 1.6GB ram on CentOS Stream 9

Wed Aug 24 12:34:21 UTC 2022
Marcin Juszkiewicz <marcin.juszkiewicz at linaro.org>

W dniu 22.08.2022 o 10:20, Florian Weimer pisze:
> * Marcin Juszkiewicz:
> 
>> As part of OpenStack deployments we deploy RabbitMQ. During current
>> cycle I looked at moving from CentOS Stream 8 to 9.
>>
>> And RabbitMQ is a problem.
>>
>> When I boot CentOS Stream 9 system and then use just built 'rabbitmq'
>> container memory use of "/usr/lib64/erlang/erts-12.3.2.2/bin/beam.smp"
>> process goes up to 1.6GB ram:
> 
>> If I boot the same container on Debian host then same process uses
>> 0.2GB ram:
> 
> I looked at this briefly and it seems that th Erlang runtime does its
> own memory accounting, so it's probably not related to different kernel
> MM defaults (THP settings in particular).  But the Erlang code is a bit
> impenetrable to me, so I could be wrong about it.
> 
> It could still be related to auto-tuning based on Erlang's
> interpretation of system/cgroup parameters.
> 
> Have you confirmed that the numbers are real?  That for a given amount
> of RAM in a VM (say 8 GB), you can 30 containers at the same time on
> Debian, and only 4 on CentOS?
> 
> The Erlang JIT likely uses the multi-mapping kludge in SELinux mode, and
> that code could have a leak or a memory accounting bug.
> 
> But I think someone who is familiar with the Erlang runtime needs to
> look at this.

Turned out that this is due to amount of available file descriptors...

https://github.com/rabbitmq/erlang-rpm/discussions/104#discussioncomment-3456571

EL9 (both CentOS Stream 9 and Rocky Linux 9) have 1073741816 while 
CentOS Stream 8 has 1048576 - after lowering it's acceptable memory usage.