On Thu, 11 Aug 2022 at 12:37, Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote:
W dniu 11.08.2022 o 17:34, Stephen Smoogen pisze:
On Thu, 11 Aug 2022 at 10:56, Marcin Juszkiewicz wrote:
As part of OpenStack deployments we deploy RabbitMQ. During current cycle I looked at moving from CentOS Stream 8 to 9. And RabbitMQ is a problem. When I boot CentOS Stream 9 system and then use just built'rabbitmq'
container memory use of"/usr/lib64/erlang/erts-12.3.2.2/bin/beam.smp"
process goes up to 1.6GB ram:CS9 on CS9
(rabbitmq)[root@kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown Reporting memory breakdown on node rabbit@kolla-cs9... other_system: 1.6233 gb (68.59%) allocated_unused: 0.5164 gb (21.82%)CS9 on Debian (versions?)
Debian 11 'bullseye' with up-to-date packages.
If I boot the same container on Debian host then same process uses 0.2GB ram: (rabbitmq)[root@debian /]# rabbitmq-diagnostics memory_breakdown Reporting memory breakdown on node rabbit@debian... binary: 0.2787 gb (70.2%) code: 0.0355 gb (8.93%) other_system: 0.0255 gb (6.44%)Debian on CS9
Stats needed for better than
(rabbitmq)[root@kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown Reporting memory breakdown on node rabbit@kolla-cs9... other_system: 1.6231 gb (73.42%) allocated_unused: 0.512 gb (23.16%) code: 0.0355 gb (1.61%) other_proc: 0.0189 gb (0.85%) binary: 0.0133 gb (0.6%) other_ets: 0.0034 gb (0.15%) plugins: 0.0015 gb (0.07%) atom: 0.0014 gb (0.06%) mgmt_db: 4.0e-4 gb (0.02%) connection_other: 4.0e-4 gb (0.02%) metrics: 3.0e-4 gb (0.01%) connection_readers: 2.0e-4 gb (0.01%) queue_procs: 1.0e-4 gb (0.01%) mnesia: 1.0e-4 gb (0.0%) connection_channels: 0.0 gb (0.0%) msg_index: 0.0 gb (0.0%) quorum_ets: 0.0 gb (0.0%) connection_writers: 0.0 gb (0.0%) stream_queue_procs: 0.0 gb (0.0%) stream_queue_replica_reader_procs: 0.0 gb (0.0%) queue_slave_procs: 0.0 gb (0.0%) quorum_queue_procs: 0.0 gb (0.0%) stream_queue_coordinator_procs: 0.0 gb (0.0%) reserved_unallocated: 0.0 gb (0.0%)
Erlang 1:24.2.1+dfsg-1~bpo11+1 RabbitMQ 3.9.22-1
Booted CS9 system and deployed OpenStack using Debian basedcontainers.
Again 1.6GB memory use.So let build CS9 based containers using Erlang/RabbitMQ from CentOS Stream 9 "messaging/rabbitmq-38" repository. Again 1.6GB memory use.(rabbitmq)[root@kolla-cs9 /]# rpm -qa|egrep "(rabbit|erlang-2)" erlang-24.3.4.2-1.el9s.x86_64 rabbitmq-server-3.9.21-1.el9s.x86_64
(rabbitmq)[root@kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown Reporting memory breakdown on node rabbit@kolla-cs9... other_system: 1.6231 gb (73.98%) allocated_unused: 0.5107 gb (23.28%) code: 0.0356 gb (1.62%) other_proc: 0.018 gb (0.82%) other_ets: 0.0034 gb (0.16%) atom: 0.0014 gb (0.06%) plugins: 9.0e-4 gb (0.04%) mgmt_db: 4.0e-4 gb (0.02%) metrics: 2.0e-4 gb (0.01%) binary: 2.0e-4 gb (0.01%) mnesia: 1.0e-4 gb (0.0%) connection_other: 0.0 gb (0.0%) msg_index: 0.0 gb (0.0%) quorum_ets: 0.0 gb (0.0%) stream_queue_procs: 0.0 gb (0.0%) stream_queue_replica_reader_procs: 0.0 gb (0.0%) connection_readers: 0.0 gb (0.0%) connection_writers: 0.0 gb (0.0%) connection_channels: 0.0 gb (0.0%) queue_procs: 0.0 gb (0.0%) queue_slave_procs: 0.0 gb (0.0%) quorum_queue_procs: 0.0 gb (0.0%) stream_queue_coordinator_procs: 0.0 gb (0.0%) reserved_unallocated: 0.0 gb (0.0%)
So if I cut and summarized your data correctly:
CS9 container of rabbitmq on CS9 base system: Other_system bloats to 1.6
GB
CS9 container of rabbitmq on Debian base system: Other system stays at 0.025 GB Debian container of rabbitmq on CS9 base system: Something bloats to
1.6GB
Built-from-scratch container of rabbitmq on CS9 base system: Something bloats to 1.6Gb
It is "other_system" in all 4 causes.
I think we would want to make sure that the 'Something' in those is also Other system, and then look to see what is 'Other system' and what controls it. After that it would be to see what is 'controlling' that.
How was this container run? podman, docker, something else?
Docker in all situations. We did not migrated to Podman yet.
How was this system configured?
Host OS was minimal install in both (CS9 host, Debian host) cases.
Base install of packages?
Only what is needed to connect over SSH and manage (from outside) using Ansible.
Later added tmux, vim, mc, htop when I started checking what is going on.
Ok thanks for the data. The following is going to be systems administration by stack overflow:
rabbitmqctl eval 'rabbit_diagnostics:top_memory_use(),rabbit_diagnostics:top_binary_refs().'
The other_system does not have much in google hits and I have not used any of these applications. I am hoping someone who knows more about this can jump in.