On Thu, 11 Aug 2022 at 12:37, Marcin Juszkiewicz < marcin.juszkiewicz at linaro.org> wrote: > W dniu 11.08.2022 o 17:34, Stephen Smoogen pisze: > > On Thu, 11 Aug 2022 at 10:56, Marcin Juszkiewicz wrote: > > > >> As part of OpenStack deployments we deploy RabbitMQ. During current > >> cycle I looked at moving from CentOS Stream 8 to 9. > >> > >> And RabbitMQ is a problem. > >> > >> When I boot CentOS Stream 9 system and then use just built > 'rabbitmq' > >> container memory use of > "/usr/lib64/erlang/erts-12.3.2.2/bin/beam.smp" > >> process goes up to 1.6GB ram: > > > > CS9 on CS9 > > >> (rabbitmq)[root at kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown > >> Reporting memory breakdown on node rabbit at kolla-cs9... > >> other_system: 1.6233 gb (68.59%) > >> allocated_unused: 0.5164 gb (21.82%) > > > > CS9 on Debian (versions?) > > Debian 11 'bullseye' with up-to-date packages. > > >> If I boot the same container on Debian host then same process uses > >> 0.2GB ram: > >> > >> (rabbitmq)[root at debian /]# rabbitmq-diagnostics memory_breakdown > >> Reporting memory breakdown on node rabbit at debian... > >> binary: 0.2787 gb (70.2%) > >> code: 0.0355 gb (8.93%) > >> other_system: 0.0255 gb (6.44%) > > > > Debian on CS9 > > > > Stats needed for better than > > (rabbitmq)[root at kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown > Reporting memory breakdown on node rabbit at kolla-cs9... > other_system: 1.6231 gb (73.42%) > allocated_unused: 0.512 gb (23.16%) > code: 0.0355 gb (1.61%) > other_proc: 0.0189 gb (0.85%) > binary: 0.0133 gb (0.6%) > other_ets: 0.0034 gb (0.15%) > plugins: 0.0015 gb (0.07%) > atom: 0.0014 gb (0.06%) > mgmt_db: 4.0e-4 gb (0.02%) > connection_other: 4.0e-4 gb (0.02%) > metrics: 3.0e-4 gb (0.01%) > connection_readers: 2.0e-4 gb (0.01%) > queue_procs: 1.0e-4 gb (0.01%) > mnesia: 1.0e-4 gb (0.0%) > connection_channels: 0.0 gb (0.0%) > msg_index: 0.0 gb (0.0%) > quorum_ets: 0.0 gb (0.0%) > connection_writers: 0.0 gb (0.0%) > stream_queue_procs: 0.0 gb (0.0%) > stream_queue_replica_reader_procs: 0.0 gb (0.0%) > queue_slave_procs: 0.0 gb (0.0%) > quorum_queue_procs: 0.0 gb (0.0%) > stream_queue_coordinator_procs: 0.0 gb (0.0%) > reserved_unallocated: 0.0 gb (0.0%) > > Erlang 1:24.2.1+dfsg-1~bpo11+1 > RabbitMQ 3.9.22-1 > > >> Booted CS9 system and deployed OpenStack using Debian based > containers. > >> Again 1.6GB memory use. > > > >> So let build CS9 based containers using Erlang/RabbitMQ from CentOS > >> Stream 9 "messaging/rabbitmq-38" repository. Again 1.6GB memory use. > > (rabbitmq)[root at kolla-cs9 /]# rpm -qa|egrep "(rabbit|erlang-2)" > erlang-24.3.4.2-1.el9s.x86_64 > rabbitmq-server-3.9.21-1.el9s.x86_64 > > (rabbitmq)[root at kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown > Reporting memory breakdown on node rabbit at kolla-cs9... > other_system: 1.6231 gb (73.98%) > allocated_unused: 0.5107 gb (23.28%) > code: 0.0356 gb (1.62%) > other_proc: 0.018 gb (0.82%) > other_ets: 0.0034 gb (0.16%) > atom: 0.0014 gb (0.06%) > plugins: 9.0e-4 gb (0.04%) > mgmt_db: 4.0e-4 gb (0.02%) > metrics: 2.0e-4 gb (0.01%) > binary: 2.0e-4 gb (0.01%) > mnesia: 1.0e-4 gb (0.0%) > connection_other: 0.0 gb (0.0%) > msg_index: 0.0 gb (0.0%) > quorum_ets: 0.0 gb (0.0%) > stream_queue_procs: 0.0 gb (0.0%) > stream_queue_replica_reader_procs: 0.0 gb (0.0%) > connection_readers: 0.0 gb (0.0%) > connection_writers: 0.0 gb (0.0%) > connection_channels: 0.0 gb (0.0%) > queue_procs: 0.0 gb (0.0%) > queue_slave_procs: 0.0 gb (0.0%) > quorum_queue_procs: 0.0 gb (0.0%) > stream_queue_coordinator_procs: 0.0 gb (0.0%) > reserved_unallocated: 0.0 gb (0.0%) > > > > > So if I cut and summarized your data correctly: > > > > CS9 container of rabbitmq on CS9 base system: Other_system bloats to 1.6 > GB > > CS9 container of rabbitmq on Debian base system: Other system stays at > > 0.025 GB > > Debian container of rabbitmq on CS9 base system: Something bloats to > 1.6GB > > Built-from-scratch container of rabbitmq on CS9 base system: Something > > bloats to 1.6Gb > > It is "other_system" in all 4 causes. > > > > I think we would want to make sure that the 'Something' in those is also > > Other system, and then look to see what is 'Other system' and what > > controls it. After that it would be to see what is 'controlling' that. > > > How was this container run? podman, docker, something else? > > Docker in all situations. We did not migrated to Podman yet. > > > How was this system configured? > > Host OS was minimal install in both (CS9 host, Debian host) cases. > > > Base install of packages? > Only what is needed to connect over SSH and manage (from outside) using > Ansible. > > Later added tmux, vim, mc, htop when I started checking what is going on. > Ok thanks for the data. The following is going to be systems administration by stack overflow: rabbitmqctl eval 'rabbit_diagnostics:top_memory_use(),rabbit_diagnostics:top_binary_refs().' The other_system does not have much in google hits and I have not used any of these applications. I am hoping someone who knows more about this can jump in. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20220811/f5faa682/attachment-0003.html>