[Ci-users] Disabling dnf execution in background on an EC2 bare metal node

Tue Sep 13 08:07:16 UTC 2022
Anoop C S <anoopcs at disr.it>


Post migration to new Duffy API we started noticing the following RPM
error intermittently on bare metal nodes reserved from EC2 pool:

. . .
Running transaction check
Waiting for process with pid 2881 to finish.
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
RPM: error: db5 error(-30969) from dbenv->open: BDB0091
DB_VERSION_MISMATCH: Database environment version mismatch
RPM: error: cannot open Packages index using db5 -  (-30969)
RPM: error: cannot open Packages database in /var/lib/rpm
The downloaded packages were saved in cache until the next successful
You can remove cached packages by executing 'dnf clean packages'.
Error: Could not run transaction.
. . .

Especially the "Waiting for process with pid 2881 to finish." indicates
some DNF operation in background which may or may not conflict with
current DNF execution. With the help of others I got to know about a
cloud-init service performing `dnf update` to cope with the very old
AMI image with which it is provisioned. Please note that nodes are
already marked ready in the pool before `dnf update` kicks in.

Therefore I put forward a suggestion to disable `dnf update` as part of
cloud-init service such that it does not interfere with other DNF
operations done after the node is reserved by a tenant.

Please feel free to correct me in any of the details mentioned above
and let me know your thoughts.

Anoop C S.