[CentOS] yum update / kernal update failed - remove or repair

Thu Jan 23 15:22:32 UTC 2020
Simon Matter <simon.matter at invoca.ch>

> On 1/22/20 3:57 PM, Gary Stainburn wrote:
>> I have managed to find out what happened in the yum update and it turns
>> out it was a mess.  It looks like the server ran out of memory in the
>> middle and things then started to fail.  Any advice on how to recover
>> from this would be greatly appreciated
>>
>> The log below shows:
>>
>> [root at vps2 ~]# yum history info 22
>> Loaded plugins: fastestmirror, langpacks
>> Repository mariadb is listed more than once in the configuration
>> Transaction ID : 22
>> Begin time     : Sun Jan 19 18:15:33 2020
>> Begin rpmdb    : 795:6c5cd82e3ada047c7071fea1524f41fd13fab6a1
>> End time       :            19:19:32 2020 (63 minutes)
>> End rpmdb      : 801:6d1b278ccbeef6f68ff494cb28eaed5c7be4124b
>> User           : root <root>
>> Return-Code    : Failure: 1
>> Command Line   : update
>> Transaction performed with:
>>     Installed     rpm-4.11.3-40.el7.x86_64
>> @anaconda
>>     Installed     yum-3.4.3-163.el7.centos.noarch
>> @anaconda
>>     Installed     yum-plugin-fastestmirror-1.1.31-52.el7.noarch
>> @anaconda
>> Packages Altered:
>>     Updated MariaDB-client-10.4.10-1.el7.centos.x86_64        @mariadb
>>     Update                 10.4.11-1.el7.centos.x86_64        @mariadb
>>     Updated MariaDB-common-10.4.10-1.el7.centos.x86_64        @mariadb
>>     Update                 10.4.11-1.el7.centos.x86_64        @mariadb
>>     Updated MariaDB-compat-10.4.10-1.el7.centos.x86_64        @mariadb
>>     Update                 10.4.11-1.el7.centos.x86_64        @mariadb
>>  ** Updated MariaDB-server-10.4.10-1.el7.centos.x86_64        @mariadb
>> Loading mirror speeds from cached hostfile
>>  * base: mirror.as29550.net
>>  * epel: www.mirrorservice.org
>>  * extras: www.mirrorservice.org
>>  * remi-php73: mirrors.ukfast.co.uk
>>  * remi-safe: mirrors.ukfast.co.uk
>>  * updates: www.mirrorservice.org
>>  ** Update                 10.4.11-1.el7.centos.x86_64        @?mariadb
>>  ** Updated MariaDB-shared-10.4.10-1.el7.centos.x86_64        @mariadb
>>     Update                 10.4.11-1.el7.centos.x86_64        @mariadb
>>  ** Updated ca-certificates-2018.2.22-70.0.el7_5.noarch       @anaconda
>>  ** Update                  2019.2.32-76.el7_7.noarch         @?updates
>>     Updated certbot-0.39.0-1.el7.noarch                       @epel
>>     Update          1.0.0-1.el7.noarch                        @epel
>>     Updated clamav-0.101.4-1.el7.x86_64                       @epel
>>     Update         0.101.5-1.el7.x86_64                       @epel
>>     Updated clamav-data-0.101.4-1.el7.noarch                  @epel
>>     Update              0.101.5-1.el7.noarch                  @epel
>>     Updated clamav-devel-0.101.4-1.el7.x86_64                 @epel
>>     Update               0.101.5-1.el7.x86_64                 @epel
>>     Updated clamav-filesystem-0.101.4-1.el7.noarch            @epel
>>     Update                    0.101.5-1.el7.noarch            @epel
>>     Updated clamav-lib-0.101.4-1.el7.x86_64                   @epel
>>     Update             0.101.5-1.el7.x86_64                   @epel
>>     Updated clamav-update-0.101.4-1.el7.x86_64                @epel
>>     Update                0.101.5-1.el7.x86_64                @epel
>>     Updated clamd-0.101.4-1.el7.x86_64                        @epel
>>     Update        0.101.5-1.el7.x86_64                        @epel
>>     Updated curl-7.29.0-54.el7.x86_64                         @anaconda
>>     Update       7.29.0-54.el7_7.1.x86_64                     @updates
>>     Updated fail2ban-0.9.7-1.el7.noarch                       @epel
>>     Update           0.10.4-1.el7.noarch                      @epel
>>     Updated fail2ban-firewalld-0.9.7-1.el7.noarch             @epel
>>     Update                     0.10.4-1.el7.noarch            @epel
>>     Updated fail2ban-sendmail-0.9.7-1.el7.noarch              @epel
>>     Update                    0.10.4-1.el7.noarch             @epel
>>     Updated fail2ban-server-0.9.7-1.el7.noarch                @epel
>>     Update                  0.10.4-1.el7.noarch               @epel
>>     Updated fail2ban-systemd-0.9.7-1.el7.noarch               @epel
>>     Update                   0.10.4-1.el7.noarch              @epel
>>     Updated fribidi-1.0.2-1.el7.x86_64                        @anaconda
>>     Update          1.0.2-1.el7_7.1.x86_64                    @updates
>>     Updated iproute-4.11.0-25.el7.x86_64                      @anaconda
>>     Update          4.11.0-25.el7_7.2.x86_64                  @updates
>>     Install kernel-3.10.0-1062.9.1.el7.x86_64                 @updates
>>     Updated kernel-headers-3.10.0-1062.4.3.el7.x86_64         @updates
>>     Update                 3.10.0-1062.9.1.el7.x86_64         @updates
>>     Updated kernel-tools-3.10.0-1062.4.3.el7.x86_64           @updates
>>     Update               3.10.0-1062.9.1.el7.x86_64           @updates
>>     Updated kernel-tools-libs-3.10.0-1062.4.3.el7.x86_64      @updates
>>     Update                    3.10.0-1062.9.1.el7.x86_64      @updates
>>     Updated libblkid-2.23.2-61.el7.x86_64                     @anaconda
>>     Update           2.23.2-61.el7_7.1.x86_64                 @updates
>>     Updated libcurl-7.29.0-54.el7.x86_64                      @anaconda
>>     Update          7.29.0-54.el7_7.1.x86_64                  @updates
>>     Updated libgs-9.25-2.el7_7.2.x86_64                       @updates
>>     Update        9.25-2.el7_7.3.x86_64                       @updates
>>     Updated libmount-2.23.2-61.el7.x86_64                     @anaconda
>>     Update           2.23.2-61.el7_7.1.x86_64                 @updates
>>     Updated libsmartcols-2.23.2-61.el7.x86_64                 @anaconda
>>     Update               2.23.2-61.el7_7.1.x86_64             @updates
>>     Updated libsss_idmap-1.16.4-21.el7.x86_64                 @anaconda
>>     Update               1.16.4-21.el7_7.1.x86_64             @updates
>>     Updated libsss_nss_idmap-1.16.4-21.el7.x86_64             @anaconda
>>     Update                   1.16.4-21.el7_7.1.x86_64         @updates
>>     Updated libuuid-2.23.2-61.el7.x86_64                      @anaconda
>>     Update          2.23.2-61.el7_7.1.x86_64                  @updates
>>     Updated libwebp7-1.0.2-1.el7.remi.x86_64                  @remi-safe
>>     Update           1.0.3-1.el7.remi.x86_64                  @remi-safe
>>  ** Updated microcode_ctl-2:2.1-53.3.el7_7.x86_64             @updates
>>     Update                2:2.1-53.7.el7_7.x86_64             @updates
>>     Updated nss-3.44.0-4.el7.x86_64                           @anaconda
>>     Update      3.44.0-7.el7_7.x86_64                         @updates
>>     Updated nss-softokn-3.44.0-5.el7.x86_64                   @anaconda
>>     Update              3.44.0-8.el7_7.x86_64                 @updates
>>     Updated nss-softokn-freebl-3.44.0-5.el7.x86_64            @anaconda
>>     Update                     3.44.0-8.el7_7.x86_64          @updates
>>     Updated nss-sysinit-3.44.0-4.el7.x86_64                   @anaconda
>>     Update              3.44.0-7.el7_7.x86_64                 @updates
>>     Updated nss-tools-3.44.0-4.el7.x86_64                     @anaconda
>>     Update            3.44.0-7.el7_7.x86_64                   @updates
>>     Updated nss-util-3.44.0-3.el7.x86_64                      @anaconda
>>     Update           3.44.0-4.el7_7.x86_64                    @updates
>>     Updated numactl-libs-2.0.12-3.el7.x86_64                  @anaconda
>>     Update               2.0.12-3.el7_7.1.x86_64              @updates
>>     Updated pgdg-redhat-repo-42.0-5.noarch
>> @/pgdg-centos10-10-2.noarch
>>     Update                   42.0-6.noarch                    @pgdg10
>>     Updated php-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update      7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-cli-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update          7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-common-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update             7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-gd-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update         7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-json-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update           7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-ldap-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update           7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-mysqlnd-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update              7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-odbc-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update           7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-pdo-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update          7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>  ** Updated php-pear-1:1.10.10-2.el7.remi.noarch
>> @remi-php70
>>  ** Update           1:1.10.10-4.el7.remi.noarch
>> @?remi-php73
>>     Updated php-pgsql-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update            7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-process-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update              7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-xml-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update          7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated php-xmlrpc-7.3.12-1.el7.remi.x86_64
>> @remi-php73
>>     Update             7.3.13-1.el7.remi.x86_64
>> @remi-php73
>>     Updated postgresql10-10.11-1PGDG.rhel7.x86_64             @pgdg10
>>     Update               10.11-2PGDG.rhel7.x86_64             @pgdg10
>>     Updated postgresql10-contrib-10.11-1PGDG.rhel7.x86_64     @pgdg10
>>     Update                       10.11-2PGDG.rhel7.x86_64     @pgdg10
>>     Updated postgresql10-libs-10.11-1PGDG.rhel7.x86_64        @pgdg10
>>     Update                    10.11-2PGDG.rhel7.x86_64        @pgdg10
>>  ** Updated postgresql10-server-10.11-1PGDG.rhel7.x86_64      @pgdg10
>>     Update                      10.11-2PGDG.rhel7.x86_64      @pgdg10
>>  ** Updated proftpd-1.3.5e-7.el7.x86_64                       @epel
>>     Update          1.3.5e-8.el7.x86_64                       @epel
>>     Updated python-perf-3.10.0-1062.4.3.el7.x86_64            @updates
>>     Update              3.10.0-1062.9.1.el7.x86_64            @updates
>>     Updated python-requests-2.6.0-7.el7_7.noarch              @updates
>>     Update                  2.6.0-8.el7_7.noarch              @updates
>>     Updated python2-acme-0.39.0-1.el7.noarch                  @epel
>>     Update               1.0.0-1.el7.noarch                   @epel
>>     Updated python2-certbot-0.39.0-1.el7.noarch               @epel
>>     Update                  1.0.0-1.el7.noarch                @epel
>>     Updated python2-certbot-apache-0.39.0-1.el7.noarch        @epel
>>     Update                         1.0.0-1.el7.noarch         @epel
>>     Updated selinux-policy-3.13.1-252.el7.1.noarch            @updates
>>     Update                 3.13.1-252.el7_7.6.noarch          @updates
>>     Updated selinux-policy-targeted-3.13.1-252.el7.1.noarch   @updates
>>     Update                          3.13.1-252.el7_7.6.noarch @updates
>>     Updated sos-3.7-7.el7.centos.noarch                       @updates
>>     Update      3.7-10.el7.centos.noarch                      @updates
>>  ** Updated sssd-client-1.16.4-21.el7.x86_64                  @anaconda
>>     Update              1.16.4-21.el7_7.1.x86_64              @updates
>>     Updated tcpdump-14:4.9.2-4.el7.x86_64                     @anaconda
>>     Update          14:4.9.2-4.el7_7.1.x86_64                 @updates
>>     Updated util-linux-2.23.2-61.el7.x86_64                   @anaconda
>>     Update             2.23.2-61.el7_7.1.x86_64               @updates
>> Scriptlet output:
>>    1 error: Couldn't fork %post(libsss_idmap-1.16.4-21.el7_7.1.x86_64):
>> Cannot allocate memory
>>    2 error: Couldn't fork
>> %post(libsss_nss_idmap-1.16.4-21.el7_7.1.x86_64): Cannot allocate
>> memory
>>    3 error: Couldn't fork
>> %pre(ca-certificates-2019.2.32-76.el7_7.noarch): Cannot allocate
>> memory
>>    4 error: ca-certificates-2019.2.32-76.el7_7.noarch: install failed
>>    5 error: Couldn't fork %post(certbot-1.0.0-1.el7.noarch): Cannot
>> allocate memory
>>    6 error: Couldn't fork
>> %pre(MariaDB-server-10.4.11-1.el7.centos.x86_64): Cannot allocate
>> memory
>>    7 error: MariaDB-server-10.4.11-1.el7.centos.x86_64: install failed
>>    8 error: Couldn't fork %post(sssd-client-1.16.4-21.el7_7.1.x86_64):
>> Cannot allocate memory
>>    9 error: Couldn't fork
>> %post(kernel-tools-3.10.0-1062.9.1.el7.x86_64): Cannot allocate
>> memory
>>   10 error: Couldn't fork %pre(php-pear-1:1.10.10-4.el7.remi.noarch):
>> Cannot allocate memory
>>   11 error: php-pear-1:1.10.10-4.el7.remi.noarch: install failed
>>   12 warning: /etc/clamd.d/scan.conf created as
>> /etc/clamd.d/scan.conf.rpmnew
>>   13 error: Couldn't fork
>> %triggerin(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate
>> memory
>>   14 error: php-pear-1:1.10.10-2.el7.remi.noarch: erase skipped
>>   15 error: MariaDB-server-10.4.10-1.el7.centos.x86_64: erase skipped
>>   16 error: Couldn't fork
>> %preun(MariaDB-shared-10.4.10-1.el7.centos.x86_64): Cannot allocate
>> memory
>>   17 error: MariaDB-shared-10.4.10-1.el7.centos.x86_64: erase failed
>>   18 error: Couldn't fork
>> %postun(MariaDB-compat-10.4.10-1.el7.centos.x86_64): Cannot allocate
>> memory
>>   19 error: ca-certificates-2018.2.22-70.0.el7_5.noarch: erase skipped
>>   20 error: Couldn't fork %preun(microcode_ctl-2:2.1-53.3.el7_7.x86_64):
>> Cannot allocate memory
>>   21 error: microcode_ctl-2:2.1-53.3.el7_7.x86_64: erase failed
>>   22 error: Couldn't fork
>> %preun(postgresql10-server-10.11-1PGDG.rhel7.x86_64): Cannot allocate
>> memory
>>   23 error: postgresql10-server-10.11-1PGDG.rhel7.x86_64: erase failed
>>   24 error: Couldn't fork %postun(libmount-2.23.2-61.el7.x86_64): Cannot
>> allocate memory
>>   25 error: Couldn't fork %preun(sssd-client-1.16.4-21.el7.x86_64):
>> Cannot allocate memory
>>   26 error: sssd-client-1.16.4-21.el7.x86_64: erase failed
>>   27 error: Couldn't fork %postun(libblkid-2.23.2-61.el7.x86_64): Cannot
>> allocate memory
>>   28 error: Couldn't fork
>> %postun(postgresql10-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
>>   29 error: Couldn't fork %postun(clamav-lib-0.101.4-1.el7.x86_64):
>> Cannot allocate memory
>>   30 error: Couldn't fork
>> %postun(kernel-tools-3.10.0-1062.4.3.el7.x86_64): Cannot allocate
>> memory
>>   31 error: Couldn't fork %postun(libcurl-7.29.0-54.el7.x86_64): Cannot
>> allocate memory
>>   32 error: Couldn't fork %postun(nss-3.44.0-4.el7.x86_64): Cannot
>> allocate memory
>>   33 error: Couldn't fork %postun(nss-softokn-3.44.0-5.el7.x86_64):
>> Cannot allocate memory
>>   34 error: Couldn't fork %postun(nss-util-3.44.0-3.el7.x86_64): Cannot
>> allocate memory
>>   35 error: Couldn't fork
>> %postun(postgresql10-libs-10.11-1PGDG.rhel7.x86_64): Cannot allocate
>> memory
>>   36 error: Couldn't fork %postun(libuuid-2.23.2-61.el7.x86_64): Cannot
>> allocate memory
>>   37 error: Couldn't fork %postun(libsss_idmap-1.16.4-21.el7.x86_64):
>> Cannot allocate memory
>>   38 error: Couldn't fork
>> %postun(libsss_nss_idmap-1.16.4-21.el7.x86_64): Cannot allocate memory
>>   39 error: Couldn't fork %postun(libsmartcols-2.23.2-61.el7.x86_64):
>> Cannot allocate memory
>>   40 error: Couldn't fork %postun(libwebp7-1.0.2-1.el7.remi.x86_64):
>> Cannot allocate memory
>>   41 error: Couldn't fork %postun(fribidi-1.0.2-1.el7.x86_64): Cannot
>> allocate memory
>>   42 error: Couldn't fork %preun(proftpd-1.3.5e-7.el7.x86_64): Cannot
>> allocate memory
>>   43 error: proftpd-1.3.5e-7.el7.x86_64: erase failed
>>   44 error: Couldn't fork %postun(numactl-libs-2.0.12-3.el7.x86_64):
>> Cannot allocate memory
>>   45 error: Couldn't fork %posttrans(kernel-3.10.0-1062.9.1.el7.x86_64):
>> Cannot allocate memory
>>   46 error: Couldn't fork
>> %posttrans(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate
>> memory
>> history info
>> [root at vps2 ~]#
>>
>
> Before you try the update again, you'll have to fix the reason for the
> failure - add memory, or at least add a swap file.
>
> You could check with
> rpm -qa --last | head -20
> which the latest packages are that were installed.
>
> If the rpm database is corrupted, rebuild it with
> rpm --rebuilddb
>
> You can reinstall packages that may not be completely installed, using
> rpm --reinstall PACKAGE_FILE

Well, there may be more to check and above steps may not help.

Do you have duplicates in rpmdb? Which one of the duplicates are already
on the filesystem?

Regards,
Simon