On 07/01/2013 01:21 PM Reindl Harald wrote:
Am 01.07.2013 19:16, schrieb ken:
About thirty minutes ago I started an update. Everything downloaded fine. But when it came to "Running Transaction", it hung:
Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : xulrunner 1/18
And there it sits still.
There's plenty of disk space. The load on the CPU is under 1.0. There is some disk activity, but no way so much as to cause anything to hang and no major writing related to yum. Disk reads are just about null, except just now when I grep'ed for something to make sure that the disk could be read. I'm not using even half of the available RAM
what about look in dmesg and/or /var/log/messages?
No. Nothing relevant there.
On 07/01/2013 12:25 PM, ken wrote:
On 07/01/2013 01:21 PM Reindl Harald wrote:
Am 01.07.2013 19:16, schrieb ken:
About thirty minutes ago I started an update. Everything downloaded fine. But when it came to "Running Transaction", it hung:
Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : xulrunner 1/18
And there it sits still.
There's plenty of disk space. The load on the CPU is under 1.0. There is some disk activity, but no way so much as to cause anything to hang and no major writing related to yum. Disk reads are just about null, except just now when I grep'ed for something to make sure that the disk could be read. I'm not using even half of the available RAM
what about look in dmesg and/or /var/log/messages?
No. Nothing relevant there.
Normally, there are a couple of big things to check:
1. DNS is properly resolving the mirror and your the computer doing the update names. 2. You do not have a full /var/ or /tmp/ partition 3. You do not have a read only filesystem (usually due to a driver error ... full partitions will act the same way). Make sure you can write to /var/ and /tmp/ 4. Look at bandwidth to ensure it is not slow, look for a transparent webproxy in the path, etc.
On 07/01/2013 02:04 PM Johnny Hughes wrote:
On 07/01/2013 12:25 PM, ken wrote:
On 07/01/2013 01:21 PM Reindl Harald wrote:
Am 01.07.2013 19:16, schrieb ken:
About thirty minutes ago I started an update. Everything downloaded fine. But when it came to "Running Transaction", it hung:
Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : xulrunner 1/18
And there it sits still.
There's plenty of disk space. The load on the CPU is under 1.0. There is some disk activity, but no way so much as to cause anything to hang and no major writing related to yum. Disk reads are just about null, except just now when I grep'ed for something to make sure that the disk could be read. I'm not using even half of the available RAM
what about look in dmesg and/or /var/log/messages?
No. Nothing relevant there.
Normally, there are a couple of big things to check:
- DNS is properly resolving the mirror and your the computer doing the
update names. 2. You do not have a full /var/ or /tmp/ partition 3. You do not have a read only filesystem (usually due to a driver error ... full partitions will act the same way). Make sure you can write to /var/ and /tmp/ 4. Look at bandwidth to ensure it is not slow, look for a transparent webproxy in the path, etc.
yum already got past the downloading phase of the update process (see above), and as said above also, the filesystem is fine. So none of those things come into play here.
try using strace -p PID-OF-YUM and output to mailinglist
-- Eero
2013/7/1 ken gebser@mousecar.com
On 07/01/2013 02:04 PM Johnny Hughes wrote:
On 07/01/2013 12:25 PM, ken wrote:
On 07/01/2013 01:21 PM Reindl Harald wrote:
Am 01.07.2013 19:16, schrieb ken:
About thirty minutes ago I started an update. Everything downloaded fine. But when it came to "Running Transaction", it hung:
Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : xulrunner 1/18
And there it sits still.
There's plenty of disk space. The load on the CPU is under 1.0.
There
is some disk activity, but no way so much as to cause anything to hang and no major writing related to yum. Disk reads are just about null, except just now when I grep'ed for something to make sure that the
disk
could be read. I'm not using even half of the available RAM
what about look in dmesg and/or /var/log/messages?
No. Nothing relevant there.
Normally, there are a couple of big things to check:
- DNS is properly resolving the mirror and your the computer doing the
update names. 2. You do not have a full /var/ or /tmp/ partition 3. You do not have a read only filesystem (usually due to a driver error ... full partitions will act the same way). Make sure you can write to /var/ and /tmp/ 4. Look at bandwidth to ensure it is not slow, look for a transparent webproxy in the path, etc.
yum already got past the downloading phase of the update process (see above), and as said above also, the filesystem is fine. So none of those things come into play here.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 07/01/2013 02:46 PM Eero Volotinen wrote:
try using strace -p PID-OF-YUM and output to mailinglist
Here's the last few lines of strace:
... open("/var/lib/rpm/__db.002", O_RDWR|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 mmap2(NULL, 1318912, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb79db000 close(4) = 0 stat64("/var/lib/rpm/__db.003", {st_mode=S_IFREG|0644, st_size=450560, ...}) = 0 open("/var/lib/rpm/__db.003", O_RDWR|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 mmap2(NULL, 450560, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb796d000 close(4) = 0 stat64("/var/lib/rpm/Packages", {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 read(4, "\0\0\0\0\1\0\0\0\0\0\0\0a\25\6\0\10\0\0\0\0\20\0\0\0\10\0\0YV\0\0"..., 512) = 512 close(4) = 0 open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 fstat64(4, {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 futex(0xb79b56ac, FUTEX_WAIT, 1, NULL
It hangs there.
Significantly, this 'rpm' command hangs also:
# rpm -vv -V rpm D: opening db environment /var/lib/rpm/Packages joinenv D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
So the problem may well be with 'rpm'.
try strace -f rpm -vv -V rpm
2013/7/1 ken gebser@mousecar.com
On 07/01/2013 02:46 PM Eero Volotinen wrote:
try using strace -p PID-OF-YUM and output to mailinglist
Here's the last few lines of strace:
... open("/var/lib/rpm/__db.002", O_RDWR|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 mmap2(NULL, 1318912, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb79db000 close(4) = 0 stat64("/var/lib/rpm/__db.003"**, {st_mode=S_IFREG|0644, st_size=450560, ...}) = 0 open("/var/lib/rpm/__db.003", O_RDWR|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 mmap2(NULL, 450560, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb796d000 close(4) = 0 stat64("/var/lib/rpm/Packages"**, {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 read(4, "\0\0\0\0\1\0\0\0\0\0\0\0a\25**6\0\10\0\0\0\0\20\0\0\0\10\0**0YV\0\0"..., 512) = 512 close(4) = 0 open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 fstat64(4, {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 futex(0xb79b56ac, FUTEX_WAIT, 1, NULL
It hangs there.
Significantly, this 'rpm' command hangs also:
# rpm -vv -V rpm D: opening db environment /var/lib/rpm/Packages joinenv D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
So the problem may well be with 'rpm'.
On 07/01/2013 03:34 PM Eero Volotinen wrote:
try strace -f rpm -vv -V rpm
Because it looked like the source of the problem (at least one problem) I mv'd off /var/lib/rpm/Packages and did a 'rpm --rebuilddb'. Now it's not hanging anymore, but now...
# yum -d 10 info yum ... Running "config" handler for "verify" plugin Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 309, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 157, in main base.getOptionsConfig(args) File "/usr/share/yum-cli/cli.py", line 187, in getOptionsConfig self.conf File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 665, in <lambda> conf = property(fget=lambda self: self._getConfig(), File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 240, in _getConfig self._conf = config.readMainConfig(startupconf) File "/usr/lib/python2.4/site-packages/yum/config.py", line 805, in readMainConfig yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg) File "/usr/lib/python2.4/site-packages/yum/config.py", line 884, in _getsysver hdr = idx.next() StopIteration
Per your request:
# strace -f rpm -vv -V rpm ... (many lines) ... write(2, "closed db index /var/lib"..., 46closed db index /var/lib/rpm/Packages ) = 46 munmap(0xb7c22000, 450560) = 0 munmap(0xb7c90000, 1318912) = 0 munmap(0xb7ff7000, 24576) = 0 write(2, "D: ", 3D: ) = 3 write(2, "closed db environment /var/lib"..., 46closed db environment /var/lib/rpm/Packages ) = 46 rt_sigaction(SIGHUP, {SIG_DFL, [], 0}, NULL, 8) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], 0}, NULL, 8) = 0 rt_sigaction(SIGTERM, {SIG_DFL, [], 0}, NULL, 8) = 0 rt_sigaction(SIGQUIT, {SIG_DFL, [], 0}, NULL, 8) = 0 rt_sigaction(SIGPIPE, {0x1, [], 0}, NULL, 8) = 0 open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=25462, ...}) = 0 mmap2(NULL, 25462, PROT_READ, MAP_SHARED, 3, 0) = 0xb7ff6000 close(3) = 0 futex(0x4faa4c, FUTEX_WAKE_PRIVATE, 2147483647) = 0 write(2, "D: ", 3D: ) = 3 write(2, "May free Score board((nil))\n", 28May free Score board((nil)) ) = 28 brk(0x8f63000) = 0x8f63000 exit_group(2) = ?
[Ends there... without hanging anymore.]
ken wrote:
On 07/01/2013 03:34 PM Eero Volotinen wrote:
try strace -f rpm -vv -V rpm
Because it looked like the source of the problem (at least one problem) I mv'd off /var/lib/rpm/Packages and did a 'rpm --rebuilddb'. Now it's not hanging anymore, but now...
<snip> One step at a time. Did you do a yum clean all after the rebuild?
mark
On 07/01/2013 03:58 PM m.roth@5-cent.us wrote:
ken wrote:
On 07/01/2013 03:34 PM Eero Volotinen wrote:
try strace -f rpm -vv -V rpm
Because it looked like the source of the problem (at least one problem) I mv'd off /var/lib/rpm/Packages and did a 'rpm --rebuilddb'. Now it's not hanging anymore, but now...
<snip> One step at a time. Did you do a yum clean all after the rebuild?
# yum clean all Loaded plugins: aliases, allowdowngrade, changelog, downloadonly, fastestmirror, : filter-data, kernel-module, keys, kmod, list-data, merge-conf, : priorities, security, tsflags, verify Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 309, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 157, in main base.getOptionsConfig(args) File "/usr/share/yum-cli/cli.py", line 187, in getOptionsConfig self.conf File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 665, in <lambda> conf = property(fget=lambda self: self._getConfig(), File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 240, in _getConfig self._conf = config.readMainConfig(startupconf) File "/usr/lib/python2.4/site-packages/yum/config.py", line 805, in readMainConfig yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg) File "/usr/lib/python2.4/site-packages/yum/config.py", line 884, in _getsysver hdr = idx.next() StopIteration
Pretty much the same happens here too:
# yum info firefox Loaded plugins: aliases, allowdowngrade, changelog, downloadonly, fastestmirror, : filter-data, kernel-module, keys, kmod, list-data, merge-conf, : priorities, security, tsflags, verify Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 309, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 157, in main base.getOptionsConfig(args) File "/usr/share/yum-cli/cli.py", line 187, in getOptionsConfig self.conf File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 665, in <lambda> conf = property(fget=lambda self: self._getConfig(), File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 240, in _getConfig self._conf = config.readMainConfig(startupconf) File "/usr/lib/python2.4/site-packages/yum/config.py", line 805, in readMainConfig yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg) File "/usr/lib/python2.4/site-packages/yum/config.py", line 884, in _getsysver hdr = idx.next() StopIteration
# cat -n /usr/bin/yum ... 26 sys.path.insert(0, '/usr/share/yum-cli') 27 try: 28 import yummain 29 yummain.user_main(sys.argv[1:], exit_code=True) 30 except KeyboardInterrupt, e: 31 print >> sys.stderr, "\n\nExiting on user cancel." 32 sys.exit(1) [eof]
I don't do python, but it looks like a library file is called at line 29, I'm guessing /usr/share/yum-cli/yummain.py
That's a long file too... well, a bit over 300 lines. And then it calls another python library, which bails with a "StopIteration". I'm in the woods now.
ken wrote:
On 07/01/2013 02:46 PM Eero Volotinen wrote:
try using strace -p PID-OF-YUM and output to mailinglist
Here's the last few lines of strace:
<snip>
... open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 fstat64(4, {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 futex(0xb79b56ac, FUTEX_WAIT, 1, NULL
It hangs there.
Significantly, this 'rpm' command hangs also:
# rpm -vv -V rpm D: opening db environment /var/lib/rpm/Packages joinenv D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
So the problem may well be with 'rpm'.
You *might* want to consider rpm --rebuilddb
I forget, when I did that, if I had to get rid of/move /var/lib/rpm/__db.*
mark
On 07/01/2013 03:46 PM m.roth@5-cent.us wrote:
ken wrote:
On 07/01/2013 02:46 PM Eero Volotinen wrote:
try using strace -p PID-OF-YUM and output to mailinglist
Here's the last few lines of strace:
<snip> > ... > open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4 > fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 > fstat64(4, {st_mode=S_IFREG|0644, st_size=110592000, ...}) = 0 > futex(0xb79b56ac, FUTEX_WAIT, 1, NULL > > It hangs there. > > Significantly, this 'rpm' command hangs also: > > # rpm -vv -V rpm > D: opening db environment /var/lib/rpm/Packages joinenv > D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 > > So the problem may well be with 'rpm'.
You *might* want to consider rpm --rebuilddb
I forget, when I did that, if I had to get rid of/move /var/lib/rpm/__db.*
mark
Yep. I'd done that... since both rpm and yum were hanging on it, it seemed the obvious things to do. See one of my previous emails. I moved off the previous Packages. It was huge compared to the new one:
# ll Packages* -rw-r--r-- 1 root root 12288 Jul 1 15:35 Packages -rw-r--r-- 1 root root 110592000 Jun 17 19:36 Packages.orig
-
Eero Volotinen -
Johnny Hughes -
ken -
m.roth＠5-cent.us