This morning I applied the 13 or so new updates to my servers. On one of them the ssh service and clients stopped working immediately after the update. I restarted the server in anticipation that there might be some instability introduced by updating on a system with active ssh connections. However, this has not cleared the problem.
The packages in question are: openssh.i386 0:4.3p2-41.el5_5.1 openssh-askpass.i386 0:4.3p2-41.el5_5.1 openssh-clients.i386 0:4.3p2-41.el5_5.1 openssh-server.i386 0:4.3p2-41.el5_5.1
The error I am getting when attempting to start the sshd service is this:
Starting sshd: Auto configuration failed 6486:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
Can anyone reading this inform me as to what this means, what the likely cause is, and how it may be repaired? I have webmin access to the server and, as it is inside our firewall I can probably enable telnet to get a terminal window do carry out modifications. But I need to know what it is that I need to modify.
Alternatively, can anyone point me to a reference regarding removing the current version of openssh and reverting to the prior version, which worked with presumably the same user configuration that the present version does not accept. The only references that I can find respecting this error message are all several years old and some of them suggest that their is a problem with accessing or using /dev/random or /dev/urandom.
This matter is somewhat urgent. I have temporarily routed essential ssh connections through a spare host but the box affected sits in front of our legacy systems providing secure access to them as their native OSs and telecom protocols do not support encryption. It is very important that the SSHD service be restored on this host as soon as is possible.
Any help with this is gratefully appreciated.
On Mon, September 13, 2010 11:01, James B. Byrne wrote:
This morning I applied the 13 or so new updates to my servers. On one of them the ssh service and clients stopped working immediately after the update.
. . .
The error I am getting when attempting to start the sshd service is this:
Starting sshd: Auto configuration failed 6486:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
I replaced the sshd_config on the affected server with a copy of that which came with the package and the same error occurs. Whatever is causing this, it does not seem to be related the the sshd_config file.
On: Mon Sep 13 11:41:17 EDT 2010, Joseph L. Casale jcasale at activenetwerx.com wrote:
Selinux enabled?
Yes.
On 09/13/2010 08:55 AM, James B. Byrne wrote:
On: Mon Sep 13 11:41:17 EDT 2010, Joseph L. Casale jcasale at activenetwerx.com wrote:
Selinux enabled?
Yes.
Then you should check your logs to see if SELinux is blocking it for some reason. You could also try turning SELinux off to directly test whether it makes a difference.
Set selinux=permissive in /etc/selinux/config Rebooted system
tried to restart sshd
Starting sshd: Auto configuration failed 3600:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
Same error. I point out at that the other servers that were updated and which show no error all have selinux enabled as well.
Set selinux=permissive in /etc/selinux/config Rebooted system
tried to restart sshd
Starting sshd: Auto configuration failed 3600:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
Same error. I point out at that the other servers that were updated and which show no error all have selinux enabled as well.
Run an `rpm -Va` maybe the ssh package or a one it needs had something tank in the upgrade...
James B. Byrne wrote:
This morning I applied the 13 or so new updates to my servers. On one of them the ssh service and clients stopped working immediately after the update. I restarted the server in anticipation that there might be some instability introduced by updating on a system with active ssh connections. However, this has not cleared the problem.
The packages in question are: openssh.i386 0:4.3p2-41.el5_5.1 openssh-askpass.i386 0:4.3p2-41.el5_5.1 openssh-clients.i386 0:4.3p2-41.el5_5.1 openssh-server.i386 0:4.3p2-41.el5_5.1
The error I am getting when attempting to start the sshd service is this:
Starting sshd: Auto configuration failed 6486:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
[ What I'm about to write is just the result of a google search. Nevertheless, it may help ]
Searching google for "configuration file routines:STR_COPY:variable has no value:conf_def.c" seems to point at openssl as the culprit (mainly, something wrong with pkcs11 and cert "handshaking": the message apears associated with in certain bind9 and openvpn similar crashes). Are you sure nothing tanked on this package at the time of this machine's upgrade (see Joseph L. Casale's sugestion) ?
HTH, Mário
On Mon Sep 13 12:34:49 EDT 2010, Joseph L. Casale jcasale at activenetwerx.com wrote:
Run an `rpm -Va` maybe the ssh package or a one it needs had something tank in the upgrade...
Did that via webmin's command interface and nothing changed insofar as I can see. Same error obtained when starting sshd.
What is the procedure to remove the latest openssh packages and replace them with the previous ones?
This is the entire session for the update on that server:
[root@inet01 ~]# yum update Loaded plugins: fastestmirror, priorities Loading mirror speeds from cached hostfile * addons: centos.mirror.iweb.ca * base: centos.mirror.iweb.ca * extras: centos.mirror.iweb.ca * updates: centos.mirror.iweb.ca Excluding Packages from CentOS-5 - Base Finished Excluding Packages from CentOS-5 - Updates Finished Setting up Update Process Resolving Dependencies --> Running transaction check ---> Package apr.i386 0:1.2.7-11.el5_5.2 set to be updated ---> Package apr-devel.i386 0:1.2.7-11.el5_5.2 set to be updated ---> Package dhclient.i386 12:3.0.5-23.el5_5.2 set to be updated ---> Package expat.i386 0:1.95.8-8.3.el5_5.3 set to be updated ---> Package expat-devel.i386 0:1.95.8-8.3.el5_5.3 set to be updated ---> Package ksh.i386 0:20100202-1.el5_5.1 set to be updated ---> Package nspr.i386 0:4.8.6-1.el5_5 set to be updated ---> Package nspr-devel.i386 0:4.8.6-1.el5_5 set to be updated ---> Package nss.i386 0:3.12.7-2.el5.centos set to be updated ---> Package nss-devel.i386 0:3.12.7-2.el5.centos set to be updated ---> Package nss-tools.i386 0:3.12.7-2.el5.centos set to be updated ---> Package openssh.i386 0:4.3p2-41.el5_5.1 set to be updated ---> Package openssh-askpass.i386 0:4.3p2-41.el5_5.1 set to be updated ---> Package openssh-clients.i386 0:4.3p2-41.el5_5.1 set to be updated ---> Package openssh-server.i386 0:4.3p2-41.el5_5.1 set to be updated ---> Package poppler.i386 0:0.5.4-4.4.el5_5.13 set to be updated ---> Package poppler-utils.i386 0:0.5.4-4.4.el5_5.13 set to be updated ---> Package popt.i386 0:1.10.2.3-20.el5_5.1 set to be updated ---> Package rpm.i386 0:4.4.2.3-20.el5_5.1 set to be updated ---> Package rpm-build.i386 0:4.4.2.3-20.el5_5.1 set to be updated ---> Package rpm-devel.i386 0:4.4.2.3-20.el5_5.1 set to be updated ---> Package rpm-libs.i386 0:4.4.2.3-20.el5_5.1 set to be updated ---> Package rpm-python.i386 0:4.4.2.3-20.el5_5.1 set to be updated ---> Package sudo.i386 0:1.7.2p1-8.el5_5 set to be updated ---> Package tzdata.i386 0:2010l-1.el5 set to be updated ---> Package xulrunner.i386 0:1.9.2.9-1.el5 set to be updated --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size ================================================================================ Updating: apr i386 1.2.7-11.el5_5.2 updates 123 k apr-devel i386 1.2.7-11.el5_5.2 updates 231 k dhclient i386 12:3.0.5-23.el5_5.2 updates 277 k expat i386 1.95.8-8.3.el5_5.3 updates 77 k expat-devel i386 1.95.8-8.3.el5_5.3 updates 132 k ksh i386 20100202-1.el5_5.1 updates 1.2 M nspr i386 4.8.6-1.el5_5 updates 120 k nspr-devel i386 4.8.6-1.el5_5 updates 112 k nss i386 3.12.7-2.el5.centos updates 1.1 M nss-devel i386 3.12.7-2.el5.centos updates 233 k nss-tools i386 3.12.7-2.el5.centos updates 1.2 M openssh i386 4.3p2-41.el5_5.1 updates 287 k openssh-askpass i386 4.3p2-41.el5_5.1 updates 40 k openssh-clients i386 4.3p2-41.el5_5.1 updates 449 k openssh-server i386 4.3p2-41.el5_5.1 updates 269 k poppler i386 0.5.4-4.4.el5_5.13 updates 3.0 M poppler-utils i386 0.5.4-4.4.el5_5.13 updates 72 k popt i386 1.10.2.3-20.el5_5.1 updates 74 k rpm i386 4.4.2.3-20.el5_5.1 updates 1.2 M rpm-build i386 4.4.2.3-20.el5_5.1 updates 302 k rpm-devel i386 4.4.2.3-20.el5_5.1 updates 1.2 M rpm-libs i386 4.4.2.3-20.el5_5.1 updates 928 k rpm-python i386 4.4.2.3-20.el5_5.1 updates 60 k sudo i386 1.7.2p1-8.el5_5 updates 230 k tzdata i386 2010l-1.el5 updates 796 k xulrunner i386 1.9.2.9-1.el5 updates 12 M
Transaction Summary ================================================================================ Install 0 Package(s) Upgrade 26 Package(s)
Total download size: 25 M Is this ok [y/N]: y Downloading Packages: (1/26): openssh-askpass-4.3p2-41.el5_5.1.i386.rpm | 40 kB 00:00 (2/26): rpm-python-4.4.2.3-20.el5_5.1.i386.rpm | 60 kB 00:00 (3/26): poppler-utils-0.5.4-4.4.el5_5.13.i386.rpm | 72 kB 00:00 (4/26): popt-1.10.2.3-20.el5_5.1.i386.rpm | 74 kB 00:00 (5/26): expat-1.95.8-8.3.el5_5.3.i386.rpm | 77 kB 00:01 (6/26): nspr-devel-4.8.6-1.el5_5.i386.rpm | 112 kB 00:01 (7/26): nspr-4.8.6-1.el5_5.i386.rpm | 120 kB 00:01 (8/26): apr-1.2.7-11.el5_5.2.i386.rpm | 123 kB 00:01 (9/26): expat-devel-1.95.8-8.3.el5_5.3.i386.rpm | 132 kB 00:01 (10/26): sudo-1.7.2p1-8.el5_5.i386.rpm | 230 kB 00:01 (11/26): apr-devel-1.2.7-11.el5_5.2.i386.rpm | 231 kB 00:01 (12/26): nss-devel-3.12.7-2.el5.centos.i386.rpm | 233 kB 00:01 (13/26): openssh-server-4.3p2-41.el5_5.1.i386.rpm | 269 kB 00:02 (14/26): dhclient-3.0.5-23.el5_5.2.i386.rpm | 277 kB 00:02 (15/26): openssh-4.3p2-41.el5_5.1.i386.rpm | 287 kB 00:02 (16/26): rpm-build-4.4.2.3-20.el5_5.1.i386.rpm | 302 kB 00:02 (17/26): openssh-clients-4.3p2-41.el5_5.1.i386.rpm | 449 kB 00:03 (18/26): tzdata-2010l-1.el5.i386.rpm | 796 kB 00:06 (19/26): rpm-libs-4.4.2.3-20.el5_5.1.i386.rpm | 928 kB 00:08 (20/26): nss-3.12.7-2.el5.centos.i386.rpm | 1.1 MB 00:09 (21/26): nss-tools-3.12.7-2.el5.centos.i386.rpm | 1.2 MB 00:10 (22/26): rpm-4.4.2.3-20.el5_5.1.i386.rpm | 1.2 MB 00:10 (23/26): ksh-20100202-1.el5_5.1.i386.rpm | 1.2 MB 00:11 (24/26): rpm-devel-4.4.2.3-20.el5_5.1.i386.rpm | 1.2 MB 00:12 (25/26): poppler-0.5.4-4.4.el5_5.13.i386.rpm | 3.0 MB 00:26 (26/26): xulrunner-1.9.2.9-1.el5.i386.rpm | 12 MB 01:43 -------------------------------------------------------------------------------- Total 114 kB/s | 25 MB 03:46 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : nspr 1/52 Updating : nss 2/52 Updating : popt 3/52 Updating : openssh 4/52 Updating : apr 5/52 Updating : poppler 6/52 Updating : expat 7/52 Updating : xulrunner 8/52 Updating : poppler-utils 9/52 Updating : openssh-clients 10/52 Updating : openssh-server 11/52 Updating : openssh-askpass 12/52 Updating : nss-tools 13/52 Updating : dhclient 14/52 Updating : ksh 15/52 Updating : sudo 16/52 Updating : nspr-devel 17/52 Updating : nss-devel 18/52 Updating : expat-devel 19/52 Updating : apr-devel 20/52 Updating : tzdata 21/52 Updating : rpm-libs 22/52 Updating : rpm 23/52 Updating : rpm-devel 24/52 Updating : rpm-python 25/52 Updating : rpm-build 26/52 Cleanup : expat 27/52 Cleanup : rpm-devel 28/52 Cleanup : rpm-python 29/52 Cleanup : nss-tools 30/52 Cleanup : poppler 31/52 Cleanup : openssh-clients 32/52 Cleanup : poppler-utils 33/52 Cleanup : openssh 34/52 Cleanup : rpm-libs 35/52 Cleanup : nspr 36/52 Cleanup : expat-devel 37/52 Cleanup : sudo 38/52 Cleanup : nspr-devel 39/52 Cleanup : rpm 40/52 Cleanup : xulrunner 41/52 Cleanup : popt 42/52 Cleanup : openssh-server 43/52 Cleanup : openssh-askpass 44/52 Cleanup : tzdata 45/52 Cleanup : nss-devel 46/52 Cleanup : apr-devel 47/52 Cleanup : ksh 48/52 Cleanup : apr 49/52 Cleanup : nss 50/52 Cleanup : dhclient 51/52 Cleanup : rpm-build 52/52
Updated: apr.i386 0:1.2.7-11.el5_5.2 apr-devel.i386 0:1.2.7-11.el5_5.2 dhclient.i386 12:3.0.5-23.el5_5.2 expat.i386 0:1.95.8-8.3.el5_5.3 expat-devel.i386 0:1.95.8-8.3.el5_5.3 ksh.i386 0:20100202-1.el5_5.1 nspr.i386 0:4.8.6-1.el5_5 nspr-devel.i386 0:4.8.6-1.el5_5 nss.i386 0:3.12.7-2.el5.centos nss-devel.i386 0:3.12.7-2.el5.centos nss-tools.i386 0:3.12.7-2.el5.centos openssh.i386 0:4.3p2-41.el5_5.1 openssh-askpass.i386 0:4.3p2-41.el5_5.1 openssh-clients.i386 0:4.3p2-41.el5_5.1 openssh-server.i386 0:4.3p2-41.el5_5.1 poppler.i386 0:0.5.4-4.4.el5_5.13 poppler-utils.i386 0:0.5.4-4.4.el5_5.13 popt.i386 0:1.10.2.3-20.el5_5.1 rpm.i386 0:4.4.2.3-20.el5_5.1 rpm-build.i386 0:4.4.2.3-20.el5_5.1 rpm-devel.i386 0:4.4.2.3-20.el5_5.1 rpm-libs.i386 0:4.4.2.3-20.el5_5.1 rpm-python.i386 0:4.4.2.3-20.el5_5.1 sudo.i386 0:1.7.2p1-8.el5_5 tzdata.i386 0:2010l-1.el5 xulrunner.i386 0:1.9.2.9-1.el5
Complete! [root@inet01 ~]# ssh inet06 Auto configuration failed 17203:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [root@inet01 ~]# service sshd restart Stopping sshd: [ OK ] Starting sshd: Auto configuration failed 17230:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED] [root@inet01 ~]# shutdown -r now
Broadcast message from root (pts/7) (Mon Sep 13 09:14:55 2010):
The system is going down for reboot NOW! [root@inet01 ~]# exit logout
[byrnejb@inet01 ~]$ Connection to inet01 closed by remote host. Connection to inet01 closed. [byrnejb@inet05 ~]$ ssh inet01 ssh: connect to host inet01 port 22: Connection refused
Looking at the install logs I see that openssl has not been touched for some time:
/var/log/yum.log.2:Jan 09 09:35:34 Updated: openssl - 0.9.8b-10.el5_2.1.i686 /var/log/yum.log.2:Jan 09 09:35:43 Updated: openssl-devel - 0.9.8b-10.el5_2.1.i386 /var/log/yum.log.2:Apr 03 19:14:55 Updated: openssl-0.9.8e-7.el5.i686 /var/log/yum.log.2:Apr 03 19:19:08 Updated: openssl-devel-0.9.8e-7.el5.i386 /var/log/yum.log.3:Jun 27 15:33:06 Updated: openssl.i686 0.9.8b-10.el5 /var/log/yum.log.3:Jun 27 15:42:29 Updated: openssl-devel.i386 0.9.8b-10.el5 /var/log/yum.log:Jan 21 14:11:30 Updated: openssl-0.9.8e-12.el5_4.1.i686 /var/log/yum.log:Jan 21 14:13:01 Updated: openssl-devel-0.9.8e-12.el5_4.1.i386 /var/log/yum.log:Mar 29 09:03:29 Updated: openssl-0.9.8e-12.el5_4.6.i686 /var/log/yum.log:Mar 29 09:03:40 Updated: openssl-devel-0.9.8e-12.el5_4.6.i386 /var/log/rpmpkgs.1:openssl-0.9.8e-12.el5_4.6.i686.rpm /var/log/rpmpkgs.1:openssl-devel-0.9.8e-12.el5_4.6.i386.rpm
On Mon, 2010-09-13 at 12:59 -0400, James B. Byrne wrote:
What is the procedure to remove the latest openssh packages and replace them with the previous ones?
I'm not sure you can do that directly with yum (though someone can correct me there) but you could do it by downloading the rpm packages that you want to downgrade, then do:
rpm -U --oldpackage filename.rpm
Frank Cox <theatre@...> writes:
On Mon, 2010-09-13 at 12:59 -0400, James B. Byrne wrote:
What is the procedure to remove the latest openssh packages and replace them with the previous ones?
I'm not sure you can do that directly with yum (though someone can correct me there) but you could do it by downloading the rpm packages that you want to downgrade, then do:
rpm -U --oldpackage filename.rpm
I had the same problem and downgraded with:
yum remove openssh-* yum install openssh openssh-server openssh-clients --disablerepo=updates
On 09/13/2010 04:01 PM, James B. Byrne wrote:
Starting sshd: Auto configuration failed 6486:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
...
This matter is somewhat urgent.
Totally! broken sshd could cause non-trivial issues for a lot of people.
However, I can confirm that:
* prov-kickstart-c5.5-i386 : sshd tests PASS * yum -y update : sshd tests PASS [1] * yum install latest updates : sshd tests PASS
[1]: the yum update runs against what is already on mirrors.centos.org, so would not have considered this latest set of new packages.
Also, the process is repeated for x86_64 and on both ipv4 and v6. However, machine and resource contrains mean that the tests are only ever run on Xen DomU's rather than real iron. The tests themselves are fairly simple:
- ensure iptables blocks port :22, test connection from localhost to localhost: - ensure iptables allows port :22 test connection from localhost to localhost: - repeat for real ipv4 IP, and IPv6 IP. - repat the whole set of tests for scp.
The connections are tested using 'ssh-keygen -t dsa -b 1024' so that tends to get tested as well ( ideally it would be cool to retain a set of keys from previous builds to make sure they still work, but I've not come up with any clean way of preserving test artifacts ).
So, I'm quite keen on finding out what is it that is causing the breakage for you. - KB
On Mon, 13 Sep 2010, Karanbir Singh wrote:
This matter is somewhat urgent.
Totally! broken sshd could cause non-trivial issues for a lot of people.
However, I can confirm that:
- prov-kickstart-c5.5-i386 : sshd tests PASS
- yum -y update : sshd tests PASS [1]
- yum install latest updates : sshd tests PASS
So, I'm quite keen on finding out what is it that is causing the breakage for you.
- KB
I am NOT rubbing this in, but I figured I'd let you know that I have 5 i386 machines, one x86_64...did a yum -y update...and all updated ssh and all related RPM's. And it's aall is working well here. I only slightly modified sshd_config in previous versions, and it didn't cause any problems with this update.
******************************************************************************* Gilbert Sebenste ******** (My opinions only!) ****** Staff Meteorologist, Northern Illinois University **** E-mail: sebenste@weather.admin.niu.edu *** web: http://weather.admin.niu.edu ** *******************************************************************************
On Mon, September 13, 2010 19:23, Gilbert Sebenste wrote:
On Mon, 13 Sep 2010, Karanbir Singh wrote:
This matter is somewhat urgent.
Totally! broken sshd could cause non-trivial issues for a lot of people.
However, I can confirm that:
- prov-kickstart-c5.5-i386 : sshd tests PASS
- yum -y update : sshd tests PASS [1]
- yum install latest updates : sshd tests PASS
So, I'm quite keen on finding out what is it that is causing the breakage for you.
- KB
I am NOT rubbing this in, but I figured I'd let you know that I have 5 i386 machines, one x86_64...did a yum -y update...and all updated ssh and all related RPM's. And it's aall is working well here. I only slightly modified sshd_config in previous versions, and it didn't cause any problems with this update.
In my original post I pointed out that this problem occurred on just one server and that all the updates applied cleanly on the rest. I do not have an explanation for why this particular server choked on the sshd update, but it did.
Interestingly, although the package on that server is now an older one
Name : openssh Relocations: (not relocatable) Version : 4.3p2 Vendor: CentOS Release : 41.el5 Build Date: Wed 31 Mar 2010 05:24:16 AM EDT Install Date: Mon 13 Sep 2010 02:28:35 PM EDT Build Host: builder10.centos.org Group : Applications/Internet Source RPM: openssh-4.3p2-41.el5.src.rpm Size : 744306
Vice
Name : openssh Relocations: (not relocatable) Version : 4.3p2 Vendor: CentOS Release : 41.el5_5.1 Build Date: Sun 12 Sep 2010 12:00:45 PM EDT Install Date: Mon 13 Sep 2010 08:50:29 AM EDT Build Host: builder17.centos.org Group : Applications/Internet Source RPM: openssh-4.3p2-41.el5_5.1.src.rpm Size : 744754
yum check-update does not report that there is a more recent openssh available. Odd.
yum check-update does not report that there is a more recent openssh available. Odd.
Clear your yum cache? Since you used yum to update the package in the first place, yum may have stored that "I already downloaded this" data somewhere.
FYI, I am not experienced in yum specifically (debian user here, learning RH based things).
Hope this helps,
Rob
Rob Del Vecchio wrote:
yum check-update does not report that there is a more recent openssh available. Odd.
Clear your yum cache? Since you used yum to update the package in the first place, yum may have stored that "I already downloaded this" data somewhere.
<snip> I agree - yum clean all, then try again.
mark
On Mon, 13 Sep 2010, Karanbir Singh wrote:
So, I'm quite keen on finding out what is it that is causing the breakage for you
Karanbir, in the middle of the thread from the archive:
James B. Byrne at Mon Sep 13 12:59:25 EDT 2010
Did that via webmin's command interface and nothing changed insofar as I can see. Same error obtained when starting sshd.
We ship no webmin of course, and it tinkers with the dependency chain for openssh ourside of the package system.
-- Russ herrold
From: James B. Byrne byrnejb@harte-lyne.ca
Starting sshd: Auto configuration failed 6486:error:0E065068:configuration file routines:STR_COPY:variable has no value:conf_def.c:629:line 207 [FAILED]
Maybe did you check line 207 of sshd_config...?
JD
-
Benjamin Franz -
chuck s -
Frank Cox -
Gilbert Sebenste -
James B. Byrne -
John Doe -
Joseph L. Casale -
Karanbir Singh -
m.roth＠5-cent.us -
Mário Barbosa -
R P Herrold -
Rob Del Vecchio