Hi All:
After upgrading to 4.4 my server(s) will not resolve using the servers listed in resolve.conf. This is not true of a server running a DNS Cache server. I can do a netstat -e and show a connection to the DNS servers but it will not resolve any domain names (please see below). Any thoughts?
[root@ftp ~]# dig yahoo.com
; <<>> DiG 9.2.4 <<>> yahoo.com ;; global options: printcmd ;; connection timed out; no servers could be reached
[root@ftp ~]# netstat -e Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode tcp 0 0 10.0.2.5:32806 66.81.0.251:domain ESTABLISHED root 17584
Thanks,
Ed
A little birdy told me that Ed Morrison said:
] After upgrading to 4.4 my server(s) will not resolve using the servers listed ] in resolve.conf. This is not true of a server running a DNS Cache server. I ] can do a netstat -e and show a connection to the DNS servers but it will not ] resolve any domain names (please see below). Any thoughts?
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place... as such, for servers to correctly reload their maps and other such tasks via "rndc" i needed to update that file to re-include the correct information for the key...
just figured it deserved a mention...
B. Karhan simon@pop.psu.edu PRI/SSRI Unix Administrator
Benjamin Karhan napsal(a):
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place... as such, for servers to correctly reload their maps and other such tasks via "rndc" i needed to update that file to re-include the correct information for the key...
just figured it deserved a mention...
Would you create ticket on http://bugs.centos.org/ please? Thanks, David
On Tue, 2006-09-05 at 09:50 +0200, David Hrbáč wrote:
Benjamin Karhan napsal(a):
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place...
<snip>
For others who may have missed it and not yet been victimized
http://lists.centos.org/pipermail/centos/2006-August/069073.html
warns you that your key may have been changed and you should check it.
Now I know why the list grows ever larger with repetition.
-- Bill
On Tue, 2006-09-05 at 08:21 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 09:50 +0200, David Hrbáč wrote:
Benjamin Karhan napsal(a):
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place...
<snip>
For others who may have missed it and not yet been victimized
http://lists.centos.org/pipermail/centos/2006-August/069073.html
warns you that your key may have been changed and you should check it.
Now I know why the list grows ever larger with repetition.
As long as I'm in my cranky mood
locate rpmsave|grep named /var/named/chroot/etc/named.conf.rpmsave
<snip>
On Tue, 2006-09-05 at 08:24 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 08:21 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 09:50 +0200, David Hrbáč wrote:
Benjamin Karhan napsal(a):
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place...
<snip>
For others who may have missed it and not yet been victimized
http://lists.centos.org/pipermail/centos/2006-August/069073.html
warns you that your key may have been changed and you should check it.
Now I know why the list grows ever larger with repetition.
As long as I'm in my cranky mood
locate rpmsave|grep named /var/named/chroot/etc/named.conf.rpmsave
<snip>
Bill ... I did ask in a previous post to someone else ... :
do you have caching-nameserver installed?
On Tue, 2006-09-05 at 07:40 -0500, Johnny Hughes wrote:
On Tue, 2006-09-05 at 08:24 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 08:21 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 09:50 +0200, David Hrbáč wrote:
Benjamin Karhan napsal(a):
<snip>
For others who may have missed it and not yet been victimized
http://lists.centos.org/pipermail/centos/2006-August/069073.html
warns you that your key may have been changed and you should check it.
Now I know why the list grows ever larger with repetition.
As long as I'm in my cranky mood
locate rpmsave|grep named /var/named/chroot/etc/named.conf.rpmsave
<snip>
Bill ... I did ask in a previous post to someone else ... :
do you have caching-nameserver installed?
# yum list installed *name* Installed Packages caching-nameserver.noarch 7.3-3 installed perl-XML-NamespaceSupport.noarch 1.08-6 installed
<snip sig stuff>
HTH -- Bill
On Tue, 2006-09-05 at 08:46 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 07:40 -0500, Johnny Hughes wrote:
On Tue, 2006-09-05 at 08:24 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 08:21 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 09:50 +0200, David Hrbáč wrote:
Benjamin Karhan napsal(a):
<snip>
For others who may have missed it and not yet been victimized
http://lists.centos.org/pipermail/centos/2006-August/069073.html
warns you that your key may have been changed and you should check it.
Now I know why the list grows ever larger with repetition.
As long as I'm in my cranky mood
locate rpmsave|grep named /var/named/chroot/etc/named.conf.rpmsave
<snip>
Bill ... I did ask in a previous post to someone else ... :
do you have caching-nameserver installed?
# yum list installed *name* Installed Packages caching-nameserver.noarch 7.3-3 installed perl-XML-NamespaceSupport.noarch 1.08-6 installed
<snip sig stuff>
caching-nameserver SHOULD_NOT_BE_ installed on a nameserver that is used to control zones as it _WILL_REPLACE config files _EVERY_TIME_ ... _BY_DESIGN_. Sorry for the caps, but this continuously comes up ... hoping to draw attention to it for future reference :)
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=145244
Thanks, Johnny Hughes
On Tue, 2006-09-05 at 07:54 -0500, Johnny Hughes wrote:
On Tue, 2006-09-05 at 08:46 -0400, William L. Maltby wrote:
On Tue, 2006-09-05 at 07:40 -0500, Johnny Hughes wrote:
<snip>
do you have caching-nameserver installed?
# yum list installed *name* Installed Packages caching-nameserver.noarch 7.3-3 installed perl-XML-NamespaceSupport.noarch 1.08-6 installed
<snip sig stuff>
caching-nameserver SHOULD_NOT_BE_ installed on a nameserver that is used to control zones as it _WILL_REPLACE config files _EVERY_TIME_ ... _BY_DESIGN_. Sorry for the caps, but this continuously comes up ... hoping to draw attention to it for future reference :)
And since folks don't read lists well, here is a second chance for them to see it and for me to say "Not Guilty"! I've had no problems in this area. :-))
On Tue, 2006-09-05 at 00:45 -0400, Benjamin Karhan wrote:
A little birdy told me that Ed Morrison said:
] After upgrading to 4.4 my server(s) will not resolve using the servers listed ] in resolve.conf. This is not true of a server running a DNS Cache server. I ] can do a netstat -e and show a connection to the DNS servers but it will not ] resolve any domain names (please see below). Any thoughts?
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place... as such, for servers to correctly reload their maps and other such tasks via "rndc" i needed to update that file to re-include the correct information for the key...
just figured it deserved a mention...
B. Karhan simon@pop.psu.edu PRI/SSRI Unix Administrator
Benjiman,
%verify(not size,not md5) %config(noreplace) %attr(0640,root,named) /etc/rndc.conf %verify(not size,not md5) %config(noreplace) %attr(0640,root,named) /etc/rndc.key
(those lines in the bind SPEC file should have only replaced files that are not changed for rndc.conf and rndc.key)
Do you maybe have the caching-nameserver RPM installed on an active DNS box that is used for zone control?
Thanks, Johnny Hughes
A little birdy told me that Johnny Hughes said:
] > my DNS servers all survived the upgrade fine... ] > with one caveat... ] > the file "/etc/rndc.conf" got updated without any ".rpmnew" ] > preservation taking place... ] > as such, for servers to correctly reload their maps and other ] > such tasks via "rndc" i needed to update that file to re-include the ] > correct information for the key... ] > ] > just figured it deserved a mention... ] > ] > B. Karhan ] > simon@pop.psu.edu ] > PRI/SSRI Unix Administrator ] ] Benjiman, ] ] %verify(not size,not md5) %config(noreplace) %attr(0640,root,named) /etc/rndc.conf ] %verify(not size,not md5) %config(noreplace) %attr(0640,root,named) /etc/rndc.key ] ] (those lines in the bind SPEC file should have only replaced files that ] are not changed for rndc.conf and rndc.key) ] ] Do you maybe have the caching-nameserver RPM installed on an active DNS ] box that is used for zone control?
nope... and my rndc.key file was preserved... it's altogether possible that the default rndc.conf worked fine before and used an "include rndc.key;" statement... but the new one definately did something else... (had a new key spec declared inconsistent with any existing rndc.key)
B. Karhan simon@pop.psu.edu PRI/SSRI Unix Administrator
Benjamin Karhan spake the following on 9/4/2006 9:45 PM:
A little birdy told me that Ed Morrison said:
] After upgrading to 4.4 my server(s) will not resolve using the servers listed ] in resolve.conf. This is not true of a server running a DNS Cache server. I ] can do a netstat -e and show a connection to the DNS servers but it will not ] resolve any domain names (please see below). Any thoughts?
my DNS servers all survived the upgrade fine... with one caveat... the file "/etc/rndc.conf" got updated without any ".rpmnew" preservation taking place... as such, for servers to correctly reload their maps and other such tasks via "rndc" i needed to update that file to re-include the correct information for the key...
just figured it deserved a mention...
B. Karhan simon@pop.psu.edu PRI/SSRI Unix Administrator
Strange, I had no such problem. I seem to have an rndc.conf.rpmnew on both of my name servers, and the original is completely safe. I wonder what is mis-firing on the broken upgrades?
Ed Morrison wrote:
Hi All:
After upgrading to 4.4 my server(s) will not resolve using the servers listed in resolve.conf. This is not true of a server running a DNS Cache server. I can do a netstat -e and show a connection to the DNS servers but it will not resolve any domain names (please see below). Any thoughts?
[root@ftp ~]# dig yahoo.com
; <<>> DiG 9.2.4 <<>> yahoo.com ;; global options: printcmd ;; connection timed out; no servers could be reached
Please show your /etc/resolv.conf (and rename it to that, if you really have a /etc/resolve.conf).
Ralph
Ralph Angenendt wrote:
Ed Morrison wrote:
Hi All:
After upgrading to 4.4 my server(s) will not resolve using the servers listed in resolve.conf. This is not true of a server running a DNS Cache server. I can do a netstat -e and show a connection to the DNS servers but it will not resolve any domain names (please see below). Any thoughts?
[root@ftp ~]# dig yahoo.com
; <<>> DiG 9.2.4 <<>> yahoo.com ;; global options: printcmd ;; connection timed out; no servers could be reached
Please show your /etc/resolv.conf (and rename it to that, if you really have a /etc/resolve.conf).
Ralph
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
My /etc/resolv.conf:
; generated by /sbin/dhclient-script search csds.local nameserver 192.168.1.4 nameserver 65.106.1.196 nameserver 65.106.7.196
Ed Morrison wrote:
Ralph Angenendt wrote:
Please show your /etc/resolv.conf (and rename it to that, if you really have a /etc/resolve.conf).
My /etc/resolv.conf:
; generated by /sbin/dhclient-script search csds.local nameserver 192.168.1.4 nameserver 65.106.1.196 nameserver 65.106.7.196
And those servers are reachable by you? Can you ping them?
Ralph
Ralph Angenendt wrote:
Ed Morrison wrote:
Ralph Angenendt wrote:
Please show your /etc/resolv.conf (and rename it to that, if you really have a /etc/resolve.conf).
My /etc/resolv.conf:
; generated by /sbin/dhclient-script search csds.local nameserver 192.168.1.4 nameserver 65.106.1.196 nameserver 65.106.7.196
And those servers are reachable by you? Can you ping them?
Ralph
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Yes they are available:
[root@ftp ~]# ping 65.106.1.196 PING 65.106.1.196 (65.106.1.196) 56(84) bytes of data. 64 bytes from 65.106.1.196: icmp_seq=0 ttl=248 time=7.38 ms 64 bytes from 65.106.1.196: icmp_seq=1 ttl=248 time=6.70 ms 64 bytes from 65.106.1.196: icmp_seq=2 ttl=248 time=7.24 ms 64 bytes from 65.106.1.196: icmp_seq=3 ttl=248 time=6.79 ms
--- 65.106.1.196 ping statistics --- 5 packets transmitted, 4 received, 20% packet loss, time 4006ms rtt min/avg/max/mdev = 6.702/7.033/7.385/0.289 ms, pipe 2 [root@ftp ~]# ping 65.106.7.196 PING 65.106.7.196 (65.106.7.196) 56(84) bytes of data. 64 bytes from 65.106.7.196: icmp_seq=0 ttl=58 time=6.72 ms 64 bytes from 65.106.7.196: icmp_seq=1 ttl=58 time=6.80 ms 64 bytes from 65.106.7.196: icmp_seq=2 ttl=58 time=6.19 ms
--- 65.106.7.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 6.194/6.575/6.808/0.287 ms, pipe 2 [root@ftp ~]#
Ed Morrison wrote:
nameserver 192.168.1.4 nameserver 65.106.1.196 nameserver 65.106.7.196
And those servers are reachable by you? Can you ping them?
Yes they are available:
[root@ftp ~]# ping 65.106.1.196
--- 65.106.1.196 ping statistics --- 5 packets transmitted, 4 received, 20% packet loss, time 4006ms [root@ftp ~]# ping 65.106.7.196 PING 65.106.7.196 (65.106.7.196) 56(84) bytes of data. --- 65.106.7.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
What about 192.168.1.4? Is there a name server running on that? You know that the latter ones are only taken into resolving if the first one isn't reachable?
Ralph
PS: Please trim your mails.
What about 192.168.1.4? Is there a name server running on that? You know that the latter ones are only taken into resolving if the first one isn't reachable?
Ralph
PS: Please trim your mails.
Yes it is available also:
[root@ftp ~]# ping 192.168.1.4 PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_seq=0 ttl=125 time=8.71 ms 64 bytes from 192.168.1.4: icmp_seq=1 ttl=125 time=8.59 ms 64 bytes from 192.168.1.4: icmp_seq=2 ttl=125 time=8.50 ms 64 bytes from 192.168.1.4: icmp_seq=3 ttl=125 time=8.54 ms
--- 192.168.1.4 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 8.506/8.590/8.717/0.103 ms, pipe 2 [root@ftp ~]#
Ed Morrison wrote:
What about 192.168.1.4? Is there a name server running on that? You know that the latter ones are only taken into resolving if the first one isn't reachable?
Yes it is available also:
[root@ftp ~]# ping 192.168.1.4
What happens if you issue
host www.centos.org 192.168.1.4?
Ralph
Ed Morrison wrote:
What happens if you issue
host www.centos.org 192.168.1.4?
[root@ftp ~]# host www.centos.org 192.168.1.4 ;; connection timed out; no servers could be reached [root@ftp ~]#
Then find out why 192.168.1.4 doesn't answer you.
Or throw that IP address out of your resolv.conf.
Ralph
Then find out why 192.168.1.4 doesn't answer you.
Or throw that IP address out of your resolv.conf.
Ralph
That is kind of the point. It doesn't matter which dns servers I query with my 4.4 boxes they will not resolve but these same dns servers resolve queries for my other boxes. The problem only occurs with the 4.4 upgraded boxes and only since they were upgraded to 4.4:
[root@ftp ~]# host www.centos.org 192.168.1.4 [root@ftp ~]# host www.centos.org 65.106.1.196 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 65.106.7.196 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 66.81.0.251 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 66.81.0.252 ;; connection timed out; no servers could be reached [root@ftp ~]#
Ed Morrison wrote:
Then find out why 192.168.1.4 doesn't answer you. Or throw that IP address out of your resolv.conf.
Ralph
That is kind of the point. It doesn't matter which dns servers I query with my 4.4 boxes they will not resolve but these same dns servers resolve queries for my other boxes. The problem only occurs with the 4.4 upgraded boxes and only since they were upgraded to 4.4:
[root@ftp ~]# host www.centos.org 192.168.1.4 [root@ftp ~]# host www.centos.org 65.106.1.196 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 65.106.7.196 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 66.81.0.251 ;; connection timed out; no servers could be reached [root@ftp ~]# host www.centos.org 66.81.0.252 ;; connection timed out; no servers could be reached [root@ftp ~]#
See my most recent reply. Turn off the firewall on the affected box and see if DNS queries "wake up". If it does, then fix your firewall config. :)
Cheers,
Ed Morrison wrote:
What happens if you issue host www.centos.org 192.168.1.4?
Ralph
[root@ftp ~]# host www.centos.org 192.168.1.4 ;; connection timed out; no servers could be reached [root@ftp ~]#
That probably means one of the following:
1. port 53 is being filtered on the ftp host 2. port 53 is being filtered on the 192.168.1.4 host 3. there is no name server running on 192.168.1.4
Try this and see if name service begins working.
/etc/rc.d/init.d/iptables stop
If that fixes things, then you need to visit your iptables config and allow DNS queries in/out.
Hope that helps.
Cheers,
That probably means one of the following:
- port 53 is being filtered on the ftp host
- port 53 is being filtered on the 192.168.1.4 host
- there is no name server running on 192.168.1.4
Try this and see if name service begins working.
/etc/rc.d/init.d/iptables stop
If that fixes things, then you need to visit your iptables config and allow DNS queries in/out.
Hope that helps.
Hi Chris:
Unfortunately, I have already stopped the firewall and DNS is running on 192.168.1.4. I took 192.168.1.4 out of the equation and used all the DNS servers that I have access to.... same result.
[root@ftp ~]# iptables -L -v -n Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination [root@ftp ~]#
On Tue, 2006-09-05 at 06:30 -0700, Ed Morrison wrote:
That probably means one of the following:
- port 53 is being filtered on the ftp host
- port 53 is being filtered on the 192.168.1.4 host
- there is no name server running on 192.168.1.4
Try this and see if name service begins working.
/etc/rc.d/init.d/iptables stop
If that fixes things, then you need to visit your iptables config and allow DNS queries in/out.
Hope that helps.
Hi Chris:
Unfortunately, I have already stopped the firewall and DNS is running on 192.168.1.4. I took 192.168.1.4 out of the equation and used all the DNS servers that I have access to.... same result.
[root@ftp ~]# iptables -L -v -n Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination [root@ftp ~]#
This is quite strange .... I can assure you that centos-4.4 can connect and do name lookups.
Do you not have bind client software installed on this machine?
Is there some kind of proxy server where port 53 udp traffic might somehow be blocked?
Johnny Hughes wrote:
This is quite strange .... I can assure you that centos-4.4 can connect and do name lookups.
Do you not have bind client software installed on this machine?
Is there some kind of proxy server where port 53 udp traffic might somehow be blocked?
I have update 2 systems with named.. one lan and one Wan and other then the rndc change that was mentioned all is working just fine on mine.
JFYI
Brian
-> > -> My /etc/resolv.conf: -> -> ; generated by /sbin/dhclient-script -> search csds.local -> nameserver 192.168.1.4 -> nameserver 65.106.1.196 -> nameserver 65.106.7.196 ->
I have read the future parts of this thread and came back to here.
I was doing some test loads and what I noticed was that when I enabled dhcp on an interface that it would look to the router ip as the primary dns and then put in a secondary and a tertiary.
So what and then the bogus search domain
So what I ended up doing for this was making a /etc/resolv.bak with just the real primary and secondary as the router doesn't have bind running of course and getting rid of the search and putting in something in /etc/rc.local like
cp -af /etc/resolv.bak /etc/resolv.conf
and the problem went away.
Remember I was just testing 4.4 upgrades and how well the mirrors were handling the stress etc etc before I blow up some production servers.
It appears to me in this case you should focus on dhcp issues or some sort...
i.e. look at this /etc/resolv.conf file and it tells you the answer as far as I am concerned. It is *generated* by the dhcp
also, is there more than one Ethernet interface on this machine?
It would have been easier to see if the resolver was working on the 192.168.1.4 box before you took it out of the loop...
I could be wrong about dhcp issues of course yet at least I tried
;->
- rh
-- Robert - Abba Communications Computer & Internet Services (509) 624-7159 - www.abbacomm.net
As it turns out DNS stopped working for my entire network. The problem manifested itself with the 4.4 servers first. The problem was a fat finger error by my ISP. They found the issue and I am working again. Sorry for the false alarm.
Ed
On Tue, 2006-09-05 at 10:09 -0700, Ed Morrison wrote:
As it turns out DNS stopped working for my entire network. The problem manifested itself with the 4.4 servers first. The problem was a fat finger error by my ISP. They found the issue and I am working again. Sorry for the false alarm.
One diagnostic that might have helped you catch this is:
dig @server_ip some.domain.name
testing each server_ip from your /etc/resolv.conf individually.
-> -> One diagnostic that might have helped you catch this is: -> -> dig @server_ip some.domain.name -> -> testing each server_ip from your /etc/resolv.conf individually. -> -> -- -> Les Mikesell
Yup, that and although I don't know if it is as effective or helpful as it used to be for me, you could consider and do this from bash shell too...
telnet nameserveripaddress 53
- rh
-- Robert - Abba Communications Computer & Internet Services (509) 624-7159 - www.abbacomm.net
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Sep 05, 2006 at 01:06:27PM -0700, Email Lists wrote:
-> One diagnostic that might have helped you catch this is: -> -> dig @server_ip some.domain.name -> -> testing each server_ip from your /etc/resolv.conf individually.
Yup, that and although I don't know if it is as effective or helpful as it used to be for me, you could consider and do this from bash shell too...
telnet nameserveripaddress 53
No go, since most DNS servers will block TCP based queries for everything except their secondaries.
Standard DNS queries use UDP.
[]s
- -- Rodrigo Barbosa "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
-> > -> > telnet nameserveripaddress 53 -> -> No go, since most DNS servers will block TCP based queries for -> everything except their secondaries. -> -> Standard DNS queries use UDP. ->
I hear you...
Yeah I usually just use it to see if something answers so to speak, or to see if a firewall or something knocks it down and refuses to allow "connect" in general.
It helped in the OLD OLD days when I started in the early to mid 1990's more so than now except telnetting to various tcp ports and stepping through tests etc
- rh
-- Robert - Abba Communications Computer & Internet Services (509) 624-7159 - www.abbacomm.net
-
Benjamin Karhan -
BrianSr -
chrism＠imntv.com -
David Hrbáč -
Ed Morrison -
Email Lists -
Johnny Hughes -
Les Mikesell -
Ralph Angenendt -
Rodrigo Barbosa -
Scott Silva -
William L. Maltby