[CentOS] ssh x11 forwarding problem

Sat Feb 18 10:47:45 UTC 2006
Ron Yorston <rmy at tigress.co.uk>

Tony Schreiner <schreian at bc.edu> wrote:
>I'm trying to run an application (rasmol -  molecule viewing program)  
>which when using the the default setup for x11 forwarding causes the  
>following error:
>X Error of failed request:  BadAccess (attempt to access private  
>resource denied)
>   Major opcode of failed request:  132 (MIT-SHM)
>   Minor opcode of failed request:  1 (X_ShmAttach)
>   Serial number of failed request:  230
>   Current serial number in output stream:  231
>It used to be possible to overcome this by setting
>X11UseLocalhost  no
>in /etc/ssh/sshd_config
>This worked in CentOS 3 but does not in CentOS 4 (or Fedora 4 for  
>that matter), and I'm guessing it is a XFree86 vs. Xorg issue.
>Now, when using
>X11UseLocalhost no
>no X applications work
>ssh myserver
># echo $DISPLAY
># xdpyinfo
>xdpyinfo:  unable to open display "myserver.fqdn:10.0".

It appears that in FC4 setting 'X11UseLocalhost no' causes sshd to listen
for X connections only on the IPV6 port:

$ netstat -ant | grep 6010
tcp        0      0 :::6010                     :::*                        LISTEN

I'd say this was a bug, and indeed there is such a report in the openssh
bugzilla, but it's supposed to have been fixed years ago.  I've tried
forcing sshd to use IPV4 only (with the -4 flag) but that just gets me
back into the land of X authentication failures.

The problem with rasmol is that it tries to use the X shared memory extension
if it thinks it's running on the same host as the X server.  To rasmol an
ssh-forwarded X connection looks like a local connection, so it tries to
use shared memory and fails.  The 'X11UseLocalhost no' trick works because
it makes the forwarded connection look like a connection to a remote machine
so the application falls back to using an alternative.

Looking at the source for rasmol I see that it uses this test to determine
if it can use MIT-SHM:

   ptr = DisplayString(dpy);
   if( !ptr || (*ptr==':') || !strncmp(ptr,"localhost:",10) ||
		   !strncmp(ptr,"unix:",5) || !strncmp(ptr,"local:",6) )

So even with 'X11UseLocalhost yes' it might be possible to fool it into not
using MIT-SHM by setting your DISPLAY to '' (or whatever).