I'm having a bit of a problem with what I can "see" over nfs. I have two machines that nfs their root to each other and it seems to work. However, I now found out that some of the root directories that are listed are "fake" ones. A comparison of the root on both machines shows that some directories are the same and I can access those files on the other machine. However, there are some directories that are different.
That is true for all the "special" directories like dev, proc, sys (which makes some sense), but also for directories that are mount points on the other machine. Does nfs not give me access to those mount points? Do I have to create a second nfs mount to that machine which mounts only that mount point? Or do I need some parameter to mount these mount points under the normal mount?
What's weird is that if I create a file in these "fake" directories I can do that and it's listed like it actually were created on the other machine. But it gets created on *this* machine. When I then unmount, it's gone (like the whole nfs mount) and when I remount it's back. Where is it actually getting created and how can I remove it? Is there some nfs cache that holds it?
Kai
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, June 26, 2008 10:24 AM To: centos@centos.org Subject: [CentOS] nfs cannot see mount points on other machine
I'm having a bit of a problem with what I can "see" over nfs. I have two machines that nfs their root to each other and it seems to work. However, I now found out that some of the root directories that are listed are "fake" ones. A comparison of the root on both machines shows that some directories are the same and I can access those files on the other machine. However, there are some directories that are different.
Was having a similar problem this morning, just listed them too as mounts on top of the other mounts.
That is true for all the "special" directories like dev, proc, sys (which makes some sense), but also for directories that are mount points on the other machine. Does nfs not give me access to those mount points? Do I have to create a second nfs mount to that machine which mounts only that mount point? Or do I need some parameter to mount these mount points under the normal mount?
From man exports
nohide
This option is based on the option of the same name provided in IRIX NFS. Normally, if a server exports two filesystems one of which is mounted on the other, then the client will have to mount both filesystems explicitly to get access to them. If it just mounts the parent, it will see an empty directory at the place where the other filesystem is mounted. That filesystem is "hidden".
Setting the nohide option on a filesystem causes it not to be hidden, and an appropriately authorised client will be able to move from the parent to that filesystem without noticing the change.
However, some NFS clients do not cope well with this situation as, for instance, it is then possible for two files in the one apparent filesystem to have the same inode number.
The nohide option is currently only effective on single host exports. It does not work reliably with netgroup, subnet, or wildcard exports.
This option can be very useful in some situations, but it should be used with due care, and only after confirming that the client system copes with the situation effectively.
The option can be explicitly disabled with hide.
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Jason Pyeron wrote on Thu, 26 Jun 2008 10:42:40 -0400:
This option is based on the option of the same name provided in IRIX NFS. Normally, if a server exports two filesystems one of which is mounted on the other, then the client will have to mount both filesystems explicitly to get access to them. If it just mounts the parent, it will see an empty directory at the place where the other filesystem is mounted. That filesystem is "hidden".
Thanks for the hint about exports, I think I would have never tried man on that. I think it's not the problem I see, but after some more reading and experiencing that the proposed solution fails I think I know the reason.
The above paragraph only applies to the cross-mounted filesystem I think, e.g. if I mount under /nfs/hostname then this will be "hidden". I tried the "nohide" solution and it didn't make a difference for my problem. There's also an interesting option crossmnt explained right after nohide, but this is not going to work either I think (and actually I don't really understand it, it's confusingly explained).
On rereading that man article it becomes obvious that you mount "filesystems". So, if you mount / on the other machine you get only what the other machine has under that mount point. If I have a mount point /home on the other machine that won't be available under the / export. I have to export and mount /nfs/hostname/home if I want to get that one as well. I think. I haven't tried yet.
What doesn't fit in this, though, is the fact that the other mount points are displayed as directories and I can work on them (just in some cache it seems). This is very confusing as you never know if you are working on the real thing or not. I would have thought that the coders would have taken care of that. So, maybe my theory is not correct. But the nohide option doesn't fix the problem either.
Kai
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, June 26, 2008 11:35 AM To: centos@centos.org Subject: Re: [CentOS] nfs cannot see mount points on other machine
Jason Pyeron wrote on Thu, 26 Jun 2008 10:42:40 -0400:
This option is based on the option of the same name
provided in IRIX NFS.
Normally, if a server exports two filesystems one of which
is mounted
on the other, then the client will have to mount both
filesystems explicitly to get
This is what I did.
access to them. If it just mounts the parent, it will
see an empty
directory at the place where the other filesystem is mounted. That filesystem is "hidden".
What doesn't fit in this, though, is the fact that the other mount points are displayed as directories and I can work on them (just in some cache it seems). This is very confusing as you never know if you are working on the real thing or not. I would have thought that the coders would have taken care of that. So, maybe my theory is not correct. But the nohide option doesn't fix the problem either.
I never tried the nohide as it said it could be dangerous to some clients as was not compatible with wild card exports which we were using.
So we mount /, /mnt/disk1, /mnt/disk2, ....
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Jason Pyeron wrote on Thu, 26 Jun 2008 13:16:51 -0400:
This is what I did.
Ok, I finally got it right. - you have to *export* all filesystems that should be usable (not just /) - you have to "nohide" each single one of them
then you can mount hostname:/ and get all the other remote mount points automatically as well.
Kai
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, June 26, 2008 1:53 PM To: centos@centos.org Subject: Re: [CentOS] nfs cannot see mount points on other machine
Jason Pyeron wrote on Thu, 26 Jun 2008 13:16:51 -0400:
This is what I did.
Ok, I finally got it right.
- you have to *export* all filesystems that should be usable
(not just /)
- you have to "nohide" each single one of them
then you can mount hostname:/ and get all the other remote mount points automatically as well.
Any ideas what the "dangerous" inode confusion is about? Is it relevant today?
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Jason Pyeron wrote on Thu, 26 Jun 2008 18:03:49 -0400:
Any ideas what the "dangerous" inode confusion is about? Is it relevant today?
I have no idea. I think the proposed problem is that the client doesn't know that it's traversing filesystems, so, the same inode number on filesystem / and /b is each time the inode number on /. I have no idea if this can actually happen or how this is worked out or if it is still a problem. But I think the way it is now by default is not a good solution. As I wrote you can work and copy to these faked folders and they disappear and reappear with mounting although they are actually somewhere on the local filesystem. It looks like the mounting creates a local directory listing that is only available when it's mounted. The way it works without nohide is really able to trick you to think you are writing to the remote side, but you aren't. I think this is dangerous. They should indeed have *hidden* those folders instead of faking them. The talk about "hidden" is wrong in my eyes. They do not hide they pretend things that are not there.
From that perspective I think using nohide is the better option. But I don't
know if that "inode problem" could really hit or not. I haven't seen it so far.
I think what would be a bad idea is to cross-mount the nfs shares themselves, but this is prevented unless you explicitely export them.
Kai
Kai Schaetzl wrote:
Jason Pyeron wrote on Thu, 26 Jun 2008 18:03:49 -0400:
Any ideas what the "dangerous" inode confusion is about? Is it relevant today?
I have no idea. I think the proposed problem is that the client doesn't know that it's traversing filesystems, so, the same inode number on filesystem / and /b is each time the inode number on /. I have no idea if this can actually happen or how this is worked out or if it is still a problem. But I think the way it is now by default is not a good solution. As I wrote you can work and copy to these faked folders and they disappear and reappear with mounting although they are actually somewhere on the local filesystem. It looks like the mounting creates a local directory listing that is only available when it's mounted. The way it works without nohide is really able to trick you to think you are writing to the remote side, but you aren't. I think this is dangerous. They should indeed have *hidden* those folders instead of faking them. The talk about "hidden" is wrong in my eyes. They do not hide they pretend things that are not there.
From that perspective I think using nohide is the better option. But I don't
know if that "inode problem" could really hit or not. I haven't seen it so far.
I think what would be a bad idea is to cross-mount the nfs shares themselves, but this is prevented unless you explicitely export them.
I don't understand your talk about "fake" directories. They are not fake, they truly exist in the filesystem. It's just that another filesystem is being overlaid in that dir. Some fs like dev, proc, sys exist only in memory and are created on the fly at boottime, others are just disk partitions. Nothing mysterious there.
That NFS only exports a single partition at a time is probably due to the duplicate inode problem - maybe other stuff - I don't know. At any rate, just export the additional fs and mount it where you like. Again, nothing mysterious and it has been done that way since NFS was invented.
Want to prove that your "fake" dirs are not fake? Boot the rescue CD and don't have it automatically find your Linux partitions. Mount what ever's your / partition on /tmp/sysimage. There's /tmp/sysimage/dev and it has an inode number. Let's mkdir /tmp/sysimage/dev/testdirontherootpartition. That has an inode number too. Boot back to the OS - /dev/testdirontherootpartition doesn't exist - it's been overlaid by the udev system. Boot back to the resue CD - /tmp/sysimage/dev/testdirontherootpartition is there - same inode as it was before.
Same goes for your /home dir when it's a separate partition - if /home is not mounted, anything you write there will be written to the root partition and will be hidden when /home is mounted.
Toby Bluhm wrote on Fri, 27 Jun 2008 08:54:56 -0400:
I don't understand your talk about "fake" directories. They are not fake, they truly exist in the filesystem.
They are nevertheless fake. Consider the following: - system A has mount points / and /home - system B nfs mounts / on A without nohide at /nfs/A
Result is that you see *all* directories of A on B, including /home. There is no way to know that it doesn't exist on A, unless you compare the directories on both machines. There is no indication that you are not writing to A:/home when you write to /nfs/A/home. That is what I call "fake". It's definitely not "hidden". "hidden" comes from "hiding" = you don't see it. I consider this behavior *very* misleading.
That NFS only exports a single partition at a time is probably due to the duplicate inode problem - maybe other stuff - I don't know. At any rate, just export the additional fs and mount it where you like. Again, nothing mysterious and it has been done that way since NFS was invented.
That may be so. I'm quite happy with this behavior as long as nfs doesn't pretend that something is there that isn't.
I was asking "where" that faked directory actually exists as it is gone when I unmount. If I understand your explanation correctly if I write to /nfs/A/home I'm actually writing to A, but not to the /home filesystem (as I think) but to a home directory on the / filesystem. Is that correct? That makes clear why it is gone when I unmount. Further, if I unmount /home on A I should still get /home when I list / on A. Just now that "faked" home on /. Correct?
I understand that this directory *does* exist on A (just not where one would think) *after* nfs mounting. However, from the standpoint of machine B it is a fake. It is artificially being created because an ls on A shows it. The correct behavior would be to *not list* any other mount points in the nfs mount.
Kai
Kai Schaetzl wrote:
Toby Bluhm wrote on Fri, 27 Jun 2008 08:54:56 -0400:
I don't understand your talk about "fake" directories. They are not fake, they truly exist in the filesystem.
They are nevertheless fake. Consider the following:
- system A has mount points / and /home
- system B nfs mounts / on A without nohide at /nfs/A
Result is that you see *all* directories of A on B, including /home. There is no way to know that it doesn't exist on A, unless you compare the directories on both machines. There is no indication that you are not writing to A:/home when you write to /nfs/A/home. That is what I call "fake". It's definitely not "hidden". "hidden" comes from "hiding" = you don't see it. I consider this behavior *very* misleading.
That NFS only exports a single partition at a time is probably due to the duplicate inode problem - maybe other stuff - I don't know. At any rate, just export the additional fs and mount it where you like. Again, nothing mysterious and it has been done that way since NFS was invented.
That may be so. I'm quite happy with this behavior as long as nfs doesn't pretend that something is there that isn't.
I was asking "where" that faked directory actually exists as it is gone when I unmount. If I understand your explanation correctly if I write to /nfs/A/home I'm actually writing to A, but not to the /home filesystem (as I think) but to a home directory on the / filesystem. Is that correct?
Yes. Let's use some examples. If A:/ is /dev/sda1 and A:/home is /dev/sda2, then in your above situation, writes to B:/nfs/A/home will be written to /dev/sda1 on A and _not_ /dev/sda2 on A. When A exports /, it's actually only exporting the filesystem on /dev/sda1.
That makes clear why it is gone when I unmount. Further, if I unmount /home on A I should still get /home when I list / on A.
Yes - you will get the /home that's on /dev/sda1
Just now that "faked" home on /. Correct?
I understand that this directory *does* exist on A (just not where one would think) *after* nfs mounting. However, from the standpoint of machine B it is a fake. It is artificially being created because an ls on A shows it. The correct behavior would be to *not list* any other mount points in the nfs mount.
Again, it's not fake, it's not artificial. It's truly there on /dev/sda1, the storage device. If you umount /home, rmdir /home, you can't mount /dev/sda2 on /home any more can you? If you mkdir /home2, you can put stuff in there until you run out of space on /dev/sda1. If you mount /dev/sda2 on /home2, the stuff you wrote to /home2 will still be on /dev/sda1 and will be hidden by the overlay of /dev/sda2. If you umount A:/ from B:/nfs/A and then write to B:/nfs/A - because B:/nfs/A will still exist as a real directory - it will be stored in /dev/sda1 on B. Whether you mount NFS exports or local disk partitions, it works the same way.
By your analogy, umounting /dev/sda2 /home should make /home disappear from / because it's not mounted. Or mounting /dev/sda1 as / should just make /dev/sda2 fall into /home without any entry in fstab.
Toby Bluhm wrote on Fri, 27 Jun 2008 12:00:45 -0400:
Again, it's not fake, it's not artificial.
It's both, really. artificial = it doesn't exist before the first nfs mount fake = from the remote machine it's an imitation of the real /home filesystem on the other machine. It looks like /home, it smells like /home, it tastes like /home, but if you eat it you get no calories/joule.
The point is: if it is not there (before the first nfs mount!) it shouldn't be there! Think of it as stepping into a broken elevator (door open, elevator not on the same level). The door opening is usually prohibited in such a case by some internal security measures. However, if it were to open nevertheless there is a chance that you don't recognize fast enough and step into it and splash ... That's exactly how nfs behaves, it's an elevator without that safety net.
Kai
That's exactly how nfs behaves, it's an elevator without that safety net.
You're right. NFS is very dangerous and it may trick you. It should only be used if you know how it works and what you are doing.
Toby Bluhm wrote on Fri, 27 Jun 2008 12:46:43 -0400:
It should only be used if you know how it works and what you are doing.
Isn't that right for everything?
Kai
On Fri, Jun 27, 2008 at 10:54 AM, Kai Schaetzl maillists@conactive.com wrote:
Toby Bluhm wrote on Fri, 27 Jun 2008 12:46:43 -0400:
It should only be used if you know how it works and what you are doing.
Isn't that right for everything?
My only comment would be that you are asking for problems when you nfs-mount the root.
mhr
-
Jason Pyeron -
Kai Schaetzl -
MHR -
Toby Bluhm