hi, we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie: - if one (or more) of the hdd or server fails the whole filesystem still usable and consistent. - any server in this farm can see the same storage/filesystem. it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers. thank you for your help in advance.
I have never done this but this is where I would start
1 - raid - hardware/software scsi/ide - depending on your money/needs (personally I like software w/ sata drives) 2 - 2 servers 3 - network raid 1 or rsync 4 - heartbeat to switch the servers when one goes down.
On 6/13/07, Farkas Levente lfarkas@bppiac.hu wrote:
hi, we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem still
usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers. thank you for your help in advance.
-- Levente "Si vis pacem para bellum!"
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
----- "Farkas Levente" lfarkas@bppiac.hu escreveu:
we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem
still usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Antonio.
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
----- "Farkas Levente" lfarkas@bppiac.hu escreveu:
we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem
still usable and consistent.
...
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
If want one filesystem overy many disks on many servers and the ability to fail a disk-volume (raid or whatever) or an entire server and still have a usable fs then you need something like GPFS with replication enabled. Either way, not a trivial config nor trivial software (GPFS for example will cost you $$$ unless you're academic).
good luck, Peter
Hi,
Well, you can take a look around at Oracle Cluster File System. I've been using it, for a while, and so far so good. It's free, runs on top of Linux, and despite it's name , it's not a file system only for Oracle apps, and it's part of the kernel since version 2.6.16 .
Check it at http://oss.oracle.com/projects/ocfs2/
Best regards, Bruno Sousa
--- Mensagem Original---
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
----- "Farkas Levente" lfarkas@bppiac.hu escreveu:
we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem
still usable and consistent.
...
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
If want one filesystem overy many disks on many servers and the ability to fail a disk-volume (raid or whatever) or an entire server and still have a usable fs then you need something like GPFS with replication enabled. Either way, not a trivial config nor trivial software (GPFS for example will cost you $$$ unless you're academic).
good luck, Peter
--------_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
afaik ocsf2 is not redundant:-(
bruno.sousa@first.pt wrote:
Hi,
Well, you can take a look around at Oracle Cluster File System. I've been using it, for a while, and so far so good. It's free, runs on top of Linux, and despite it's name , it's not a file system only for Oracle apps, and it's part of the kernel since version 2.6.16 .
Check it at http://oss.oracle.com/projects/ocfs2/
Best regards, Bruno Sousa
--- Mensagem Original---
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
----- "Farkas Levente" lfarkas@bppiac.hu escreveu:
we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem
still usable and consistent.
...
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
If want one filesystem overy many disks on many servers and the ability to fail a disk-volume (raid or whatever) or an entire server and still have a usable fs then you need something like GPFS with replication enabled. Either way, not a trivial config nor trivial software (GPFS for example will cost you $$$ unless you're academic).
good luck, Peter
--------_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Thu, Jun 14, 2007 at 11:35:30PM +0200, Farkas Levente wrote:
afaik ocsf2 is not redundant:-(
It isn't redundant per se, but it allows you to access a shared storage (hardware that supports it or network raid as I explained in my other mail) simultaneously in multiple servers. Thus, you can have the service available if multiple servers. If the service doesn't support multiple instances, then you have to start it when the primary fails. In which case, you wouldn't even need OCFS2 and could use ext3/xfs/...
On Thu, Jun 14, 2007 at 01:43:22PM +0200, Peter Kjellstrom wrote:
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
You can add standard raid for eliminating disk failures as points of failure.
Then there are drbd[1], ddraid[2] and (g)nbd + md for server raid.
I've been using drbd, and now I'm migrating to use ocfs2 + nfs on top of it.
Coda also has a replicated server mode, but my system exceeded the number of files supported by coda.
[1] http://www.drbd.org/ [2] http://sourceware.org/cluster/ddraid/
Luciano Rocha wrote:
On Thu, Jun 14, 2007 at 01:43:22PM +0200, Peter Kjellstrom wrote:
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
You can add standard raid for eliminating disk failures as points of failure.
Then there are drbd[1], ddraid[2] and (g)nbd + md for server raid.
I've been using drbd, and now I'm migrating to use ocfs2 + nfs on top of it.
what will do the redundancy here?
On Thu, Jun 14, 2007 at 11:39:12PM +0200, Farkas Levente wrote:
Luciano Rocha wrote:
On Thu, Jun 14, 2007 at 01:43:22PM +0200, Peter Kjellstrom wrote:
On Wednesday 13 June 2007, Antonio da Silva Martins Junior wrote:
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
Neither will eliminate servers and disks as single points of failiure.
You can add standard raid for eliminating disk failures as points of failure.
Then there are drbd[1], ddraid[2] and (g)nbd + md for server raid.
I've been using drbd, and now I'm migrating to use ocfs2 + nfs on top of it.
what will do the redundancy here?
Hm? They work as software raid over the network, keeping the data in sync. So, if one fails, the other still has the data and can keep serving it.
You can have more than one master[1], if you use a cluster filesystem (GFS/OCFS2/?) or if you just export the redudant block device. Otherwise, use heartbeat/ucarp/... to change the secondary to primary.
[1] drbd only supports two peers. drbd+, a commercial version, supports more, I think.
Antonio da Silva Martins Junior wrote:
----- "Farkas Levente" lfarkas@bppiac.hu escreveu:
we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem
still usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
Hi Farkas,
I think a start is to look on PVFS2 (www.pvfs.org).
Or maybe using nbd and softwareraid ???
i just read on linux-raid nbd has problem with swraid:-(
Farkas Levente napsal(a):
hi, we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem still
usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
what about souch FS blocks path
server->gfs->LVM-> raid1(disc0)-> iscsi initiator->local-eth0->LANswitch0->remote-eth0->iscsi-target-on_file_server0->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......) ->LVM-> raid1(disc1)-> iscsi initiator->local-eth1->LANswitch1->remote-eth0->iscsi-target-on_file_server1->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......) -------------||----------------
in souch setup you have no single point of failure
bacause of money you can decrease number of switches (FE to 2 and connect each server and FS to both of them)
If you do not need concurent access from 1+ servers to one fs (partition, logicalvolume, same file ...) you do not need gfs ..
Petr "Qaxi" Klíma wrote:
Farkas Levente napsal(a):
hi, we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem still
usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
what about souch FS blocks path
server->gfs->LVM-> raid1(disc0)-> iscsi initiator->local-eth0->LANswitch0->remote-eth0->iscsi-target-on_file_server0->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......)
\->LVM-> raid1(disc1)-> iscsiinitiator->local-eth1->LANswitch1->remote-eth0->iscsi-target-on_file_server1->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......)
\-------------||----------------in souch setup you have no single point of failure
bacause of money you can decrease number of switches (FE to 2 and connect each server and FS to both of them)
If you do not need concurent access from 1+ servers to one fs (partition, logicalvolume, same file ...) you do not need gfs ..
sorry this graph is a bit complicated:-( how comes the iscsi into this image? iscsi under raid? can you explain it a bit more detailed? thanks.
Farkas Levente wrote:
Petr "Qaxi" Klíma wrote:
Farkas Levente napsal(a):
hi, we've a few 10-20 server in a lan each has 4-8 hdd. we'd like to create one big file server on these server hard disks and we'd like to create it in a redundant way ie:
- if one (or more) of the hdd or server fails the whole filesystem still
usable and consistent.
- any server in this farm can see the same storage/filesystem.
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
what about souch FS blocks path
server->gfs->LVM-> raid1(disc0)-> iscsi initiator->local-eth0->LANswitch0->remote-eth0->iscsi-target-on_file_server0->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......)
\->LVM-> raid1(disc1)-> iscsiinitiator->local-eth1->LANswitch1->remote-eth0->iscsi-target-on_file_server1->LVM->raid(1,5,6,...)->HDD(1,2,3,4,......)
\-------------||----------------in souch setup you have no single point of failure
bacause of money you can decrease number of switches (FE to 2 and connect each server and FS to both of them)
If you do not need concurent access from 1+ servers to one fs (partition, logicalvolume, same file ...) you do not need gfs ..
sorry this graph is a bit complicated:-( how comes the iscsi into this image? iscsi under raid?
In this scenario, iscsi provides the devices remotely and the server handles the raiding of the devices.
can you explain it a bit more detailed?
The boxes with disks are now just 'disk servers' and those disks are exported to the servers that will provide the filesystem layer. iscsi is the technology used to export the disks in this scenario.
Feizhou wrote:
In this scenario, iscsi provides the devices remotely and the server handles the raiding of the devices.
can you explain it a bit more detailed?
The boxes with disks are now just 'disk servers' and those disks are exported to the servers that will provide the filesystem layer. iscsi is the technology used to export the disks in this scenario.
Is there a neat way to get start the partitions and get them mounted in this setup? I have a server on which I want to create /dev/md0 from two iscsi partitions. I then want to export /md0 to other servers using GFS.
Do I have to hand-craft my own scripts to start the md devices at boot time and get the filesystems mounted before starting GFS, or is there already a way to do that? (I don't have a problem with doing so, but if there's already a "right" way to do it...)
James
James Fidell wrote:
Feizhou wrote:
In this scenario, iscsi provides the devices remotely and the server handles the raiding of the devices.
can you explain it a bit more detailed?
The boxes with disks are now just 'disk servers' and those disks are exported to the servers that will provide the filesystem layer. iscsi is the technology used to export the disks in this scenario.
Is there a neat way to get start the partitions and get them mounted in this setup? I have a server on which I want to create /dev/md0 from two iscsi partitions. I then want to export /md0 to other servers using GFS.
What are you going to use to export the /dev/md0? gnbd? Does your iscsi_target have a maximum of one connection limit?
Do I have to hand-craft my own scripts to start the md devices at boot time and get the filesystems mounted before starting GFS, or is there already a way to do that? (I don't have a problem with doing so, but if there's already a "right" way to do it...)
No idea. I have not actually had a chance to get GFS working whether with iscsi or with some other device exporting solution. Sorry I cannot help here. Maybe those who have had real experience can chip in.
On 6/13/07, Farkas Levente lfarkas@bppiac.hu wrote:
it's someting a big network raid5-6... storage where we have about 40<partition added to the same network volume. and there is an fs over it which hide all internal network raid functionality. is there any such solution? i can't find any way to do this on our linux servers.
I saw a presentation from the guy behind mogilefs. It sounds exactly like what you're looking for.
-
"Petr \"Qaxi\" Klíma" -
Antonio da Silva Martins Junior -
Bazooka Joe -
bruno.sousa＠first.pt -
Farkas Levente -
Feizhou -
Francois Caen -
James Fidell -
Luciano Rocha -
Peter Kjellstrom