Hi,
I've just got possession of a Dell PE R720 with 2 MD1200 disk enclosures.
Both MD1200 are fully populated with 12 x 3 TB disks
The system will basically be a student file-server running CentOS 6.x serving various size files from small c programs to multi gigabyte audio and video files over GB ethernet.
The first MD1200 will be configured as the NFS disk. The requirements are for 6 fixed equally sized partitions, one for each cohort of students. For this I was thinking of splitting the MD1200 into 2 RAID5 arrays with a hot spare each. Then partitioning each into 3 ext4 partitions.
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Any suggestions.
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
Thanks,
Tony
--
Tony Molloy
CTO, Dept. of Comp. Sci. University of Limerick Limerick. Ireland
Hi Tony,
because I suggest just something very general I post off list :)
From my POV as I'm currently facing similar setups with different hardware rolling back from fine granular setups to simple 'bigger' less complex configurations. (we do have 6 iscsi storages from 2TB (sun ZFS) up to 32 TB)
keep it small and simple! :)
I think you are very familiar with the general problems of big HW raids and big filesystems like rebuild or check times, but splitting up and adding more complex layers like multiple raids joining in lvm etc. makes debugging and general handling very hard.
On the other hand, I checked and read a lot about filesystems the last days being faced with serving user windows samba profiles with lot of small files and big video/audio data etc.
Long story short:
I usually do one raidvolume per hardware raid box; e.g. we use 16*1TB drives. Raid6 or Raid5 with spare. I did not notice big performance differences.
I use LVM to make partitions or I prefer using just one big partition.
I tried xfs and ext4 and will go with ext4 as some test went better for my setup and from what I read it looks not bad :)
I think you can combine block level devices (like multiple raid boxes) by LVM into one bigger LV.
And last but not least: The CPU/RAM/Network of the host serving the files is also very important! :)
I noticed, that the same iscsi storage got about 70MB/s on a new server (xeon multicore), while on the old fileserver it just got up to 40MB/s.
my2cents :) regards . Götz
May be worth reading:
http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_sma...
http://monolight.cc/2011/02/linux-filesystems-small-file-performance-on-hdds...
http://unix.stackexchange.com/questions/28756/what-is-the-most-high-performa...
https://wiki.archlinux.org/index.php/Ext3
https://wiki.archlinux.org/index.php/Ext4#Tips_and_tricks
Am 04.09.12 13:10, schrieb Tony Molloy:
Hi,
I've just got possession of a Dell PE R720 with 2 MD1200 disk enclosures.
Both MD1200 are fully populated with 12 x 3 TB disks
The system will basically be a student file-server running CentOS 6.x serving various size files from small c programs to multi gigabyte audio and video files over GB ethernet.
The first MD1200 will be configured as the NFS disk. The requirements are for 6 fixed equally sized partitions, one for each cohort of students. For this I was thinking of splitting the MD1200 into 2 RAID5 arrays with a hot spare each. Then partitioning each into 3 ext4 partitions.
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Any suggestions.
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
Thanks,
Tony
--
Tony Molloy
CTO, Dept. of Comp. Sci. University of Limerick Limerick. Ireland _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Tuesday 04 September 2012 12:44:26 Götz Reinicke wrote:
Hi Tony,
because I suggest just something very general I post off list :)
From my POV as I'm currently facing similar setups with different hardware rolling back from fine granular setups to simple 'bigger' less complex configurations. (we do have 6 iscsi storages from 2TB (sun ZFS) up to 32 TB)
keep it small and simple! :)
I think you are very familiar with the general problems of big HW raids and big filesystems like rebuild or check times, but splitting up and adding more complex layers like multiple raids joining in lvm etc. makes debugging and general handling very hard.
On the other hand, I checked and read a lot about filesystems the last days being faced with serving user windows samba profiles with lot of small files and big video/audio data etc.
Long story short:
I usually do one raidvolume per hardware raid box; e.g. we use 16*1TB drives. Raid6 or Raid5 with spare. I did not notice big performance differences.
I use LVM to make partitions or I prefer using just one big partition.
I tried xfs and ext4 and will go with ext4 as some test went better for my setup and from what I read it looks not bad :)
I think you can combine block level devices (like multiple raid boxes) by LVM into one bigger LV.
And last but not least: The CPU/RAM/Network of the host serving the files is also very important! :)
I noticed, that the same iscsi storage got about 70MB/s on a new server (xeon multicore), while on the old fileserver it just got up to 40MB/s.
my2cents :) regards . Götz
May be worth reading:
http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tunin g_for_small_files_parallel_i_o_on_debian#.UEPSI1RqYso
http://monolight.cc/2011/02/linux-filesystems-small-file-performanc e-on-hdds/
http://unix.stackexchange.com/questions/28756/what-is-the-most-high -performance-linux-filesystem-for-storing-a-lot-of-small-fi
https://wiki.archlinux.org/index.php/Ext3
https://wiki.archlinux.org/index.php/Ext4#Tips_and_tricks
Am 04.09.12 13:10, schrieb Tony Molloy:
Hi,
I've just got possession of a Dell PE R720 with 2 MD1200 disk enclosures.
Both MD1200 are fully populated with 12 x 3 TB disks
The system will basically be a student file-server running CentOS 6.x serving various size files from small c programs to multi gigabyte audio and video files over GB ethernet.
The first MD1200 will be configured as the NFS disk. The requirements are for 6 fixed equally sized partitions, one for each cohort of students. For this I was thinking of splitting the MD1200 into 2 RAID5 arrays with a hot spare each. Then partitioning each into 3 ext4 partitions.
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Any suggestions.
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
Thanks,
Tony
--
Tony Molloy
CTO, Dept. of Comp. Sci. University of Limerick Limerick. Ireland _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Tony Molloy wrote the following on 9/4/2012 6:10 AM:
Hi,
I've just got possession of a Dell PE R720 with 2 MD1200 disk enclosures.
Both MD1200 are fully populated with 12 x 3 TB disks
The system will basically be a student file-server running CentOS 6.x serving various size files from small c programs to multi gigabyte audio and video files over GB ethernet.
The first MD1200 will be configured as the NFS disk. The requirements are for 6 fixed equally sized partitions, one for each cohort of students. For this I was thinking of splitting the MD1200 into 2 RAID5 arrays with a hot spare each. Then partitioning each into 3 ext4 partitions.
I think that sounds like a balanced approach. With 12 SATA drives you're bound to run into failures during the life of these devices (3-5 yrs). By splitting the disks into 2 arrays, a failure of one disk will only degrade 1 array, affecting less users. Rebuild times will be reduced if there is a failure and performance for multi-user workloads should be increased by separating into multiple arrays.
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
I've never found any value in LVM. To me, it adds another layer of complexity or room for bugs to creep in. Arrays can be resized, GPT or MSDOS partitions can be resized, file systems can be resized, your hardware can do the spanning that LVM provides, why would you want to add another layer of abstraction? For similar reasons, I would stick to a file system that comes stock with your OS, ext4 or XFS should be fine, as they are both well supported in CentOS.
Any suggestions.
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
At first, I was going to recommend 1 hot spare, but if you think the new admin may not have as much time, or you want things a little more simple, then making two identical arrays is probably going to be easier. If he's not as familiar with your hardware, then I would print out a cheat sheet on how to rebuild the array, check the status (MegaCLI?, OpenManage?), etc. As always, document what you have, how you put it together, and who to call or where to look if he's stuck.
--Blake
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Just a thought about the backup volume.
Because it going to be used for backups and your retirement (congrats!) I would configure one partition and no volume management on top. Why? Because backups are like insurance policies. That are not fun and no one reviews them, but when you need them, you really hope that everything is in order. So you don't want the next admin to find out several months or years down the road that the backup software ran out of space and has been failing to back anything up for the previous few months.
On Tuesday 04 September 2012 15:20:26 Paul Tader wrote:
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Just a thought about the backup volume.
Because it going to be used for backups and your retirement (congrats!) I would configure one partition and no volume management on top. Why? Because backups are like insurance policies. That are not fun and no one reviews them, but when you need them, you really hope that everything is in order. So you don't want the next admin to find out several months or years down the road that the backup software ran out of space and has been failing to back anything up for the previous few months.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks to Gotz, Blake, Paul and Nate for your advice.
I was pretty happy with the split RAID 5 for the NFS partitions on the first MD1200.
I think I'll go with the simple solution for the backup MD1200. A single RAID 5 encompassing all 12 disks probably with 2 hot spares. Put an ext4 on top of that without the complexity of LVM.
Thanks,
Tony
aside Paul, unfortunately my retirement is not voluntary. It's due to a spinal cord injury but such is life ;-)
On Wed, Sep 5, 2012 at 12:02 PM, Tony Molloy tony.molloy@ul.ie wrote:
On Tuesday 04 September 2012 15:20:26 Paul Tader wrote:
The second MD1200 will be used to backup the first, using BackupPC and for other storage purposes.
As I won't know the storage requirements for the "backup partition" and they will probably change over time anyway. I was thinking of using LVM for it. So how to partition the MD1200 for LVM. I don't want to put all 12 disks in a RAID5 and put a LVM volume on it. Can I split it into 2 RAID5 and have a LVM volume spanning both.
Just a thought about the backup volume.
Because it going to be used for backups and your retirement (congrats!) I would configure one partition and no volume management on top. Why? Because backups are like insurance policies. That are not fun and no one reviews them, but when you need them, you really hope that everything is in order. So you don't want the next admin to find out several months or years down the road that the backup software ran out of space and has been failing to back anything up for the previous few months.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks to Gotz, Blake, Paul and Nate for your advice.
I was pretty happy with the split RAID 5 for the NFS partitions on the first MD1200.
I think I'll go with the simple solution for the backup MD1200. A single RAID 5 encompassing all 12 disks probably with 2 hot spares. Put an ext4 on top of that without the complexity of LVM.
Thanks,
Tony
aside Paul, unfortunately my retirement is not voluntary. It's due to a spinal cord injury but such is life ;-)
I would highly suggest RAID 6 if you are dealing with drives over 2TB in size. Errors during rebuild happen and rebuild times for 2TB+ drives can get pretty long, especially if you are building one 12 disk RAID.
Also important if a non-savvy admin is going to take over administration. What if they pull the wrong drive during a replacement on a failed RAID5 array?
On Sep 4, 2012, at 5:10 AM, Tony Molloy tony.molloy@ul.ie wrote:
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
Hand him the machine and tell him to load Windows on it or whatever he wants to maintain for the next X years, relax for 30 days and enjoy retirement. It's his problem now. :-) :-) ;-)
Nate
On Tuesday 04 September 2012 19:25:05 Nate Duehr wrote:
On Sep 4, 2012, at 5:10 AM, Tony Molloy tony.molloy@ul.ie wrote:
Just remember I'm due to retire at the end of this month so this will be my last big job for the Dept. And due to financial constraints I will not be replaced. So I will be handing this machine over to a co- worker who is basically a Windoze admin with only a basic knowledge of Linux so nothing too fancy. ;-)
Hand him the machine and tell him to load Windows on it or whatever he wants to maintain for the next X years, relax for 30 days and enjoy retirement. It's his problem now. :-) :-) ;-)
Nate
Sounds good but it's not the Linux way is it ;-)
Anyway I want to force him to become a better person, to grow and expand his knowledge and in the end maybe drag him away from the evil empire to the land of the free !!!!!
LOL
Tony
-
Blake Hudson -
Dean Jones -
Götz Reinicke -
Nate Duehr -
Paul Tader -
Tony Molloy