Hello
Since linux 2.6, the md layer has a feature called partitionable arrays. So instead of having two disks, creating an identical partition table on both and then putting those partitions in RAID 1, you take those two disks and put them in one partitionable RAID 1 array (in mdadm terms, "mdp") and create a partition table on the new RAID device. The advantages are quite clear compared to the old non-partitionable arrays.
My question is, is this supported by CentOS? The GTK installer doesn't provide a way to create such an mdp device and the integrated partitioning tool does not see for example md_d0 when I create it manually from the console.
Another way to get CentOS on such a configuration would be to do everything manually, thus installing the base system by creating the necessary disk allocations and then rpm -i all the required packages to get it to boot. (I've done this before, it's not a big deal, you just need to follow a certain order - I remember documenting it somewhere but forgot). But since this method is probably not officially documented anywhere or even supported I'll most likely won't get any support if this setup were to fail somehow (like when upgrading between minor versions).
I've tried STFW'ing, but searching for centos and partitionable arrays is too ambiguous.
Thanks,
Best regards,
Glenn Matthys
RedShift wrote:
Hello
Since linux 2.6, the md layer has a feature called partitionable arrays. So instead of having two disks, creating an identical partition table on both and then putting those partitions in RAID 1, you take those two disks and put them in one partitionable RAID 1 array (in mdadm terms, "mdp") and create a partition table on the new RAID device. The advantages are quite clear compared to the old non-partitionable arrays.
My question is, is this supported by CentOS? The GTK installer doesn't provide a way to create such an mdp device and the integrated partitioning tool does not see for example md_d0 when I create it manually from the console.
Another way to get CentOS on such a configuration would be to do everything manually, thus installing the base system by creating the necessary disk allocations and then rpm -i all the required packages to get it to boot. (I've done this before, it's not a big deal, you just need to follow a certain order - I remember documenting it somewhere but forgot). But since this method is probably not officially documented anywhere or even supported I'll most likely won't get any support if this setup were to fail somehow (like when upgrading between minor versions).
I've tried STFW'ing, but searching for centos and partitionable arrays is too ambiguous.
Thanks,
Best regards,
Glenn Matthys
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
# First, setup your disks to your liking. You can use whatever you want here, # RAID, LVM, etc... Remember your disk configuration because you'll need it # to configure grub, menu.lst and fstab. Using RAID, LVM, or others will require # more configuration than this guide covers. To keep it simple I'm using a # single disk. An example:
$ fdisk /dev/sda $ mount /dev/sda3 /target $ mkdir /target/boot $ mount /dev/sda1 /target/boot
# Depending on the host OS you're using, you may need to initialize the rpm db # on the host OS $ rpm --initdb
# Use the following command to install the packages. I'll be addressing this # command as $rpm.
$ rpm --root /target -i
# Use your shell's tab completion to complete the package filenames. I # deliberatly left out the versions so these instructions apply to a wide range # of versions
# Let's install some basics $rpm setup basesystem filesystem
# Install bash first, this is needed for post-install scripts $rpm bash glibc glibc-common termcap libgcc tzdata mktemp libtermcap
# Install some dependencies (this is mainly to keep the next command smaller) $rpm grep pcre libstdc++ info ncurses zlib gawk sed ethtool
# Install the bulk of the system $rpm coreutils libselinux libacl libattr pam audit-libs cracklib-dicts \ cracklib libsepol mcstrans libcap chkconfig python db4 openssl readline \ bzip2-libs gdbm findutils krb5-libs initscripts util-linux popt udev MAKEDEV \ centos-release shadow-utils keyutils-libs iproute sysfsutils SysVinit \ net-tools module-init-tools e2fsprogs e2fsprogs-libs glib2 mingetty \ device-mapper sysklogd psmisc centos-release-notes procps libsysfs iputils
# Install package manager $rpm rpm beecrypt elfutils-libelf rpm-libs sqlite
# Install YUM $rpm yum python-elementtree rpm-python yum-metadata-parser python-sqlite \ expat libxml2 python-urlgrabber m2crypto python-iniparse
# You may also want to install your favorite editor $rpm nano
# This provides /root with some defaults, like color highlighting on `ls` $rpm rootfiles
# Right now you have system which you can chroot to, so we can start setting up # the basics
# Mount directories for chroot operation $ mount --bind /dev /target/dev $ mount -t proc none /target/proc $ mount -t sysfs none /target/sysfs $ chroot /target
# This constructs /etc/shadow $ pwconv
# Configure fstab $ nano -w /etc/fstab
# Installing the kernel. Do this back outside the chroot in the host OS system $ exit $rpm kernel mkinitrd cpio device-mapper-multipath dmraid gzip kpartx lvm2 nash \ tar less device-mapper-event
# Install the bootloader, grub. $rpm grub diffutils redhat-logos
# Let's chroot again to configure our bootloader $ chroot /target
# We start by configuring the bootloader. Open /boot/grub/menu.lst, and put the # following there
<<<MENU.LST timeout 5 default 0
# (0) CentOS title CentOS root (hd0,0) kernel /vmlinuz-2.6.18-92.el5 root=/dev/sda3 ro initrd /initrd-2.6.18-92.el5.img
# If this command gives an error, you can safely ignore this because it's not # of importance. What is important is that grub-install copied the right files # to /boot/grub that we need for booting. $ /sbin/grub-install /dev/sda
# Manually install grub if the previous step failed. - means type it in the grub # shell $ grub $- root (hd0,0) $- setup (hd0)
# Optional packages # You may want to install passwd so you can set passwords ;-) $rpm passwd libuser openldap cyrus-sasl-lib
# These are used to set the keyboard language (loadkeys) $rpm kbd usermode
# ** Right now you should have a bootable system! Here are some tips to help you # through your 1st boot ***
# Most of the system configuration happens in /etc/sysconfig. See
/usr/share/doc/initscripts
for full documentation.
Some quick post-install tips: * Configure your keyboard in /etc/sysconfig/keyboard using the KEYTABLE variable
* Configure networking Take a look at /etc/sysconfig/network-scripts. See ifcfg-lo for an example.
# This recreates the RPM database. If the host OS you used has a different # version of db, rpm will complain with # rpmdb: unable to lock mutex: Invalid argument $ rpmdb --rebuilddb
Best regards,
Glenn Matthys
(PS: I've also attached the documentation as install_centos.txt, but mailman will probably strip it)
# First, setup your disks to your liking. You can use whatever you want here, # RAID, LVM, etc... Remember your disk configuration because you'll need it # to configure grub, menu.lst and fstab. Using RAID, LVM, or others will require # more configuration than this guide covers. To keep it simple I'm using a # single disk. An example:
$ fdisk /dev/sda $ mount /dev/sda3 /target $ mkdir /target/boot $ mount /dev/sda1 /target/boot
# Depending on the host OS you're using, you may need to initialize the rpm db # on the host OS $ rpm --initdb
# Use the following command to install the packages. I'll be addressing this # command as $rpm.
$ rpm --root /target -i
# Use your shell's tab completion to complete the package filenames. I # deliberatly left out the versions so these instructions apply to a wide range # of versions
# Let's install some basics $rpm setup basesystem filesystem
# Install bash first, this is needed for post-install scripts $rpm bash glibc glibc-common termcap libgcc tzdata mktemp libtermcap
# Install some dependencies (this is mainly to keep the next command smaller) $rpm grep pcre libstdc++ info ncurses zlib gawk sed ethtool
# Install the bulk of the system $rpm coreutils libselinux libacl libattr pam audit-libs cracklib-dicts \ cracklib libsepol mcstrans libcap chkconfig python db4 openssl readline \ bzip2-libs gdbm findutils krb5-libs initscripts util-linux popt udev MAKEDEV \ centos-release shadow-utils keyutils-libs iproute sysfsutils SysVinit \ net-tools module-init-tools e2fsprogs e2fsprogs-libs glib2 mingetty \ device-mapper sysklogd psmisc centos-release-notes procps libsysfs iputils
# Install package manager $rpm rpm beecrypt elfutils-libelf rpm-libs sqlite
# Install YUM $rpm yum python-elementtree rpm-python yum-metadata-parser python-sqlite \ expat libxml2 python-urlgrabber m2crypto python-iniparse
# You may also want to install your favorite editor $rpm nano
# This provides /root with some defaults, like color highlighting on `ls` $rpm rootfiles
# Right now you have system which you can chroot to, so we can start setting up # the basics
# Mount directories for chroot operation $ mount --bind /dev /target/dev $ mount -t proc none /target/proc $ mount -t sysfs none /target/sysfs $ chroot /target
# This constructs /etc/shadow $ pwconv
# Configure fstab $ nano -w /etc/fstab
# Installing the kernel. Do this back outside the chroot in the host OS system $ exit $rpm kernel mkinitrd cpio device-mapper-multipath dmraid gzip kpartx lvm2 nash \ tar less device-mapper-event
# Install the bootloader, grub. $rpm grub diffutils redhat-logos
# Let's chroot again to configure our bootloader $ chroot /target
# We start by configuring the bootloader. Open /boot/grub/menu.lst, and put the # following there
<<<MENU.LST timeout 5 default 0
# (0) CentOS title CentOS root (hd0,0) kernel /vmlinuz-2.6.18-92.el5 root=/dev/sda3 ro initrd /initrd-2.6.18-92.el5.img
# If this command gives an error, you can safely ignore this because it's not # of importance. What is important is that grub-install copied the right files # to /boot/grub that we need for booting. $ /sbin/grub-install /dev/sda
# Manually install grub if the previous step failed. - means type it in the grub # shell $ grub $- root (hd0,0) $- setup (hd0)
# Optional packages # You may want to install passwd so you can set passwords ;-) $rpm passwd libuser openldap cyrus-sasl-lib
# These are used to set the keyboard language (loadkeys) $rpm kbd usermode
# ** Right now you should have a bootable system! Here are some tips to help you # through your 1st boot ***
# Most of the system configuration happens in /etc/sysconfig. See
/usr/share/doc/initscripts
for full documentation.
Some quick post-install tips: * Configure your keyboard in /etc/sysconfig/keyboard using the KEYTABLE variable
* Configure networking Take a look at /etc/sysconfig/network-scripts. See ifcfg-lo for an example.
# This recreates the RPM database. If the host OS you used has a different # version of db, rpm will complain with # rpmdb: unable to lock mutex: Invalid argument $ rpmdb --rebuilddb
RedShift wrote:
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
That looks useful. Do you have any hints about how to get the right drivers installed if you wanted to build a disk to be moved to a different machine?
On Mar 26, 2009, at 8:55 AM, Les Mikesell lesmikesell@gmail.com wrote:
RedShift wrote:
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
That looks useful. Do you have any hints about how to get the right drivers installed if you wanted to build a disk to be moved to a different machine?
That's even easier.
Add the disk driver names in modprobe.conf the ones for system disks in the top half, data disks below. Then run a mkinitrd.
Modprobe.conf excerpt:
alias scsi_adapter ata_piix alias scsi_adapter0 ahci alias scsi_adapter1 mega_sas alias scsi_adapter2 mpt
# mv /boot/init-$(uname -r).img /boot/init-$(uname -r).img # mkinitrd /boot/init-$(uname -r).img $(uname -r)
That should make an initrd with the drivers necessary to boot your other boxes (of course using your own disk drivers and not mine).
-Ross
Ross Walker wrote:
RedShift wrote:
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
That looks useful. Do you have any hints about how to get the right drivers installed if you wanted to build a disk to be moved to a different machine?
That's even easier.
Add the disk driver names in modprobe.conf the ones for system disks in the top half, data disks below. Then run a mkinitrd.
Modprobe.conf excerpt:
alias scsi_adapter ata_piix alias scsi_adapter0 ahci alias scsi_adapter1 mega_sas alias scsi_adapter2 mpt
# mv /boot/init-$(uname -r).img /boot/init-$(uname -r).img # mkinitrd /boot/init-$(uname -r).img $(uname -r)
That should make an initrd with the drivers necessary to boot your other boxes (of course using your own disk drivers and not mine).
Thanks - but there is another half to that question. How do you find the names of the drivers that match any particular hardware without running the installer?
I'd like to have a generic backup/restore mechanism that would drop in a tar image (etc.) from one machine and come up running on something different - or a fixup procedure for disks that have been moved from one chassis to another. Even where the machines are identical and I put the target machine's MAC addresses in the ifcfg-ethX file, something seems to rename them and screw things up when a disk is moved.
At Thu, 26 Mar 2009 08:44:34 -0500 CentOS mailing list centos@centos.org wrote:
Ross Walker wrote:
RedShift wrote:
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
That looks useful. Do you have any hints about how to get the right drivers installed if you wanted to build a disk to be moved to a different machine?
That's even easier.
Add the disk driver names in modprobe.conf the ones for system disks in the top half, data disks below. Then run a mkinitrd.
Modprobe.conf excerpt:
alias scsi_adapter ata_piix alias scsi_adapter0 ahci alias scsi_adapter1 mega_sas alias scsi_adapter2 mpt
# mv /boot/init-$(uname -r).img /boot/init-$(uname -r).img # mkinitrd /boot/init-$(uname -r).img $(uname -r)
That should make an initrd with the drivers necessary to boot your other boxes (of course using your own disk drivers and not mine).
Thanks - but there is another half to that question. How do you find the names of the drivers that match any particular hardware without running the installer?
Installing the kernel source RPM is one option. Then it is a matter of the use of grep to search in the source code for the devices you need drivers for.
You can also 'cheat' by copying the PXE boot initrd file from the first CD or the DVD and unpack this file. It is a compressed cpio file:
gunzip < /path/to/pxeinitrd|(cd /some/temp/directory;cpio -i)
In the /some/temp/directory there will be a subdir named modules, with a file named modules.alias. This file contains a mapping of PCI vendor/device IDs to module names. lspci will give you the vendor/device IDs of the devices in question or you can look them up in /usr/share/hwdata/pci.ids.
I'd like to have a generic backup/restore mechanism that would drop in a tar image (etc.) from one machine and come up running on something different - or a fixup procedure for disks that have been moved from one chassis to another. Even where the machines are identical and I put the target machine's MAC addresses in the ifcfg-ethX file, something seems to rename them and screw things up when a disk is moved.
RedShift wrote:
RedShift wrote:
Hello
Since linux 2.6, the md layer has a feature called partitionable arrays. So instead of having two disks, creating an identical partition table on both and then putting those partitions in RAID 1, you take those two disks and put them in one partitionable RAID 1 array (in mdadm terms, "mdp") and create a partition table on the new RAID device. The advantages are quite clear compared to the old non-partitionable arrays.
My question is, is this supported by CentOS? The GTK installer doesn't provide a way to create such an mdp device and the integrated partitioning tool does not see for example md_d0 when I create it manually from the console.
I expect we may have to wait on upstream to have installer support. Anybody with RHEL want to put in an official request?
Another way to get CentOS on such a configuration would be to do everything manually, thus installing the base system by creating the necessary disk allocations and then rpm -i all the required packages to get it to boot. (I've done this before, it's not a big deal, you just need to follow a certain order - I remember documenting it somewhere but forgot). But since this method is probably not officially documented anywhere or even supported I'll most likely won't get any support if this setup were to fail somehow (like when upgrading between minor versions).
I've tried STFW'ing, but searching for centos and partitionable arrays is too ambiguous.
I tried googling too, and came up with lots of docs on "partitionable arrays", but nothing on installing. Can't say for sure without testing, but I suspect GRUB would choke on this. Would probably still need at least a /boot on a separate partition, or a standard RAID1.
Thanks,
Best regards,
Glenn Matthys
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
# First, setup your disks to your liking. You can use whatever you want here,
... snip ...
(PS: I've also attached the documentation as install_centos.txt, but mailman will probably strip it)
Attachment came through fine for me. Very interesting - might make a nice Wiki article, and could be included on a LiveCD as a way of bootstrapping a CentOS install.
Phil
Phil Schaffner wrote:
RedShift wrote:
Another way to get CentOS on such a configuration would be to do everything manually, thus installing the base system by creating the necessary disk allocations and then rpm -i all the required packages to get it to boot. (I've done this before, it's not a big deal, you just need to follow a certain order - I remember documenting it somewhere but forgot). But since this method is probably not officially documented anywhere or even supported I'll most likely won't get any support if this setup were to fail somehow (like when upgrading between minor versions).
I've tried STFW'ing, but searching for centos and partitionable arrays is too ambiguous.
I tried googling too, and came up with lots of docs on "partitionable arrays", but nothing on installing. Can't say for sure without testing, but I suspect GRUB would choke on this. Would probably still need at least a /boot on a separate partition, or a standard RAID1.
GRUB works at least with a RAID 1 setup. (I run it in production on another distro). On a partitionable RAID 1, the data can still be read independently from the disks (that allows GRUB to work). If you have two disks you would install your GRUB MBR twice, once on both disks using the GRUB shell. I haven't tried other RAID forms but I see no reason why the built-in RAID 10 would not work as well.
Thanks,
Best regards,
Glenn Matthys
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
# First, setup your disks to your liking. You can use whatever you want here,
... snip ...
(PS: I've also attached the documentation as install_centos.txt, but mailman will probably strip it)
Attachment came through fine for me. Very interesting - might make a nice Wiki article, and could be included on a LiveCD as a way of bootstrapping a CentOS install.
I'll have a go at that.
RedShift wrote:
Glenn Matthys
As a follow-up, I found the documentation I wrote how to install CentOS without any installer:
# First, setup your disks to your liking. You can use whatever you want here,
... snip ...
(PS: I've also attached the documentation as install_centos.txt, but mailman will probably strip it)
Attachment came through fine for me. Very interesting - might make a nice Wiki article, and could be included on a LiveCD as a way of bootstrapping a CentOS install.
I'll have a go at that.
You can find the wikipage here: http://wiki.centos.org/HowTos/ManualInstall
Best regards,
Glenn
On Thu, Mar 26, 2009 at 5:42 PM, RedShift redshift@pandora.be wrote:
Since linux 2.6, the md layer has a feature called partitionable arrays. So instead of having two disks, creating an identical partition table on both and then putting those partitions in RAID 1, you take those two disks and put them in one partitionable RAID 1 array (in mdadm terms, "mdp") and create a partition table on the new RAID device. The advantages are quite clear compared to the old non-partitionable arrays.
For the uninitiated, would you be kind enough to elaborate the advantages of mdp?
I have always created identical partitions on the raw disks first, and the used mdadm on top. I also create my partitions ~200MB smaller than raw disk capacity to ensure minor size differences between disks (eg. 160GB HDD from Seagate is not exactly same size as a 160GB disk from Samsung) will not prevent me from adding them to a raid set.
Does mdp handle this scenario?
- Raja
Raja Subramanian wrote:
On Thu, Mar 26, 2009 at 5:42 PM, RedShift redshift@pandora.be wrote:
Since linux 2.6, the md layer has a feature called partitionable arrays. So instead of having two disks, creating an identical partition table on both and then putting those partitions in RAID 1, you take those two disks and put them in one partitionable RAID 1 array (in mdadm terms, "mdp") and create a partition table on the new RAID device. The advantages are quite clear compared to the old non-partitionable arrays.
For the uninitiated, would you be kind enough to elaborate the advantages of mdp?
I have always created identical partitions on the raw disks first, and the used mdadm on top. I also create my partitions ~200MB smaller than raw disk capacity to ensure minor size differences between disks (eg. 160GB HDD from Seagate is not exactly same size as a 160GB disk from Samsung) will not prevent me from adding them to a raid set.
Does mdp handle this scenario?
When you run a partitionable array you don't have to care about the partition table on the independent disks. So when a disk fails, you don't have to recreate the exact same partition table, you just swap the device using the mdadm tool.
Another advantage is that you don't have to do the disk swap for every partition. For example, in the old scenario, you have two disks (sda and sdb) with 4 partitions on them each. On those 4 partitions you create your RAID arrays, like md0=sda1,sdb1; md1=sda2,sdb2; and so forth. When sda fails, you have to remove the failed disk from all of the 4 RAID arrays and when you've put in the new disk, you have to signal all 4 arrays that the new disk is to be used. Not only do you have to execute 4 times as much commands, reconstruction of the 4 arrays will take place in parallel leading to slow disk access during reconstruction. When reconstructing a partitionable array using whole disks, reconstruction will always be sequential.
In the disks not being equal scenario: you can limit the size of the RAID array during creation with the -z parameter (man mdadm, chapter "For create, build, or grow:"). So instead of limiting the size of the partitions you create, you limit the whole size of the RAID array. So having a smaller replacement disk is no problem as long as it's equal or bigger than the array size you defined during creation.
Best regards,
Glenn Matthys
RedShift wrote: ...
Not only do you have to execute 4 times as much commands, reconstruction of the 4 arrays will take place in parallel leading to slow disk access during reconstruction.
Is this right?
When I have replaced a disk and added several partitions to an array, the rebuild is done one partition at a time.
The /proc/mdstat would say "delayed" on the partitiones waiting.
Mogens
on 3-26-2009 6:58 AM Mogens Kjaer spake the following:
RedShift wrote: ...
Not only do you have to execute 4 times as much commands, reconstruction of the 4 arrays will take place in parallel leading to slow disk access during reconstruction.
Is this right?
When I have replaced a disk and added several partitions to an array, the rebuild is done one partition at a time.
The /proc/mdstat would say "delayed" on the partitions waiting.
Mogens
That is what I remember also.
Mogens Kjaer wrote:
RedShift wrote: ...
Not only do you have to execute 4 times as much commands, reconstruction of the 4 arrays will take place in parallel leading to slow disk access during reconstruction.
Is this right?
When I have replaced a disk and added several partitions to an array, the rebuild is done one partition at a time.
The /proc/mdstat would say "delayed" on the partitiones waiting.
Mogens
I must be mistaken then, it's been a long time since I've used regular md devices.
Best regards,
Glenn Matthys
2009/3/27 RedShift redshift@pandora.be
Mogens Kjaer wrote:
RedShift wrote: ...
Not only do you have to execute 4 times as much commands, reconstruction of the 4 arrays will take place in parallel leading to slow disk access during reconstruction.
Is this right?
When I have replaced a disk and added several partitions to an array, the rebuild is done one partition at a time.
The /proc/mdstat would say "delayed" on the partitiones waiting.
Mogens
I must be mistaken then, it's been a long time since I've used regular md devices.
I can confirm this and furthermore, the default sync max transfer rate is very low for modern disks, so unless you've increased it to speed up sync or you have a very heavy disk workload, it's probably not going to impact normal disk access that much...
That said... I'd much prefer partitionable arrays from a management point of view... It's how all the hardware solutions work and they can't all be wrong ;)
d
D Tucny wrote:
> ... >> Not only do you >> have to execute 4 times as much commands, reconstruction of the 4 >> arrays will take place in parallel leading to slow disk access during >> reconstruction. > > Is this right? > > When I have replaced a disk and added several partitions > to an array, the rebuild is done one partition at a time. > > The /proc/mdstat would say "delayed" on the partitiones > waiting. > > Mogens I must be mistaken then, it's been a long time since I've used regular md devices.I can confirm this and furthermore, the default sync max transfer rate is very low for modern disks, so unless you've increased it to speed up sync or you have a very heavy disk workload, it's probably not going to impact normal disk access that much...
That said... I'd much prefer partitionable arrays from a management point of view... It's how all the hardware solutions work and they can't all be wrong ;)
And if someone is thinking of changing things, what would really be nice would be a default install on a single disk where you could add the mirror disk later and sync it in. Is that possible?
on 3-26-2009 12:50 PM D Tucny spake the following:
2009/3/27 RedShift <redshift@pandora.be mailto:redshift@pandora.be>
Mogens Kjaer wrote: > RedShift wrote: > ... >> Not only do you >> have to execute 4 times as much commands, reconstruction of the 4 >> arrays will take place in parallel leading to slow disk access during >> reconstruction. > > Is this right? > > When I have replaced a disk and added several partitions > to an array, the rebuild is done one partition at a time. > > The /proc/mdstat would say "delayed" on the partitiones > waiting. > > Mogens I must be mistaken then, it's been a long time since I've used regular md devices.I can confirm this and furthermore, the default sync max transfer rate is very low for modern disks, so unless you've increased it to speed up sync or you have a very heavy disk workload, it's probably not going to impact normal disk access that much...
That said... I'd much prefer partitionable arrays from a management point of view... It's how all the hardware solutions work and they can't all be wrong ;)
d
And it would make it easier for a single hot-spare to be available to several arrays that were configured differently.
-
D Tucny -
Les Mikesell -
Mogens Kjaer -
Phil Schaffner -
Raja Subramanian -
RedShift -
Robert Heller -
Ross Walker -
Scott Silva