[CentOS] Strange behavior from software RAID

Sun Mar 3 02:35:38 UTC 2013
Harold Pritchett <harold at uga.edu>

Somewhere, mdadm is cacheing information.  Here is my /etc/mdadm.conf file:

more /etc/mdadm.conf
# mdadm.conf written out by anaconda
DEVICE partitions
MAILADDR root
ARRAY /dev/md0 level=raid1 num-devices=4 metadata=0.90 UUID=55ff58b2:0abb5bad:42911890:5950dfce
ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=315eaf5c:776c85bd:5fa8189c:68a99382
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=5b017f95:b7e266cc:f17a7611:8b752a02
ARRAY /dev/md3 level=raid1 num-devices=2 metadata=0.90 UUID=4cc310ee:60201e16:c7017bd4:9feea350
ARRAY /dev/md4 level=raid1 num-devices=2 metadata=0.90 UUID=ea205046:3c6e78c6:ab84faa4:0da53c7c

After a system re-boot, here is the contents of /proc/mdstat

# cat /proc/mdstat
Personalities : [raid1]
md125 : active raid1 sdc3[0]
       455482816 blocks [2/1] [U_]

md0 : active raid1 sdd1[3] sdc1[0] sdb1[1] sda1[2]
       1000320 blocks [4/4] [UUUU]

md127 : active raid1 sdd3[1] sdb3[0]
       971747648 blocks [2/2] [UU]

md3 : active raid1 sdf1[1] sde1[0]
       1003904 blocks [2/2] [UU]

md4 : active raid1 sdf3[1] sde3[0]
       1948491648 blocks [2/2] [UU]

md1 : active raid1 sda3[1]
       455482816 blocks [2/1] [_U]

unused devices: <none>

There are six physical disks in this system:

Disk /dev/sda:  500.1 GB, 500107862016 bytes
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
Disk /dev/sdc:  500.1 GB, 500107862016 bytes
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
Disk /dev/sde: 2000.3 GB, 2000398934016 bytes
Disk /dev/sdf: 2000.3 GB, 2000398934016 bytes

I used mdadm --examine /dev/sda1 to find the internal UUID for each of the physical volumes making up these volume groups

/dev/sda1:  Magic : a92b4efc  Version : 0.90.00  UUID : 55ff58b2:0abb5bad:42911890:5950dfce
/dev/sdb1:  Magic : a92b4efc  Version : 0.90.00  UUID : 55ff58b2:0abb5bad:42911890:5950dfce
/dev/sdc1:  Magic : a92b4efc  Version : 0.90.00  UUID : 55ff58b2:0abb5bad:42911890:5950dfce
/dev/sdd1:  Magic : a92b4efc  Version : 0.90.00  UUID : 55ff58b2:0abb5bad:42911890:5950dfce
/dev/sda3:  Magic : a92b4efc  Version : 0.90.00  UUID : 315eaf5c:776c85bd:5fa8189c:68a99382
/dev/sdc3:  Magic : a92b4efc  Version : 0.90.00  UUID : 315eaf5c:776c85bd:5fa8189c:68a99382
/dev/sdb3:  Magic : a92b4efc  Version : 0.90.00  UUID : 5b017f95:b7e266cc:f17a7611:8b752a02
/dev/sdd3:  Magic : a92b4efc  Version : 0.90.00  UUID : 5b017f95:b7e266cc:f17a7611:8b752a02
/dev/sde1:  Magic : a92b4efc  Version : 0.90.00  UUID : 4cc310ee:60201e16:c7017bd4:9feea350
/dev/sdf1:  Magic : a92b4efc  Version : 0.90.00  UUID : 4cc310ee:60201e16:c7017bd4:9feea350
/dev/sde3:  Magic : a92b4efc  Version : 0.90.00  UUID : ea205046:3c6e78c6:ab84faa4:0da53c7c
/dev/sdf3:  Magic : a92b4efc  Version : 0.90.00  UUID : ea205046:3c6e78c6:ab84faa4:0da53c7c

As you can see, the UUID on the various PVs match the values in the /etc/mdadm.conf file.

My question is What the heck is going on.  When I boot the system, I end up with two unexpected, unconfigured volume groups.  Where the heck are /dev/md125 and /dev/md127 coming 
from?  They don't appear in /etc/mdadm.conf and if I re-boot they keep coming back.  It appears that somewhere mdadm is keeping information.  How can I get rid of it so the 
mdadm.conf file is used.

Harold