On a random hunch or sheer desperation, I inserted an old brandless 1GB USB thumbdrive, installed and it booted.
Thinking that the Sandisk Ultra Backup 16GB was incompatible with CentOS/grub for some unknown reason. I switched to a brandless 16GB, installed the same way and it failed at grub prompt again.
Now it is starting to look as if either the newer drives have something on them that prevents booting. Or it's the total size of the drive, regardless of the partition size. I say this because in every case I used a single 1GB ext4 / partition for testing, to reduce the mkfs time.
Since now I have a bootable drive, I started to compare them with fdisk to see if there was any difference.
Bootable 255 heads, 63 sectors/track, 124 cylinders, start sector 63 With a warning that "Partition 1 has different physical/logical endings: phys=(123, 254, 63) logical=(124, 140, 16)"
Unbootable 16GB Sandisk Ultra Backup 54 heads, 48 sectors/track, 12158 cylinders, start sector 2048
Unbootable 8GB Sandisk Blade 20 heads, 4 sectors/track, 195417 cylinders, start sector 2048
Unbootable 16GB Unbranded 171 heads, 40 sectors/track, 4634 cylinders, start sector 2048
It seems that the common issue is starting at sector 2048 so I fdisk the 16GB Sandisk to manually create a 1GB partition started at sector 63. Using this, I installed and was able to boot successfully.
To verify, I use the same 16GB disk to install again, deleting the partition in anaconda and creating a new 1GB partition, ctrl-alt-f2 and fdisk confirms it was again at sector 2048.
As a side note, for some reason, drive geometry changed for the 16GB after a reinsert to varying values such as 6 heads, 33 sectors/track, 159164 cylinders 16 heads, 32 sectors/track, 61551 cylinders
Not sure if this had any significance to why anaconda decides to fdisk the drive starting from sector 2048.
My only guess is that it considers anything bigger than 4GB as a normal HDD drive and does this for hard disk alignment issue? But if grub could boot normally from such a HDD, why would it choke on a USB flash drive?
Should I report this as a bug with anaconda or ?