I think I recently got bit by this as well.  I have 3 disks with part in a 3 way raid10 for my root fs, and the rest in a raid5 for media storage.  After some updates, I rebooted and got all kinds of weird errors trying to load any kernel.  I booted from a liveusb and poked around and it turned out that one of the disks had been marked as failed in the raid10 for some reason a month or two before and I hadn't noticed.  After re-adding the disk to the array and resyncing it, I was able to boot again.  GRUB really needs to pay attention to the failed flags in the most recently updated metadata block.

Hi,

I haven't yet attempted to reproduce this as it occurred on a production system that I needed to get up as soon as I could, but when it was an issue, it was fully repeatable.

Scenario:

  1. /boot stored on a four disk MD RAID-1 array.
  2. Two disks had failed simultaneously.
  3. Upon GRUB boot, GRUB was assembling the two failed disks (which comprised a valid but somewhat old filesystem) instead of the two more recently updated disks.

My best guess is that GRUB isn't taking account the relevant timestamps on the RAID members which should specify when they were last updated.

It should be possible to reproduce this by failing two md devices simultaneously and then getting GRUB to use it as its filesystem. I'm not sure which two would need to fail (e.g. does GRUB just use the devices with the lowest numbers?) but I think some simple checking of timestamps would be required.