bugGNU GRUB - Bugs: bug #52657, grub-mkconfig does not guarantee...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #52657: grub-mkconfig does not guarantee changes are on disk before exiting

Submitter:  Chris Murphy <chrismurphy>
Submitted:  Wed 13 Dec 2017 11:58:08 AM UTC
   
 
Category:  Installation Severity:  Major
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open Release:  2.02~rc1
Release:  Reproducibility:  Intermittent
Planned Release:  None

Wed 13 Dec 2017 11:58:08 AM UTC, original submission:  

Summary:
grub-mkconfig does not call sync() or FIFREEZE() to ensure data and fs metadata are completely committed to disk before exiting. As a result if the system is uncleanly rebooted, the system might not be bootable at all due to missing or zero byte length grub.cfg.

Reproducibility: Intermittant and somewhat rare.

Example:

So the example I have is actually with grubby doing the modification of grub.cfg. I understand grubby is not at all related to GRUB but it does a similar read, modify in memory, write a new file, delete old file, move/rename new to old. [1] And I have a roughly 8 out of 10 failure rate in a particular setup:

a. /boot/grub/grub.cfg is on XFS which uses delayed allocation and journal
b. plymouth is exempting itself from being killed by systemd at reboot
c. due to b. systemd fails to remount read-only /boot/grub and eventually does a force reboot

Result is a grub> prompt. Inspection of the file system without journal replay shows zero byte grub.cfg or sometimes missing grub.cfg entirely. However if the journal is replayed, the file system metadata is fixed up, and now GRUB works again.

grub2-mkconfig does not call either sync() [2]  or FIFREEZE() [3] after writing out and renaming the new grub.cfg. I suspect that the grub-mkconfig case will run into the same problem as grubby, even though I haven't reproduced tried to reproduce it yet.

XFS devs have explicitly said both grubby and grub-mkconfig are responsible for fully committing their bootloader configuration changes to disk before exiting in order to ensure the new configuration is available should an unclean reboot happen immediately after bootloader configuration changes.[4] GRUB can't replay the journal, therefore it's not sufficient for grub-mkconfig to depend on XFS journaling to save things in this case.


[1] grubby bug already filed.
https://github.com/rhboot/grubby/issues/25

[2] applies to non-journaled file systems)

[3] applies to journaled file systems were sync is only guaranteed to write out metadata to the journal not complete file system metadata commit

[4] (note that this patch starts the conversation of the problem, the patch is ultimately not being applied in the kernel, and later in that long long thread they also are fairly certain the exact same problem happens on ext4. Btrfs is probably going to manifest with the old grub.cfg which might contain a stale kernel entry and thus still fail to boot.
https://www.spinics.net/lists/linux-xfs/msg06883.html


Chris Murphy <chrismurphy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by herrold
  • -email is unavailable- added by chrismurphy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-22 herrold Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code