bugGNU GRUB - Bugs: bug #46014, XEN domU crash when PV grub...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #46014: XEN domU crash when PV grub chainloads 32-bit domU grub

Submitter:  Andreas Sundstrom <sunkan>
Submitted:  Mon 21 Sep 2015 07:48:56 PM UTC
Votes: 1
 
Category:  Booting Severity:  Major
Priority:  5 - Normal Item Group:  Software Error
Status:  None Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open Release:  other
Release:  Reproducibility:  Intermittent
Planned Release:  2.02

Jump to the original submission

Thu 09 Mar 2017 10:26:36 PM UTC, comment #11: 

Much better, thank you. I took commit bcf3c55531349dfec135cad9bf359960214dd0f6 and applied it on top of the version mentioned in comment #9. The resulting dom0 image is now able to chainload the guest kernel as desired.

I also cherrypicked the same commit into the Debian package (version 2.02~beta2-22+deb8u1, which is the one mentioned in the original submission with the fix for CVE-2015-8370 on top). Replacing grub-i386-xen.bin in dom0 with the patched version was enough to cure the problem.

Sergio Gelato <gelato>
Sun 05 Mar 2017 09:33:40 AM UTC, comment #10: 

I committed a fix. Please try again

Vladimir Serbinenko <phcoder>
Group administrator
Fri 10 Feb 2017 07:42:23 PM UTC, comment #9: 

Tested git master at 529818738fb2aafb05721830dc99cc6c8736a048. Same outcome as before, i.e. chainloading fails, configfile fallback succeeds. (I expected this as the Debian patches seemed orthogonal to this issue, but of course it's better to confirm it. My initial gambit was because I had hoped to find that the bug was fixed, in which case the packaging would have been of immediate use to me.)

Binaries and corresponding source code (from "git archive master") at https://ttt.astro.su.se/~gelato/6bxIjsee/ .

Sergio Gelato <gelato>
Thu 09 Feb 2017 07:51:28 PM UTC, comment #8: 


> debianized it


This is upstream bug tracker. We cannot usually know what patches distributions add. Please test clean upstream build, without any addon patches.

Andrei Borzenkov <arvidjaar>
Group Member
Thu 09 Feb 2017 07:45:52 PM UTC, comment #7: 

I downloaded grub-2.02~rc1.tar.xz, debianized it based on grub2_2.02~beta3-4.debian.tar.xz (two patches dropped as they are included in rc1), and applied commit 8a96fc409373fffda99e18a8438ddfc6bb787bc8 on top of it all. I did not include the "xen: Fix parsing of XZ kernel" patch; I take your question about git master to mean that I should. Built .deb packages using pbuilder and the Debian "stretch" repositories; the source and binary packages, and the build log, are all available at https://ttt.astro.su.se/~gelato/noIo7WNwqA/ . The guest has grub-common, grub2-common, grub-xen and grub-xen-bin installed, then "grub-install --target=i386-xen /dev/xvda" was run.

I had initially updated only the guest's GRUB but have now tested with the new grub-i386-xen.bin (same build as above, package grub-xen-host) on the dom0 side; no difference in behaviour when booting a guest with /boot/xen/pvboot-i386.elf.

(Aside: there are differences in behaviour in circumstances beyond the scope of this bug; e.g., the new grub-i386-xen.bin doesn't like Debian 8 guests built with grub-install --target=i386-pc, complains about the initrd; maybe the old guest grub.cfg isn't quite compatible?)

The host runs Debian 8.7 "jessie" and its Xen 4.4.1-9+deb8u8 packages. x86_64 of course: this version of Xen does not support 32-bit dom0.

Sergio Gelato <gelato>
Wed 08 Feb 2017 09:57:05 PM UTC, comment #6: 

Did you take branch or applied patch from ML (not from branch) ? Are both GRUBs updated? Does git master work? What is xen version? Is host 32 or 64-bit? I CNR on master. Can you send your binaries?

Vladimir Serbinenko <phcoder>
Group administrator
Wed 08 Feb 2017 08:58:56 PM UTC, comment #5: 

I regret to report that the patch referred to by comment #3, when applied by itself on top of 2.02-rc1, doesn't fix this bug for me.
I went to some length to purge previous versions of GRUB from the PV guest, in an attempt to avoid further red herrings. Yet the trace information from "xl dmesg" is unaffected.

Sergio Gelato <gelato>
Tue 07 Feb 2017 03:21:49 PM UTC, comment #4: 

Yes, my comment #2 was wrong. On closer examination the change simply causes pvboot-i386.elf to fail and the dom0 to fall back to searching for grub.cfg directly. Sorry for the noise. Will try your patch.

Sergio Gelato <gelato>
Tue 07 Feb 2017 08:36:25 AM UTC, comment #3: 

I believe this is red herring. Please try the patch that I've posted on grub-devel mailing list (https://lists.gnu.org/archive/html/grub-devel/2017-02/msg00031.html) or the branch xenfixes (http://git.savannah.gnu.org/cgit/grub.git/commit/?h=phcoder/xenfixes )

Vladimir Serbinenko <phcoder>
Group administrator
Tue 07 Feb 2017 07:40:19 AM UTC, comment #2: 

I think I've found a solution. If I change Xen ELF note 9 (PAE_MODEL) from the current "yes" to "no", the resulting image boots successfully all the way into Linux (i686-pae).

The attached patch should take care of this problem, and also of #50237 as a side effect.

Disclaimer 1: I haven't compiled it yet; for my testing I found it faster to patch the binary image with a hex editor.

Disclaimer 2: my testing has only limited coverage, and I'm not 100% confident that pvboot-i386.elf will not need PAE_MODEL=yes under any circumstances. Maybe one will need to add PAE support to grub-core/kern/i386/xen/startup.S after all.

(file #39687)

Sergio Gelato <gelato>
Mon 06 Feb 2017 09:52:37 AM UTC, comment #1: 

I can reproduce this, so far 100% of the time, with GRUB 2.02 beta 3. Only when booting a 32-bit PV guest; 64-bit guests boot fine.

Same symptoms: an unhandled page fault fetching instructions from address 0 (which if I've understood correctly is the entry point specified by the Xen ELF notes set by grub-mkimage).

Sergio Gelato <gelato>
Mon 21 Sep 2015 07:48:56 PM UTC, original submission:  

This is using Debian Jessie and grub 2.02~beta2-22 (with Debian patches
applied) and Xen 4.4.1

I originally posted a bug report with Debian but got the suggestion to
file bugs with upstream as well.
Debian bug report:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799480

Here's the information from my original bug report:

Using 64-bit dom0 and 32-bit domU PV (para-virtualized) grub sometimes
fail when chainloading the domU's grub. 64-bit domU seem to work 100%
of the time.

My understanding of the process:

  • dom0 launches domU with grub that is loaded from dom0's disk.
  • Grub reads config file from memdisk, and then looks for grub binary

  in domU filesystem.

  • If grub is found in domU it then chainloads (multiboot) that grub

  binary and the domU grub reads grub.cfg and continue booting.

  • If grub is not found in domU it reads grub.cfg and continues with

  boot.

It fails at step 3 in my list of the boot process, but sometimes it
does work so it may be something like a race condition that causes the
problem?

A workaround is to not install or rename /boot/xen in domU so that the
first grub that is loaded from dom0's disk will not find the grub
binary in the domU filesystem and hence continues to read grub.cfg and
boot. The drawback of this is of course that the two versions can't
differ too much as there are different setups creating grub.cfg and
then reading/parsing it at boot time.

I am not sure at this point whether this is a problem in XEN or a
problem in grub but I compiled the legacy pvgrub that uses some minios
from XEN (don't really know much more about it) and when that legacy
pvgrub chainloads the domU grub it seems to work 100% of the time. Now
the legace pvgrub is not a real alternative as it's not packaged for
Debian though.

When it fails "xl create vm -c" outputs this:
Parsing config from /etc/xen/vm
libxl: error: libxl_dom.c:35:libxl__domain_type: unable to get domain type for domid=16
Unable to attach console
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console child [0] exited with error status 1

And "xl dmesg" shows errors like this:
(XEN) traps.c:2514:d15 Domain attempted WRMSR 00000000c0010201 from 0x0000000000000000 to 0x000000000000ffff.
(XEN) d16:v0: unhandled page fault (ec=0010)
(XEN) Pagetable walk from 0000000000000000:
(XEN)  L4[0x000] = 0000000200256027 000000000000049c
(XEN)  L3[0x000] = 0000000200255027 000000000000049d
(XEN)  L2[0x000] = 0000000200251023 00000000000004a1
(XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08021feb0 compat_create_bounce_frame+0xc6/0xde
(XEN) Domain 16 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.4.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e019:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: 0000000000499000   rdi: 0000000000800000
(XEN) rbp: 000000000000000a   rsp: 00000000005a5ff0   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: ffff83023e9b9000   r11: ffff83023e9b9000
(XEN) r12: 0000033f3d335bfb   r13: ffff82d080300800   r14: ffff82d0802ea940
(XEN) r15: ffff83005e819000   cr0: 000000008005003b   cr4: 00000000000506f0
(XEN) cr3: 0000000200b7a000   cr2: 0000000000000000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=005a5ff0:
(XEN)   00000010 00000000 0001e019 00010046 0016b38b 0016b38a 0016b389 0016b388
(XEN)   0016b387 0016b386 0016b385 0016b384 0016b383 0016b382 0016b381 0016b380
(XEN)   0016b37f 0016b37e 0016b37d 0016b37c 0016b37b 0016b37a 0016b379 0016b378
(XEN)   0016b377 0016b376 0016b375 0016b374 0016b373 0016b372 0016b371 0016b370
(XEN)   0016b36f 0016b36e 0016b36d 0016b36c 0016b36b 0016b36a 0016b369 0016b368
(XEN)   0016b367 0016b366 0016b365 0016b364 0016b363 0016b362 0016b361 0016b360
(XEN)   0016b35f 0016b35e 0016b35d 0016b35c 0016b35b 0016b35a 0016b359 0016b358
(XEN)   0016b357 0016b356 0016b355 0016b354 0016b353 0016b352 0016b351 0016b350
(XEN)   0016b34f 0016b34e 0016b34d 0016b34c 0016b34b 0016b34a 0016b349 0016b348
(XEN)   0016b347 0016b346 0016b345 0016b344 0016b343 0016b342 0016b341 0016b340
(XEN)   0016b33f 0016b33e 0016b33d 0016b33c 0016b33b 0016b33a 0016b339 0016b338
(XEN)   0016b337 0016b336 0016b335 0016b334 0016b333 0016b332 0016b331 0016b330
(XEN)   0016b32f 0016b32e 0016b32d 0016b32c 0016b32b 0016b32a 0016b329 0016b328
(XEN)   0016b327 0016b326 0016b325 0016b324 0016b323 0016b322 0016b321 0016b320
(XEN)   0016b31f 0016b31e 0016b31d 0016b31c 0016b31b 0016b31a 0016b319 0016b318
(XEN)   0016b317 0016b316 0016b315 0016b314 0016b313 0016b312 0016b311 0016b310
(XEN)   0016b30f 0016b30e 0016b30d 0016b30c 0016b30b 0016b30a 0016b309 0016b308
(XEN)   0016b307 0016b306 0016b305 0016b304 0016b303 0016b302 0016b301 0016b300
(XEN)   0016b2ff 0016b2fe 0016b2fd 0016b2fc 0016b2fb 0016b2fa 0016b2f9 0016b2f8
(XEN)   0016b2f7 0016b2f6 0016b2f5 0016b2f4 0016b2f3 0016b2f2 0016b2f1 0016b2f0

An easy way to find out which grub you are in if the machine boots is
to hit 'c' and type 'ls', only the grub from dom0 will know about
(memdisk). So when trying to replicate the issue (and the domU
actually starts) you can hit 'c', type 'ls' (check for memdisk) and
then type 'halt' and relaunch the domU. Usually I can't launch more
than 4-5 times in a row before it fails, often it fails on my first
try.

For information I have reproduced on two different AMD desktop
processor machines, not sure if Intel would be any different.  I'm
pretty sure I did tests with grub from unstable with same result at
some point, but can test again if that is likely to work.

The package that is in installed on the domU side is "grub-xen".

I am unable to understand how to debug grub further on my own, I have
printed out text from grub so that I understood that it is the
chainload that fails. I see no output from the domU grub (except when
it works as it should of course). I can help with further testing if
needed.

/Andreas

Andreas Sundstrom <sunkan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by arvidjaar (Posted a comment)
  • -email is unavailable- added by gelato (Posted a comment)
  • -email is unavailable- added by gelato (Voted in favor of this item)
  • -email is unavailable- added by phcoder (Updated the item)
  • -email is unavailable- added by sunkan (Submitted the item)
  •  

    There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-02-07 gelato Attached File- Added 0001-Xen-i386-PV-guest-bootloader-does-NOT-support-PAE.patch, #39687
    2017-02-06 gelato Carbon-Copy- Added gelato
    2016-03-02 phcoder Planned ReleaseNone 2.02

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code