bugGNU GRUB - Bugs: bug #35422, Out of memory. Aborted error when...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #35422: Out of memory. Aborted error when trying to boot with ServeRaid M1015 card

Submitter:  Thomas Fjellstrom <tomasu>
Submitted:  Wed 01 Feb 2012 04:59:17 AM UTC
   
 
Category:  Booting Severity:  Major
Priority:  5 - Normal Item Group:  Software Error
Status:  Fixed Privacy:  Public
Assigned to:  None Originator Name:  Thomas Fjellstrom
Open/Closed:  Closed Release:  other
Release:  Reproducibility:  Every Time
Planned Release:  None

Jump to the original submission

Fri 28 Feb 2014 09:11:09 AM UTC, comment #27: 

Given lack of feedback, I assume the patch worked.

Vladimir Serbinenko <phcoder>
Group administrator
Tue 04 Feb 2014 01:29:34 PM UTC, comment #26: 

Attached patch should workaround the problem, even though the real bug is in the Option rom of this RAID card.

(file #30456)

Vladimir Serbinenko <phcoder>
Group administrator
Sun 02 Feb 2014 05:14:11 AM UTC, comment #25: 

Sorry for the delay getting back to you, have been away and haven't had a chance to get to this until now.

It looks like that patch won't build correctly as your printf specifications include two parameters but you're only passing a single register in.

I have corrected the format spec and build/installed the code onto the boot drive for this machine.

Output from GRUB running this is as follows:

e820 returned eax=0x534d4150, ebx=0x0, ecx = 0x14, entry=0x0, 0x0, 0x0
out of memory

Picture of output attached for reference.


Will D <dowlingw>
Tue 07 Jan 2014 03:54:33 PM UTC, comment #24: 

Please run with attached patch to gather more data.

(file #30214)

Vladimir Serbinenko <phcoder>
Group administrator
Tue 07 Jan 2014 03:11:34 PM UTC, comment #23: 

Wow, this BIOS has no e820 with 3G of RAM. That would certainely get it into Top 10 stupid BIOS bugs. I'll prepare diagnostic/workaround patches.

Vladimir Serbinenko <phcoder>
Group administrator
Tue 07 Jan 2014 03:07:23 PM UTC, comment #22: 

Can replicate this issue on Ubuntu 13.10 (Grub 2.00-19ubuntu2.1).

Booting the system via SYSLINUX livecd works, have used this method to provide dmesg output as requested.

Have also provided card details below for reference.

Please let me know if there is any additional information that would be useful to troubleshoot this.


Basics :
======
Controller = 1
Model = ServeRAID M1015 SAS/SATA Controller
Current Controller Date/Time = 01/06/2014, 14:10:26
Current System Date/time = 01/06/2014, 14:10:28
SAS Address = 500605b0035788b0
Mfg Date = 04/02/11
Rework Date = 00/00/00
Revision No = 02C


Version :
=======
Firmware Package Build = 20.12.1-0150
Firmware Version = 2.130.394-2550
Bios Version = 4.36.00_4.12.05.00_0x05270000
NVDATA Version = 3.09.03-0052
Boot Block Version = 2.02.00.00-0001
Driver Name = megaraid_sas
Driver Version = 06.504.01.00-rc

(file #30210, file #30211)

Will D <dowlingw>
Tue 26 Mar 2013 04:56:34 PM UTC, comment #21: 

can you somehow extract e820 memory map when booted with this controller (it's in the beginning of dmesg and is available through "lsmmap"). If none of these method are applicable you need to add a printf in mm.c.

Vladimir Serbinenko <phcoder>
Group administrator
Tue 26 Mar 2013 04:35:25 PM UTC, comment #20: 

I completely retract my previous post. It seems that I somehow disabled the raid adapter in BIOS, that's what allowed me to boot. The bug is still present in Ubunut 13.04's grub2 (2.00-13ubuntu2) and it prevents me from booting.

FYI-- my boot disk is not attached to the raid adapter.

good time <l3iggs>
Tue 26 Mar 2013 04:13:07 AM UTC, comment #19: 

I just experienced this same exact bug under Ubuntu 12.10. I can confirm that it's resolved by updating grub using the 13.04 repo.

good time <l3iggs>
Sun 03 Jun 2012 05:21:12 PM UTC, comment #18: 

Well the patch like this can be applied to GRUB2 as well but the patch makes no sense. volatile means that the variable is accessed by external source (e.g. DMA) but this variable isn't. My guess is that asm helper wasn't conformant to ABI. If so it was fixed in 2.00~beta6. Can you confirm?

Vladimir Serbinenko <phcoder>
Group administrator
Sun 26 Feb 2012 06:23:02 PM UTC, comment #17: 

Hi, Someone just emailed me saying they managed to fix the problem for themselves, but using grub 0.97, not grub2. So I'm sure the fix isn't directly applicable, but it might be a similar issue?

See the bug: https://bugzilla.redhat.com/show_bug.cgi?id=250299

Thomas Fjellstrom <tomasu>
Sun 12 Feb 2012 10:53:20 PM UTC, comment #16: 

Ok, and just to make sure theres a RESULTS.txt that was generated when my custom compiled grub was installed, I'm attaching that now as well. (I think the previous two already had a proper grub re-installed, #3 deffinitely does, not sure about #2)

(file #25054)

Thomas Fjellstrom <tomasu>
Sun 12 Feb 2012 10:50:58 PM UTC, comment #15: 

If you mean the bootinfo script from sourceforge, I've attached two results, one from the machine booted into the rescue cd, with the sas card installed, and 7 drives plugged into it, and the other is a normal boot of the machine with debian's stock grub install, without the sas card installed.

Also, it seems I was wrong about the amount of memory that is seen when the card is installed and the machine boots off the system rescue cd. It only seems to see 3-4GB rather than the full 8. Though I could swear at least one time it saw the full 8. At any rate, with the card installed and setup the way I intend to use it, it only sees 4GB.

(file #25052, file #25053)

Thomas Fjellstrom <tomasu>
Sun 12 Feb 2012 09:57:36 PM UTC, comment #14: 

Can you supply the results of bootinfo script?

Vladimir Serbinenko <phcoder>
Group administrator
Sun 12 Feb 2012 09:52:19 PM UTC, comment #13: 

I appreciate the help so far.

I've played with it again some more, and it seems the syslinux loader on the system rescue cd seems to work just fine, and the system sees all 8GB ram after booting.

I have also not been able to get a manually compiled version of GRUB 0.99 to boot. Everything I've tried only gives me 'Symbol not found: grub_divmod64_full' error. I only had three drives installed, two of which I grub-install'ed to, and the third doesn't even have a boot sector (raw XFS filesystem directly on the drive). And I just tried unplugging all but one 500G drive that I grub-install'ed the new grub image to. Nothing works.

Thomas Fjellstrom <tomasu>
Sun 05 Feb 2012 06:01:58 AM UTC, comment #12: 

That is odd. The system had no problems booting with the same exact config before I customized grub and reinstalled it.

After playing with it a bit more, purging the customized grub, and reinstalling debian's grub (after having purged it to try a custom grub), the machine boots just fine. purging the debian grub, and reinstalling the custom grub dies with the divmod64 symbol error. I even removed the third disk I had installed, so the only two left are the two I had manually ran "update-grub" and "grub-install" on after installing grub each time.

Thomas Fjellstrom <tomasu>
Fri 03 Feb 2012 08:03:09 PM UTC, comment #11: 

Note that some BIOSes will skip disks if no partition is marked active. Just mark any partition active.

Vladimir Serbinenko <phcoder>
Group administrator
Fri 03 Feb 2012 05:21:14 PM UTC, comment #10: 

I'm about 99% sure I got them right, as they are Seagate Drives, and the sizes are in the drive identifier. Only three drives in there, two 500G, and one 640G.

I'll get the bootinfo info later today/tonight.

Thomas Fjellstrom <tomasu>
Fri 03 Feb 2012 04:42:00 PM UTC, comment #9: 

Can you run bootinfo script and post the results? Are you sure you don't confuse the disks in the boot order? The names in BIOS can be very unclear

Vladimir Serbinenko <phcoder>
Group administrator
Fri 03 Feb 2012 04:33:44 PM UTC, comment #8: 

It is possible, but I had those two drives first in the boot order. So the last drive booting would mean the boot sector on the first two weren't good. (the ones I installed the modified grub to)

Thomas Fjellstrom <tomasu>
Fri 03 Feb 2012 04:22:28 PM UTC, comment #7: 

It's common that system boots from unexpected drive, one that you think about as being "data only"

Vladimir Serbinenko <phcoder>
Group administrator
Fri 03 Feb 2012 02:48:50 PM UTC, comment #6: 

Thats what I had thought, but I re-ran it on both of the main drives (sda and sdb, which make up the raid1 for /).

Thomas Fjellstrom <tomasu>
Fri 03 Feb 2012 09:05:21 AM UTC, comment #5: 

You have run grub-install on wrong device (not the one you actually boot from)

Vladimir Serbinenko <phcoder>
Group administrator
Fri 03 Feb 2012 06:53:02 AM UTC, comment #4: 

So I tried that, but I had to have done something wrong, as when I tried to boot, I get a 'grub_divmod64_full symbol not found' type error. I had assumed it was because I installed the new grub over the distro grub, so I tried again after purging the distro grub, and still no love. I did re-run grub-install between each attempt. I'll try again in a day or so, and I might try flashing the M1015 firmware to a different one that just does pass through, it may also fix the problem.

Thomas Fjellstrom <tomasu>
Wed 01 Feb 2012 04:37:37 PM UTC, comment #3: 

I think the bug is in int13h handler but it's more of a guess. I'd start with 0xbfffffff (which basically says "don't touch outside 3GiB")

Vladimir Serbinenko <phcoder>
Group administrator
Wed 01 Feb 2012 04:29:08 PM UTC, comment #2: 

I will try that later today. What would you suggest for a value? 0xefffffff? Or something much smaller like 0xffffefff?

If it helps, the board has the option to hook the boot disk interrupt disabled. When you say "when accessed through BIOS" do you mean when the bios config is started? Or just when the card itself initializes?

Thomas Fjellstrom <tomasu>
Wed 01 Feb 2012 03:43:36 PM UTC, comment #1: 

The card in question is known corrupt memory when access through BIOS (probably a buggy option ROM). Pretty difficult to workaround without having an idea what's going on and which regions are corrupted and which are safe. Can you try decreasing the 0xffffffff in the file
grub-core/kern/i386/pc/init.c, lines 206 and 211?

Vladimir Serbinenko <phcoder>
Group administrator
Wed 01 Feb 2012 04:59:17 AM UTC, original submission:  

Grub will fail to load when a ServeRaid M1015 (aka: LSI MegaRaid 9220) PCIe card is installed. Without the card installed, grub loads, and boots the machine just fine.

The GRUB output is as follows:


GRUB Loading.
Welcome to GRUB!   (this line is black on white)
                   (as is this one)
out of memory      (text is white on black, rest of line is black on white)
Aborted. Press any key to exit.


This is fully reproducible.

The machine has GRUB 1.99-14 installed (debian sid).

Installing grub with `grub-install --debug-image=all /dev/sda` did not give any more output than it did with the stock image.

Added detail:
Machine has a Gigabyte MA790FXT-UD5P motherboard, Phenom II x4 810 cpu, 8GB DDR3 ram, dinky fanless gpu, extra 10/100 NIC, and of course the SAS controller, with 7 drives hooked up to it. System drive is /not/ on the SAS controller. Root filesystem lives on ext4, over top LVM2, over md raid1, over two Seagate 500G drives connected to the motherboard's built in SATA ports.

Please let me know if I've left anything out.

Thomas Fjellstrom <tomasu>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #30456:  e820.diff added by phcoder (3KiB - text/x-diff)
file #30214:  mmap.diff added by phcoder (1KiB - text/x-diff)
file #30210:  dowlingw-dmesg.txt added by dowlingw (50KiB - text/plain)
file #30211:  dowlingw-dmidecode.txt added by dowlingw (11KiB - text/plain)
file #25054:  RESULTS.txt added by tomasu (12KiB - text/plain)
file #25052:  RESULTS2.txt added by tomasu (20KiB - text/plain)
file #25053:  RESULTS3.txt added by tomasu (9KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dowlingw (Updated the item)
  • -email is unavailable- added by l3iggs (Posted a comment)
  • -email is unavailable- added by phcoder (Posted a comment)
  • -email is unavailable- added by tomasu (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-02-28 phcoder StatusNone Fixed
        Open/ClosedOpen Closed
    2014-02-04 phcoder Attached File- Added e820.diff, #30456
    2014-02-02 dowlingw Attached File- Added bug35422-patchoutput.png, #30433
    2014-01-07 phcoder Attached File- Added mmap.diff, #30214
    2014-01-07 dowlingw Attached File- Added dowlingw-dmesg.txt, #30210
        Attached File- Added dowlingw-dmidecode.txt, #30211
    2012-02-12 tomasu Attached File- Added RESULTS.txt, #25054
    2012-02-12 tomasu Attached File- Added RESULTS2.txt, #25052
        Attached File- Added RESULTS3.txt, #25053

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code