bugGNU GRUB - Bugs: bug #64471, grub efi memory allocation...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #64471: grub efi memory allocation (efi/mm) does not work on every machine

Submitter:  Alexey Kuznetsov <axet>
Submitted:  Tue 25 Jul 2023 06:04:05 PM UTC
   
 
Category:  Booting Severity:  Major
Priority:  5 - Normal Item Group:  Software Error
Status:  None Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open Release:  other
Release:  Reproducibility:  Every Time
Planned Release:  None

Jump to the original submission

Fri 28 Jul 2023 01:33:42 PM UTC, comment #24: 

One thing to ask? Did grub has any code related to reading / parsing paging tables? Looks like it dosn't.

How to read physical memory? Is here grub_ method for that? Or should I switch to some protected mode back and forth using asm code?

 Alexey Kuznetsov <axet>
Fri 28 Jul 2023 01:31:50 PM UTC, comment #23: 

What I understand from Paging is that I had to request memory from efi using allocate_pages then check if pages table contains this virtual address, so it was mapped correctly by the efi.

page table located at physical address pointed by part of CR3 register (12-32+ bits) then parse the table and make sure virtual adress is there. If not, free efi requested memory and return out of memory error.

Which is raise more even questions...

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 08:28:58 PM UTC, comment #22: 

Ok. I simplified the patch, now it does not change grub_efi_allocate_pages_real. Wild guess does it fix a) issue, since grub_efi_mm_add_regions only affecting grub_malloc?

I'm going to bed, I'll check paging later..

(file #54987)

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 08:20:51 PM UTC, comment #21: 


commentaire #20 :

> Ok I understand now.
>
> a) I guess you didn't mention that allocating disk buffers from above 4GB can corrupt the data? Otherwise why not let it fail during to disk operation if it going to fail due to out of memory error?


I'm concerned it might return garbage or corrupt data silently. Also even if no big buffers are available below 4 GiB we can read parts in sub-4GiB temporary bug and then copy

>
> b) No idea what is paging map, I could not find it.


https://wiki.osdev.org/Paging

 Vladimir Serbinenko <phcoder>
Group administrator
Thu 27 Jul 2023 08:03:16 PM UTC, comment #20: 

Ok I understand now.

a) I guess you didn't mention that allocating disk buffers from above 4GB can corrupt the data? Otherwise why not let it fail during to disk operation if it going to fail due to out of memory error?

b) No idea what is paging map, I could not find it.

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 07:37:52 PM UTC, comment #19: 


commentaire #18 :

>
> comment #17:
> >
> > commentaire #16 :
> > >
> > > comment #15:
> > > > Where do you check the page table?
> > >
> > > I do not understand. Can you view the smartmem.patch?
> >
> > Yes, I've seen it. And you need to check that memory above 4GiB is mapped at all before you can use them. For this you need to read cr3 and parse page mapping and if needed error out or add missing mapping.
> > Additionally buffers for disk reads and other hardware-related tasks need to be under 4 GiB.
>
> You are saying that I'm missing something here. But I do not understand it. I'm not using any low level functions which can allocate bad regions, I call grub_efi_allocate_any_pages which using EFI allocate_pages method, which is accepting maximum memory address as input argument and requested pages as second option, so I did. I ask to allocate any pages from all pages range. And do EFI to care about looking for usable address and return the allocated address. So it works!


Nope some firmwares correctly add available memory above 4GiB to memory map but fail on either of 2 bugs:

  • Their disk routines fail with memory above 4 GiB
  • They don't map regions above 4 GiBs contrary to the spec


You need to avoid both bugs by:
(a) disk routines should use memory under 4GiB either always or after failing
(b) you need to check paging map.

Ideally neither would be necessary but due to firmware bugs they are

 Vladimir Serbinenko <phcoder>
Group administrator
Thu 27 Jul 2023 06:45:41 PM UTC, comment #18: 


comment #17:

>
> commentaire #16 :
> >
> > comment #15:
> > > Where do you check the page table?
> >
> > I do not understand. Can you view the smartmem.patch?
>
> Yes, I've seen it. And you need to check that memory above 4GiB is mapped at all before you can use them. For this you need to read cr3 and parse page mapping and if needed error out or add missing mapping.
> Additionally buffers for disk reads and other hardware-related tasks need to be under 4 GiB.


You are saying that I'm missing something here. But I do not understand it. I'm not using any low level functions which can allocate bad regions, I call grub_efi_allocate_any_pages which using EFI allocate_pages method, which is accepting maximum memory address as input argument and requested pages as second option, so I did. I ask to allocate any pages from all pages range. And do EFI to care about looking for usable address and return the allocated address. So it works! Here is already grub function for filter_memory_map for looking for appropriate ranges and checks and ONLY WHEN it failed, I ask EFI to solve it for me from all possible ranges...

I already test smartmem patch for two machines: Desktop amd64 (intel) which failed WITHOUT bigmem patch (4GB+ below and above ranges) and laptop amd64 (amd) which failed WITH bigmem patch (only <4GB works). Right now I'm testing old laptop x32 (intel) with all my patches.

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 06:28:52 PM UTC, comment #17: 


commentaire #16 :

>
> comment #15:
> > Where do you check the page table?
>
> I do not understand. Can you view the smartmem.patch?


Yes, I've seen it. And you need to check that memory above 4GiB is mapped at all before you can use them. For this you need to read cr3 and parse page mapping and if needed error out or add missing mapping.
Additionally buffers for disk reads and other hardware-related tasks need to be under 4 GiB.

 Vladimir Serbinenko <phcoder>
Group administrator
Thu 27 Jul 2023 06:20:29 PM UTC, comment #16: 


comment #15:

> Where do you check the page table?


I do not understand. Can you view the smartmem.patch?

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 05:57:27 PM UTC, comment #15: 

Where do you check the page table?

 Vladimir Serbinenko <phcoder>
Group administrator
Thu 27 Jul 2023 05:35:17 PM UTC, comment #14: 

Desktop e820 with 32GB output.


dmesg | awk '/usable/ && /BIOS-e820/ && match($0, /mem ([0-9a-z]+)-([0-9a-z]+)/,aa){s=strtonum(aa[1]);e=strtonum(aa[2]);if(e-s>1*1024*1024*1024)printf("0x%x-0x%x %dGB\n",s,e,((e-s)/1024/1024/1024))}'
0x100000000-0x8bfffffff 30GB



2023-07-16T11:19:50.846357+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
2023-07-16T11:19:50.846359+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
2023-07-16T11:19:50.846359+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
2023-07-16T11:19:50.846360+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
2023-07-16T11:19:50.846360+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000035f37fff] usable
2023-07-16T11:19:50.846361+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000035f38000-0x0000000039037fff] reserved
2023-07-16T11:19:50.846362+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000039038000-0x00000000392b0fff] ACPI data
2023-07-16T11:19:50.846363+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000392b1000-0x0000000039370fff] ACPI NVS
2023-07-16T11:19:50.846364+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000039371000-0x0000000039f65fff] reserved
2023-07-16T11:19:50.846364+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000039f66000-0x0000000039ffefff] type 20
2023-07-16T11:19:50.846365+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000039fff000-0x0000000039ffffff] usable
2023-07-16T11:19:50.846365+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x000000003a000000-0x000000003dffffff] reserved
2023-07-16T11:19:50.846366+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x000000003ee00000-0x000000003fffffff] reserved
2023-07-16T11:19:50.846368+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
2023-07-16T11:19:50.846368+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000fc00ffff] reserved
2023-07-16T11:19:50.846368+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
2023-07-16T11:19:50.846369+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
2023-07-16T11:19:50.846369+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
2023-07-16T11:19:50.846370+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
2023-07-16T11:19:50.846370+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
2023-07-16T11:19:50.846382+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
2023-07-16T11:19:50.846387+03:00 axet-desktop kernel: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000008bfffffff] usable

Alexey Kuznetsov <axet>
Thu 27 Jul 2023 05:05:00 PM UTC, comment #13: 

I made it work by adding smart memory patch as I described below in comment 9. Here is a code. It working!

https://gitlab.com/axet/homebin/-/blob/debian/dbuild.d/bookworm/grub2/smartmem.patch

(file #54986)

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 04:52:19 PM UTC, comment #12: 

When I said "no patches" I mean grub 2.06+argon2 patches+new grub2.12 memory patches. no bigmem (4GB+) patches.

 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 04:37:16 PM UTC, comment #11: 

Without any patches (<4GB usage range). On desktop Grub corrupt screen when trying to allocate a lot of memory. Looks like it touching the regions of memory it should not touch. Photo attached.


 Alexey Kuznetsov <axet>
Thu 27 Jul 2023 04:08:25 PM UTC, comment #10: 

I had to install 4GB into my machine here is a dmesg for 4GB.

Almost the same, only 1GB+ at 0x100000000



dmesg | awk '/usable/ && /BIOS-e820/ && match($0, /mem ([0-9a-z]+)-([0-9a-z]+)/,aa){s=strtonum(aa[1]);e=strtonum(aa[2]);if(e-s>1.5*1024*1024*1024)printf("0x%x-0x%x %dGB\n",s,e,((e-s)/1024/1024/1024))}'
0x100000000-0x1bfffffff 2GB




[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000035f3bfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000035f3c000-0x000000003903bfff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003903c000-0x00000000392aefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000392af000-0x000000003936efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000003936f000-0x0000000039f65fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000039f66000-0x0000000039ffefff] type 20
[    0.000000] BIOS-e820: [mem 0x0000000039fff000-0x0000000039ffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003a000000-0x000000003dffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003ee00000-0x000000003fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000fc00ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000001bfffffff] usable

Alexey Kuznetsov <axet>
Wed 26 Jul 2023 02:51:52 PM UTC, comment #9: 

I guess we need smi-automatic GRUB_EFI_MAX_USABLE_ADDRESS variable.

If we have enough memory from low memory segment (<4GB), then use it. If we have no memory and grub about to return out of memory error, instead crashing, why not to try allocate from upper address region (4GB+) and lets hope for the best?

Basically we need new allocation algorithm, filter_memory_map should not filter based on address, and grub_efi_allocate_pages_real should be smart to allocate from different regions low and high.

 Alexey Kuznetsov <axet>
Wed 26 Jul 2023 12:44:46 PM UTC, comment #8: 

base_addr = 0x0, length = 0x92000, available RAM
base_addr = 0x92000, length = 0xc000, available RAM
base_addr = 0x9e000, length = 0x1000, reserved RAM
base_addr = 0x9f000, length = 0x1000, available RAM
base_addr = 0x100000, length = 0x1a210000, available RAM
base_addr = 0x1a310000, length = 0x80000, available RAM
base_addr = 0x1a390000, length = 0x14e74000, available RAM
base_addr = 0x2f204000, length = 0x403000, available RAM
base_addr = 0x2f607000, length = 0x78000, available RAM
base_addr = 0x2f67f000, length = 0x23000, available RAM
base_addr = 0x2f6a2000, length = 0x5000, available RAM
base_addr = 0x2f6a7000, length = 0xd000, available RAM
base_addr = 0x2f6b4000, length = 0x16ce000, available RAM
base_addr = 0x30d62000, length = 0x1c000, available RAM
base_addr = 0x30d9e000, length = 0x2c000, available RAM
base_addr = 0x30dca000, length = 0x1000, available RAM
base_addr = 0x30dcb000, length = 0x5000, available RAM
base_addr = 0x30dd0000, length = 0x2000, available RAM
base_addr = 0x30dd2000, length = 0x5000, available RAM
base_addr = 0x30dd7000, length = 0x4000, available RAM
base_addr = 0x30ddb000, length = 0x8000, available RAM
base_addr = 0x30de5000, length = 0x2000, available RAM
base_addr = 0x30de5000, length = 0x6000, available RAM
base_addr = 0x30deb000, length = 0x1000, available RAM
base_addr = 0x30dec000, length = 0x13000, available RAM
base_addr = 0x30dff000, length = 0x4000, available RAM
base_addr = 0x30e03000, length = 0x8000, available RAM
base_addr = 0x30e0b000, length = 0x8000, available RAM
base_addr = 0x30e13000, length = 0x18000, available RAM
base_addr = 0x30e2b000, length = 0x6000, available RAM
base_addr = 0x30e31000 length = 0x10000 available RAM
base_addr = 0x30e41000 length = 0x46000 available RAM
base_addr = 0x30e87000 length = 0x3f000 available RAM
base_addr = 0x30ec6000 length = 0x6000 available RAM
base_addr = 0x30ecc000 length = 0x96000 available RAM
base_addr = 0x30f62000 length = 0x7000 available RAM
base_addr = 0x30f69000 length = 0xb000 available RAM
base_addr = 0x30f74000 length = 0x5000 available RAM
base_addr = 0x30f79000 length = 0x9000 available RAM
base_addr = 0x30f82000 length = 0x1000 available RAM
base_addr = 0x30f83000 length = 0x11000 available RAM
base_addr = 0x30f94000 length = 0x3000 available RAM
base_addr = 0x30f97000 length = 0x13000 available RAM
base_addr = 0x30faa000 length = 0x1000 available RAM
base_addr = 0x30fab000 length = 0x11000 available RAM
base_addr = 0x30fbc000 length = 0x3c000 available RAM
base_addr = 0x30ff8000 length = 0x14000 available RAM
base_addr = 0x3100c000 length = 0x1000 available RAM
base_addr = 0x3100d000 length = 0x11000 available RAM
base_addr = 0x3101e000 length = 0x4000 available RAM
base_addr = 0x31022000 length = 0x12a2000 available RAM
base_addr = 0x322c4000 length = 0x19000 available RAM
base_addr = 0x322dd000 length = 0x10000 available RAM
base_addr = 0x322ed000 length = 0x8000 available RAM
base_addr = 0x322f5000 length = 0x2e1a000 available RAM
base_addr = 0x3510f000 length = 0x2d6000 available RAM
base_addr = 0x353e5000 length = 0xb5a000 available RAM
base_addr = 0x35f3f000 length = 0x3100000 reserved RAM
base_addr = 0x3903f000 length = 0x273000 ACPI reclaimable RAM
base_addr = 0x392b2000 length = 0xc0000 ACPI non-volatile storage RAM
base_addr = 0x39372000 length = 0xbf4000 reserved RAM
base_addr = 0x39f66000 length = 0x99000 RAM holding firmware code
base_addr = 0x30fab000, length = 0x11000, available RAM
base_addr = 0x30fbc000, length = 0x3c000, available RAM
base_addr = 0x30ff8000, length = 0x14000, available RAM
base_addr = 0x3100c000, length = 0x1000, available RAM
base_addr = 0x3100d000, length = 0x11000, available RAM
base_addr = 0x3101e000, length = 0x4000, available RAM
base_addr = 0x31022000, length = 0x12a2000, available RAM
base_addr = 0x322c4000, length = 0x19000, available RAM
base_addr = 0x322dd000, length = 0x10000, available RAM
base_addr = 0x322ed000, length = 0x8000, available RAM
base_addr = 0x322f5000, length = 0x2e1a000, available RAM
base_addr = 0x3510f000, length = 0x2d6000, available RAM
base_addr = 0x353e5000, length = 0xb5a000, available RAM
base_addr = 0x35f3f000, length = 0x3100000, reserved RAM
base_addr = 0x3903f000, length = 0x273000, ACPI reclaimable RAM
base_addr = 0x392b2000, length = 0xc0000, ACPI non-volatile storage RAM
base_addr = 0x39372000, length = 0xbf4000, reserved RAM
base_addr = 0x39f66000, length = 0x99000, RAM holding firmware code
base_addr = 0x39fff000, length = 0x1000, available RAM
base_addr = 0x100000000, length = 0x7c0000000, available RAM
base_addr = 0xa0000, length = 0x60000, reserved RAM
base_addr = 0x3a000000, length = 0x4000000, reserved RAM
base_addr = 0x3ee00000, length = 0x200000, reserved RAM
base_addr = 0x3f000000, length = 0x1000000, reserved RAM
base_addr = 0xc0000000, length = 0x10000000, reserved RAM
base_addr = 0xfc000000, length = 0x10000, reserved RAM
base_addr = 0xfe000000, length = 0x11000, reserved RAM
base_addr = 0xfec00000, length = 0x1000, reserved RAM
base_addr = 0xfed00000, length = 0x1000, reserved RAM
base_addr = 0xfed20000, length = 0x60000, reserved RAM
base_addr = 0xfee00000, length = 0x1000, reserved RAM
base_addr = 0xff000000, length = 0x1000000, reserved RAM

Alexey Kuznetsov <axet>
Wed 26 Jul 2023 10:53:41 AM UTC, comment #7: 

Can you post complete map from desktop as well?

 Vladimir Serbinenko <phcoder>
Group administrator
Wed 26 Jul 2023 10:51:27 AM UTC, comment #6: 

On desktop the only memory segment larger than 1GB is at

0x100000000 - 0x7c0000000

which is equivalent to 31G

Desktop has no 1G+ blocks below 4GB (0x100000000)

 Alexey Kuznetsov <axet>
Wed 26 Jul 2023 10:32:19 AM UTC, comment #5: 

I have secure boot disabled all the time. Since I compile grub manually for argon2id

 Alexey Kuznetsov <axet>
Wed 26 Jul 2023 10:25:18 AM UTC, comment #4: 

You are right. Not every EFI works with GRUB_EFI_MAX_USABLE_ADDRESS 0xffffffffffULL. My laptop won't boot if I change it. Grub simply unable to open drive (ssd) and unlock the key. Not sure what is going on.

Laptop:

[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000009cfefff] usable
[    0.000000] BIOS-e820: [mem 0x0000000009cff000-0x000000000a000fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000000a001000-0x000000000a1fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000000a200000-0x000000000a20efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000000a20f000-0x00000000bb251fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bb252000-0x00000000bc755fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bc756000-0x00000000bc7b1fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bc7b2000-0x00000000bc964fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bc965000-0x00000000bd140fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bd141000-0x00000000bd1fefff] type 20
[    0.000000] BIOS-e820: [mem 0x00000000bd1ff000-0x00000000bdffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000fdffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feb80000-0x00000000fec01fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fedc4000-0x00000000fedc9fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fedcc000-0x00000000fedcefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fedd5000-0x00000000fedd5fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000041e2fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000041e300000-0x000000043fffffff] reserved




dmesg | awk '/usable/ && /BIOS-e820/ && match($0, /mem ([0-9a-z]+)-([0-9a-z]+)/,aa){s=strtonum(aa[1]);e=strtonum(aa[2]);if(e-s>1*1024*1024*1024)printf("0x%x-0x%x %dGB\n",s,e,((e-s)/1024/1024/1024))}'



0xa20f000-0xbb251fff 2GB
0x100000000-0x41e2fffff 12GB


Desktop memory map on the screenshots.

 Alexey Kuznetsov <axet>
Wed 26 Jul 2023 08:21:42 AM UTC, comment #3: 

It's not that simple. Some firmwares fail to map memory above 4GiB and attempt to use it will lead to a bad crash. We need to parse mapping table to be sure that those are safe to use. Can you attach your EFI memory map? Does disabling secure boot makes a difference? It increases heap usage a lot

 Vladimir Serbinenko <phcoder>
Group administrator
Wed 26 Jul 2023 07:16:51 AM UTC, comment #2: 

My laptop wont boot with GRUB_EFI_MAX_USABLE_ADDRESS 0xffffffffffULL (asus), but desktop works fine (gigabyte).

Looks like we can detect how much memory we have.

Running simple script can show how much memory we have below 4GB mark and above:

dmesg | awk '/usable/ && /BIOS-e820/ && match($0, /mem ([0-9a-z]+)-([0-9a-z]+)/,aa){s=strtonum(aa[1]);e=strtonum(aa[2]);if(s<0x100000000 && e-s>1*1024*1024*1024)printf("0x%x-0x%x %dGB\n",s,e,((e-s)/1024/1024/1024))}'

following script print all memory blocks more then 1GB size:

dmesg | awk '/usable/ && /BIOS-e820/ && match($0, /mem ([0-9a-z]+)-([0-9a-z]+)/,aa){s=strtonum(aa[1]);e=strtonum(aa[2]);if(e-s>1*1024*1024*1024)printf("0x%x-0x%x %dGB\n",s,e,((e-s)/1024/1024/1024))}'

And if we do not have 1-2GB blocs below 4GB we need to increase GRUB_EFI_MAX_USABLE_ADDRESS

 Alexey Kuznetsov <axet>
Tue 25 Jul 2023 07:53:28 PM UTC, comment #1: 

GRUB_EFI_MAX_USABLE_ADDRESS 0xffffffffffULL

make it work! Default for amd64 0x0xffffffff

 Alexey Kuznetsov <axet>
Tue 25 Jul 2023 06:04:05 PM UTC, original submission:  

Hello!

I have installed grub 2.06 + argon2 and new memory management from grub 2.12 and new algorithm failed to allocate additional memory on some machines.

Looks like it trying to allocate memory from heap, then try few more times using low address spaces (where only small free blocks of few kb available) and stop before reaching really big blocks of free space of 30GB.

I tried different patches (4500-grub-2.06-runtime-memregion-alloc.patch):


And always grub failed to allocate 1GB, only works with 256M grub_malloc buffers.

But issues only happens on my desktop machine. Laptop works fine with those patches.

Here are photos of my display where grub failed. First tree photos about trying to allocate grub_malloc(256M) - success, then grub_malloc(512MB) - failed.

https://ibb.co/8gkS5pz https://ibb.co/khd8Mbv https://ibb.co/LzWJfN3

Second three photos from lsmmap command:

https://ibb.co/mykD0w1 https://ibb.co/nCbvWLR https://ibb.co/CVGK240

Basically you can see grub trying to allocate from:

0x8310000, then 0x1a390000, then 0x3510f000, then 0x2f151000, then 0x3510f000, then get back to 0x2f6a2000 and stops with out of memory error.

But real space starts at 0x100000000 with 0x7c0000000 (30GB) space available. So grub stops too early and fail to allocate when here is a plenty of space to work with.

 Alexey Kuznetsov <axet>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #54987:  smartmem.patch added by axet (1KiB - text/x-patch)
file #54986:  smartmem.patch added by axet (2KiB - text/x-patch)
file #54985:  2023-07-27@19.30.45.jpg added by axet (2MiB - image/jpeg)
file #54969:  lsmmap2.jpg added by axet (2MiB - image/jpeg)
file #54970:  lsmmap3.jpg added by axet (2MiB - image/jpeg)
file #54966:  grub_malloc1G-1.jpg added by axet (2MiB - image/jpeg)
file #54967:  grub_malloc1G-3.jpg added by axet (2MiB - image/jpeg)
file #54968:  lsmmap1.jpg added by axet (2MiB - image/jpeg)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by phcoder (Posted a comment)
  • -email is unavailable- added by axet (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-07-27 axet Attached File- Added smartmem.patch, #54987
    2023-07-27 axet Attached File- Added smartmem.patch, #54986
    2023-07-27 axet Attached File- Added 2023-07-27@19.30.45.jpg, #54985
    2023-07-25 axet Attached File- Added lsmmap2.jpg, #54969
        Attached File- Added lsmmap3.jpg, #54970
    2023-07-25 axet Attached File- Added grub_malloc1G-1.jpg, #54966
        Attached File- Added grub_malloc1G-3.jpg, #54967
        Attached File- Added lsmmap1.jpg, #54968

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-72d9.
    Corresponding source code