bugMIT/GNU Scheme - Bugs: bug #60267, Possible method of resolving the...

 
 

bug #60267: Possible method of resolving the Apple M-1 W^X issue

Submitter:  John Cowan <johnwcowan>
Submitted:  Mon 22 Mar 2021 03:03:27 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open
Keywords: 
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 07 Dec 2024 04:52:26 PM UTC, comment #7: 


comment #6:

> My impression from Brooks was that his Common Lisp doesn't separate code from data, either.  He specifically said that self-modifying code was the issue for him.


I think it has to separate code and data.

The model of MAP_JIT is:

1. After pthread_jit_write_protect_np(0), you can write to the memory but you can't execute code in it.

2. After pthread_jit_write_protect_np(1), you can execute code in the memory but you can't write to it.

If there is only one heap with the same mapping characteristics (which is how MIT Scheme works now), if you map it with MAP_JIT and you want to write to anything in the heap (and that includes consing up new data, which writes to the heap pointer and advances the heap pointer), you can't execute code stored in that heap.  And vice versa: if you want to execute anything in the heap, you can't write to it (and that means you can't cons up new data).

So you can't put compiled code blocks into the same heap that you're trying to use for consing, at all -- even if you're not writing to the compiled code blocks.

What you need to do -- if you want to allow compiled code to write to memory or cons up new objects -- is have separate mappings, with separate MAP_* mapping characteristics, for code and data.  The logic you quoted certainly looks like it's doing exactly that, particularly this part:

>     if (i<2) { /* this is a code segment */
>       if (i==0) pthread_jit_write_protect_np(0);
>       segaddress[i] = (addrint)mmap(0, segdesired[i],
>     PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_JIT,
>     -1, 0);
>       if (0<seglength[i]) {
> /* fread interacts badly with mmap for some reason, so we roll our own fread here */
> place = (char *)segaddress[i];
> for(jj=0;jj<seglength[i];jj++,place++) *place=255&fgetc(loadfile);}}
>     else {
>       segaddress[i] = Lalloc(segdesired[i]);
>       fread((char *)segaddress[i], seglength[i], 1, loadfile);}


For code segments it uses mmap(MAP_JIT); for non-code segments it uses Lalloc.

Taylor R. Campbell <riastradh>
Group Member
Sat 07 Dec 2024 02:34:27 AM UTC, comment #6: 

My impression from Brooks was that his Common Lisp doesn't separate code from data, either.  He specifically said that self-modifying code was the issue for him.

Arthur A. Gleckler <aag>
Group Member
Fri 06 Dec 2024 11:51:10 PM UTC, comment #5: 

Yep, the usage model of Apple's MAP_JIT and pthread_jit_write_protect_np (and analogous mechanisms on other platforms) is well-understood.  The hard part is that MIT Scheme doesn't have separate heaps for code and data.  The garbage collector (and fasdumper and fasloader) has to be taught to separate them.

I started drafting some code for that a while ago but it hasn't been a priority to work on since I don't have any systems that require it:

https://lists.gnu.org/archive/html/mit-scheme-devel/2022-12/msg00001.html
https://git.savannah.gnu.org/cgit/mit-scheme.git/commit/?h=wx&id=0a6ca0e12df7f722783e1d1a4235af8bad5b8166

Taylor R. Campbell <riastradh>
Group Member
Fri 06 Dec 2024 10:52:37 PM UTC, comment #4: 

Just in case it's helpful, I'm recording the message below in this issue.  I ran into Prof. Rodney Brooks recently, and we talked about his Common Lisp implementation, which ran into the same issue on the M1 as MIT Scheme has hit.  He sent me this message, which gives a detailed explanation of how he solved this problem in his implementation:

> My lisp image is pretty much independent of the operating system so
> there is a C program, the coldloader, that handles all of that side.
> There is a coldload file that has multiple code segments and data
> segments and the coldloader load that all into memory at the start
> and then code there gets jumped into, and now it is running lisp.
> When lisp needs to communicate in any way with the operating system
> (e.g., to read or write a file, or print to the TTY, or talk to
> Xwindows, it does call backs into C code in the coldloader to handle
> it.
>
> Here is all the C code for the special cases for M1 silicon, that
> comes directly from my coldloader.
>
> Note that the type "addrint" is really uint64_t for the Mac M1 in 64
> bit mode (the code for the coldloader runs on many different
> architectures, endianness, and size of memory pointers, so it is
> conditionalized and configured with typedefs).
>
> Header files
>
> #ifdef NEEDS_MAC_JIT
> #include <libkern/OSCacheControl.h>
> #include <pthread.h>
> #endif
>
>
> And this is the code that loads code and data segments into memory
> before being jumped into--segments 0 and 1 are both code segments,
> though they can also get read as data if the inspector wants to
> display the code bits in memory.
>
>
> #ifndef NEEDS_MAC_JIT
>     if (staticid==i)
>       segaddress[i] = vregbytes+Lalloc(segdesired[i]+vregbytes);
>     else segaddress[i] = Lalloc(segdesired[i]);
>     fread((char *)segaddress[i], seglength[i], 1, loadfile);
> #endif
> #ifdef NEEDS_MAC_JIT
>     if (i<2) { /* this is a code segment */
>       if (i==0) pthread_jit_write_protect_np(0);
>       segaddress[i] = (addrint)mmap(0, segdesired[i],
>     PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_JIT,
>     -1, 0);
>       if (0<seglength[i]) {
> /* fread interacts badly with mmap for some reason, so we roll our own fread here */
> place = (char *)segaddress[i];
> for(jj=0;jj<seglength[i];jj++,place++) *place=255&fgetc(loadfile);}}
>     else {
>       segaddress[i] = Lalloc(segdesired[i]);
>       fread((char *)segaddress[i], seglength[i], 1, loadfile);}
> #endif
>   }
>   fclose(loadfile);
> #ifndef NEEDS_MAC_JIT
>   /* make the code segments exectuable, in case they are not by default.
>      this makes the assumption that 0=pure code and 1=usercode. */
>   mprotect((void *)segaddress[0], segdesired[0], PROT_READ|PROT_WRITE|PROT_EXEC);
>   mprotect((void *)segaddress[1], segdesired[1], PROT_READ|PROT_WRITE|PROT_EXEC);
> #endif
> #ifdef NEEDS_MAC_JIT
>   pthread_jit_write_protect_np(1);
>   sys_icache_invalidate((void *)segaddress[0], seglength[0]);
>   if (0<seglength[1]) sys_icache_invalidate((void *)segaddress[1], seglength[1]);
> #endif
>
> Now here is the code that the compiler or the lisp code loader need
> to use to copy code bytes into memory and then let them be
> executable (note that I don't know if you can run an M1 mac in 32
> bit mode, but I just put that here in case...):
>
>
> // with JIT we can only run this original C proram when we are writing any instructions to memory
> // so faslin and compile in place need to access write_jit_code_vector to do their dirty work.
> // so far this is only for a Macintosh JIT.
>
> #ifdef _L_64BIT
> #define byteshifter (3)
> #endif
> #ifdef _L_32BiT
> #define byteshifter (2)
> #endif
>
> addrint write_jit_code_vector (addrint *cvect, addrint pointercount, addrint *fromplace, addrint header) {
> #ifdef NEEDS_MAC_JIT
>   int ii;
>   pthread_jit_write_protect_np(0);
>   *(cvect-1) = header; // the header wasn't put there in %make-code for this machine
>   for(ii=0;ii<pointercount;ii++) cvect[ii]=fromplace[ii];
>   pthread_jit_write_protect_np(1);
>   sys_icache_invalidate((void *)cvect, pointercount<<byteshifter);
> #endif
>   return 0;}
>
> --Rod

Arthur A. Gleckler <aag>
Group Member
Mon 06 Nov 2023 02:19:45 PM UTC, comment #3: 

Yes, the nature of the issue is well-understood; the problem is a lack of effort to implement it.  See https://lists.gnu.org/archive/html/mit-scheme-devel/2022-12/msg00001.html for the partial effort I started.  Personally I don't have any systems that require this anyway so it hasn't been a priority for me.

Taylor R. Campbell <riastradh>
Group Member
Mon 06 Nov 2023 02:09:34 PM UTC, comment #2: 

Another possible approach is to use a memory region that has two mappings at different addresses: one with W and one with X. GNU libffcall uses this, see https://git.savannah.gnu.org/gitweb/?p=libffcall.git;a=blob;f=trampoline/trampoline.c;h=fdcbcd456aeb20b87b846a8d9f930ce2781a1195;hb=HEAD (search for EXECUTABLE_VIA_MMAP_FILE_SHARED), and it works even on HardenedBSD and macOS.

The drawbacks are:
- This memory is not anonymous; it must be taken from a file.
- Some address management is needed, because in some places you want to address the memory through the W mapping and in other places through the X mapping.

Bruno Haible <haible>
Sat 04 Jun 2022 09:16:00 PM UTC, comment #1: 

We would certainly have to use this approach, but it's not enough because at any given time you can only toggle between W xor X -- you can't have both set at the same time.

Currently compiled code (executable data) and mutable objects (writable data) are both stored in the same heap.  What we need to do is have a separate heap for compiled code, and whenever we make (or load) compiled code:

1. Toggle the compiled-code heap from executable to writable.
2. Run a GC and move any new compiled-code blocks from the object heap where they were created into the compiled-code heap (GC is needed both to free up unreferenced compiled-code blocks and ensure pointers in the object heap to new compiled-code blocks are relocated to point at the compiled-code heap instead).
3. Toggle the compiled-code heap back from writable to executable.

Other systems like NetBSD will use a slightly different approach: create two virtual address windows into the same anonymous mapping, one mapped executable and the other mapped writable (and maybe only created while the GC is happening).

It's doable, just a pain to deal with all the bookkeeping, which is why I haven't bothered.  (I started drafting some logic to do it back in 2020 but I can't find it now.  Not sure I got far enough to bother looking instead of just starting over, though.)

Taylor R. Campbell <riastradh>
Group Member
Mon 22 Mar 2021 03:03:27 AM UTC, original submission:  

Apple has posted a method for supporting JITs on Apple M-1 machines.  Basically one must allocate a segment with mmap using the MAP_JIT option.  Then the function pthread_jit_write_protect_np must be called with a boolean argument:  false turns off write-protection and enables execution, and true turns turns on write-protection and disables execution.

I hope this is helpful.

John Cowan <johnwcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by aag (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by riastradh (Posted a comment)
  • -email is unavailable- added by johnwcowan (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-04e1.
    Corresponding source code