bugGNU Octave - Bugs: bug #54788, Implementation of memmapfile

 
 

bug #54788: Implementation of memmapfile

Submitter:  Guillaume <gyom>
Submitted:  Fri 05 Oct 2018 01:01:02 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Guillaume Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 11 Sep 2020 05:04:26 PM UTC, comment #2: 

Adding a link from bug #58953 that might be useful to deal with m.Data(end):
https://stackoverflow.com/questions/23016606/what-are-the-semantics-of-end-in-matlab/23017087#comment35192810_23017087
The issue here is that m.Data(end) will first evaluate m.Data, i.e. read the entire mapped data into memory and then return its last value, defeating the point of memory mapping.
Or is it that the dependent property Data should contain an object from the  _memmapfile_handle_ handle class (instead of private property MemoryMap) so that an end() method can be implemented in that class?

Guillaume <gyom>
Mon 08 Oct 2018 09:01:27 AM UTC, comment #1: 

As a follow-up, I attach a version of the toy example I mentioned previously, _mmap_.cc, rewritten as a DEFUN_DLD. It works like this:


octave> fid = fopen ("data.dat", "w");
octave> fwrite (fid, 1:128, "uint8");
octave> fclose (fid);
octave>
octave> h = __mmap__ ("data.dat", 128, 0); % mmap
octave> __mmap__ (h, "uint8", 0:7)
ans =
  1  2  3  4  5  6  7  8
octave> typecast (ans, "double")
ans =   5.4476e-270
octave> __mmap__ (h, "double", 0)
ans =   5.4476e-270
octave> __mmap__ (h, 128); % munmap


One question I have pertains to subsref/subsasgn: instead of having my own inefficient implementation (speed- and memory-wise), is there a way to reuse Octave's versions? I.e., given:


void *ptr; /* from mmap */
dim_vector dim; /* array dimensions */
std::string precision; /* eg, "uint32" */
octave_value_list idx; /* eg, {":", [3 4 5], 2, ":", [2 3]} */


would return a uint32NDArray containing the values from *prt corresponding to idx.

Guillaume <gyom>
Fri 05 Oct 2018 01:01:02 PM UTC, original submission:  

Matlab has a class to perform memory-mapping on a file from disk:

https://www.mathworks.com/help/matlab/ref/memmapfile.html

I attach two files, memmapfile.m and _memmapfile_handle_.m, as an initial attempt at implementing this functionality in Octave and would welcome feedbacks and help to finish it.

As far as I can see, the code in memmapfile.m is feature complete. The internal handle class in _memmapfile_handle_.m is where further work is required. An extra handle class has to be defined because memmapfile is a value class and, given that the memory mapping can take place at any time including when the data is first accessed for reading, a mechanism has to be in place to store the mapping pointer. At the moment, memory mapping isn't actually taking place and fread/fwrite is used to mock the real behavior (I know, disappointing!). For example, instead of having:
  this.handle = fopen (filename, writable);
there should be a call:
  [this.handle, err, msg] = _mmap_ (filename, length, offset, writable);
to a low-level (oct-file or core?) _mmap_ that would open/mmap/close the file and return the address of the mapping as a double. This function would also have to deal with page size for the offset parameter so it would probably need to have extra output variables containing the eventually modified offset and length of the mapping.
Same thing with munmap instead of fclose. I have a toy example showing how memory mapping would work but I need help/guidance in specifying and writing the DEFUN_DLD function(s).

Other aspects to consider:

  • On Windows, CreateFile/CreateFileMapping/MapViewOfFile will have to be used instead of mmap:

  https://docs.microsoft.com/en-us/windows/desktop/memory/file-mapping

  • There are problems with the 'end' keyword. One is a separate bug #54783 that prevents m.Data(end) = 0; to work. But, more generally, the use of the 'end' keyword is problematic here in the context of memory mapping because its evaluation will try to access to all of the mapped data. Until a satisfactory solution is found, I would document this and discourage its use.
  • I noticed that some of the unit tests for errors do not report a bug if an error is not raised.
  • Care will have to be taken if a memmapfile object is copied or saved into a mat-file such that the mapping is deleted first.
  • The local function get_dimensions() in memmapfile.m could be cached so that its evaluation is only done once for a given mapping but it is very light anyway so I thought it was preferable than adding extra private properties.


Thanks to jwe and carandraug for their help.

Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45163:  __mmap__.cc added by gyom (2KiB - text/x-c++src)
file #45149:  memmapfile.m added by gyom (22KiB - text/x-matlab)
file #45150:  __memmapfile_handle__.m added by gyom (3KiB - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-08 gyom Attached File- Added _mmap_.cc, #45163
    2018-10-05 gyom Attached File- Added memmapfile.m, #45149
        Attached File- Added _memmapfile_handle_.m, #45150

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code