bugGNU Octave - Bugs: bug #57872, newly compiled function...

 
 

bug #57872: newly compiled function (.mex/.oct) with same name as m-file does not immediately override function file

Submitter:  None
Submitted:  Fri 21 Feb 2020 05:09:22 AM UTC
   
 
Category:  Interpreter Severity:  2 - Minor
Priority:  1 - Later Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 24 Feb 2020 04:56:53 PM UTC, comment #11: 

Interesting that Matlab notices the new file from within the invocation of a function.

I thought that we only needed to update the loadpath (or the cache of what is in the loadpath) when the prompt is displayed.

Is the mex function triggering the update, or is it just the appearance of the mex file?  What happens if you compile the mex file somewhere that is outside of the loadpath and then do something like


  system ('cp testme.mex .')
  testme (varargin{:})


in the testme.m function?  No call to mex in the testme.m function, just a system call to make it available.

For performance reasons, I don't think we want to be updating (or  checking whether the loadpath needs to be updated) every time we are resolving a symbol that could refer to a function.

John W. Eaton <jwe>
Group administrator
Mon 24 Feb 2020 04:30:39 PM UTC, comment #10: 

This behavior exists with any compiled file (.mex or .oct) so I have re-titled the issue report.

The issue only exists for a compiled file with exactly the same name as the executing m-file.  Presumably this is because Octave uses the function name as the key in its internal cache and the name doesn't change so no update of the symbol table occurs.

This is a very odd code pattern, and there are several workarounds, so I have lowered the priority to 1.  If someone wants to code a patch for this they can, but there are other bugs that would be more useful to fix first.

Workaround number one is simply to compile to a different file name and have the m-file dispatch to the differently named mex/oct file.  Octave uses this pattern internally all the time where the help string and input validation are done in an m-file, and the actual hard work is done in a compiled file.  See, for example, mkdir.m which uses an internal function written in C++ to do the work.


    [status, msg, msgid] = __mkdir__ (parent, dirname);


Workaround number two is to force an update of Octave's symbol table from within the m-file.  For performance, Octave normally only updates the symbol table upon returning to a command prompt.  See test code below (also attached as testme.m)


function retval = testme ()

  disp ("Entering m-file testme.m");

  if (! exist ("testme.mex", "file"))
    mex ("-v", "testme.cpp", "-output", "testme.mex");
  endif

  which ("testme")
  rehash
  which ("testme")
  disp ("Calling testme() from within m-file testme.m");
  testme ();

  disp ("Leaving m-file testme.m");

endfunction


When executed, results are


octave:1> testme
Entering m-file testme.m
g++ -c  -fPIC -I/home/rik/local/include/octave-7.0.0/octave/.. -I/home/rik/local/include/octave-7.0.0/octave -I/home/rik/local/include  -pthread -fopenmp -O2 -pipe   -I.  testme.cpp -o /tmp/oct-fqahp6.o
g++ -I/home/rik/local/include/octave-7.0.0/octave/.. -I/home/rik/local/include/octave-7.0.0/octave -I/home/rik/local/include  -pthread -fopenmp -O2 -pipe -shared -Wl,-Bsymbolic   -o testme.mex  /tmp/oct-fqahp6.o    -L/home/rik/local/lib
'testme' is a function from the file /home/rik/wip/Projects_Mine/octave-dev/comp2tst/testme.m
'testme' is a function from the file /home/rik/wip/Projects_Mine/octave-dev/comp2tst/testme.mex
Calling testme() from within m-file testme.m
Executing mex file testme.cpp.
Leaving m-file testme.m


You can see that the call to 'rehash' forces Octave to recognize testme.mex.



(file #48493)

Rik <rik5>
Group administrator
Sun 23 Feb 2020 07:58:26 PM UTC, comment #9: 

Ok, I finally tested the original example, and there are two things going on here.

Forget the '@testme' directory first. When the user calls 'testme' and the mex file doesn't exist yet, compiling the mex file inside the testme.m function file doesn't allow the mex file to be called immediately, that probably has to do with lazy rehashing of the load path. We may have another open bug report about that.

Now, with the '@testme' directory, the mex file does not take precedence even if it's already present when the user calls 'testme'. That's a weird new addition in this bug report that I don't really see a practical application for supporting, but if someone wants to look into it, may be easy to fix.

Mike Miller <mtmiller>
Group Member
Sun 23 Feb 2020 06:35:26 PM UTC, comment #8: 

@Anonymous: In the sequence posted in comment #7, does Octave return "Executing mex file" if you stay within the same Octave session that you used to compile the mex file with?  Or does it work if you quit and restart Octave?

This may be something extremely simple like re-hashing of the PATH after the compilation so that Octave sees there is a new compiled mex file.

Rik <rik5>
Group administrator
Sun 23 Feb 2020 05:26:05 PM UTC, comment #7: 

The usual solution to keep all stuff together (m-file, cpp file, etc.) is to create a namespace for the functions.  The namespace is created by using '+' as the first character of the directory name, and then you can access those functions with "dir_name.function_name"

But, I still don't see any difference between Octave and Matlab.  I used the archive I already uploaded (tst_precedence.zip) for testing.


# In shell
unzip tst_precedence.zip
mv tst_prec @testme
octave

# In Octave
cd @testme
mex -v testme.cpp
cd ..
testme


Result is


Executing mex file testme.cpp.



Rik <rik5>
Group administrator
Sun 23 Feb 2020 03:44:39 PM UTC, comment #6: 

I guess it's an abuse of the class functionality. It's convenient to keep all the stuff (m-file, cpp file and mex) all in one space. Not sure about the class constructor... in Matlab you can just call testme() like any other function.

Anonymous
Sat 22 Feb 2020 09:34:48 PM UTC, comment #5: 

I understand that part, but not the part of including it in a class definition '@' directory.

After creating the mex file, the class constructor would no longer work.

Mike Miller <mtmiller>
Group Member
Sat 22 Feb 2020 09:21:48 PM UTC, comment #4: 

The point of overloading the mex file is to have an m-file that compiles it on first use. Seems very useful to me. Anyway, this "works" in Matlab however frustrating it must be to purists ;)

Anonymous
Fri 21 Feb 2020 07:22:17 PM UTC, comment #3: 

The '@' directory naming convention is also used for new-style classdef classes.

But I'm not sure what the utility would be of creating a mex file that shadows a class definition. So I think it's right that a class definition or constructor function file always takes precedence in a '@' directory.

Mike Miller <mtmiller>
Group Member
Fri 21 Feb 2020 03:46:22 PM UTC, comment #2: 

This works for me.  First, don't use '@' as the first character in the name of a directory.  That creates a Matlab class using their first attempt at introducing object oriented programming to the language.  The new way to create classes is to use the classdef keywoard, but the old syntax still remains (confusing many people).

I made a test case which is attached as tst_precedence.zip.

It is a directory with two files in it


tst_prec/
--testme.cpp
--testme.m


The file testme.cpp is


#include "mex.h"

void
mexFunction (int nlhs, mxArray *plhs[],
             int nrhs, const mxArray *prhs[])
{
  mexPrintf ("Executing mex file testme.cpp.\n");
}


The second file is testme.m


function testme ()
  disp ('Executing m-file testme.m.');
end


A sample run of Octave shows that the m-file is picked up, unless there is a mex file which overrides it.


octave:1> testme
Executing m-file testme.m.
octave:2> mex -v testme.cpp
g++ -c  -fPIC -I/home/rik/local/include/octave-7.0.0/octave/.. -I/home/rik/local/include/octave-7.0.0/octave -I/home/rik/local/include  -pthread -fopenmp -O2 -pipe   -I.  testme.cpp -o /tmp/oct-gym0u6.o
g++ -I/home/rik/local/include/octave-7.0.0/octave/.. -I/home/rik/local/include/octave-7.0.0/octave -I/home/rik/local/include  -pthread -fopenmp -O2 -pipe -shared -Wl,-Bsymbolic   -o testme.mex  /tmp/oct-gym0u6.o    -L/home/rik/local/lib
octave:3> testme
Executing mex file testme.cpp.




(file #48462)

Rik <rik5>
Group administrator
Fri 21 Feb 2020 05:16:40 AM UTC, comment #1: 

Actually something else must be going on with the @.


>> cd @testme
>> testme
testme cpp


Anonymous
Fri 21 Feb 2020 05:09:22 AM UTC, original submission:  

In MATLAB, the mexfunction has highest priority. In Octave my code errors out with a max recursion limit error. Looks like m-file has priority over mexfunction.

1. Create a directory @testme with the following files.
2. At Octave prompt: >> testme
3. >> testme
g++ -c  -fPIC -I/usr/local/include/octave-5.2.0/octave/.. -I/usr/local/include/octave-5.2.0/octave -I/usr/local/include  -pthread -fopenmp -flto -s -fomit-frame-pointer -march=native   -I.  -DMEX_DEBUG /home/work/octave/@testme/testme.cpp -o /tmp/oct-3Gdij3.o
g++ -I/usr/local/include/octave-5.2.0/octave/.. -I/usr/local/include/octave-5.2.0/octave -I/usr/local/include -pthread -fopenmp -flto -s -fomit-frame-pointer -march=native -shared -Wl,-Bsymbolic   -o /home/mark/octave/@testme/testme.mex  /tmp/oct-3Gdij3.o    -L/usr/local/lib
error: max_recursion_depth exceeded
error: called from
    testme at line 10 column 8

4. Incidentally the clickable link in the error message brings up the wrong file.

++ver
testme.m

function output = testme(varargin)

mexbin = [mfilename('fullpath') '.' mexext];

if ~exist(mexbin,'file')
    mexcpp = [mfilename('fullpath') '.cpp'];
    mex('-v',mexcpp,'-output',mexbin);
end

output = testme(varargin{:});



testme.cpp

#include "mex.h"
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <iostream>

void mexFunction(int nlhs, mxArray plhs[], int nrhs, const mxArray prhs[])
{
        std::cout << "testme cpp" << std::endl;
}
--verbatim--

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48493:  testme.m added by rik5 (337B - text/x-matlab)
file #48462:  tst_precedence.zip added by rik5 (168B - application/zip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-02-24 rik5 Attached File- Added testme.m, #48493
        CategoryNone Interpreter
        Priority3 - Low 1 - Later
        Item GroupNone Matlab Compatibility
        Release5.2.0 dev
        Summarynewly compiled mex function with same name does not immediately override function file newly compiled function (.mex/.oct) with same name as m-file does not immediately override function file
    2020-02-23 mtmiller Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
        SummaryPriority of mex functions with the same name newly compiled mex function with same name does not immediately override function file
    2020-02-23 mtmiller StatusWorks For Me Confirmed
    2020-02-21 rik5 Attached File- Added tst_precedence.zip, #48462
        StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code