bugGNU Octave - Bugs: bug #53856, MEX file in a private directory of...

 
 

bug #53856: MEX file in a private directory of a class

Submitter:  Guillaume <gyom>
Submitted:  Mon 07 May 2018 10:38:14 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 15 May 2018 04:58:27 PM UTC, comment #17: 

I think we can close this as fixed now. The additional problem of a mex or oct function appearing after the directory has already been  loaded is covered by bug #40824.

Mike Miller <mtmiller>
Group Member
Tue 15 May 2018 09:20:15 AM UTC, comment #16: 

Than you, I can confirm it now works and solve the issue I reported here. Thank you also for pushing the fix to stable.

Guillaume <gyom>
Tue 15 May 2018 05:18:22 AM UTC, comment #15: 

OK, I think I see what is happening.  It's just luck that one type of file (.m, .oct, or .mex) is preferred when more than one type exists in a private directory because we weren't properly noting the multiple types.  Does the following patch fix this part of the problem for you?

  http://hg.savannah.gnu.org/hgweb/octave/rev/cc40e47d3a44

It still won't handle the case of a .mex or .oct file appearing after the .m file has been used because the out_of_date_check code doesn't properly handle that.  But it should at least make it so that the function we call doesn't depend on the order of the list of files returned by the directory reading function.

John W. Eaton <jwe>
Group administrator
Mon 14 May 2018 09:52:05 PM UTC, comment #14: 

I still get the following behavior (with Rik's mex_private example posted here, freshly extracted):


$ cd mex_private/private/
$ mkoctfile-4.4.0 --mex myprivfcn.c
$ cd  ..
$ octave -q
>> version -hgid
ans = c8f49ee7a687
>> tst_privfcn
error: Calling the m-file!
error: called from
    myprivfcn at line 2 column 3
    tst_privfcn at line 4 column 3
>> unlink private/myprivfcn.m;
>> tst_privfcn
error: myprivfcn: Calling the MEX-file!
error: called from
    tst_privfcn at line 4 column 3


And after restoring the m-file function, I looked at the '__dump_load_path__' function you mentioned:


>> __dump_load_path__


*** private functions in ./private:

  myprivfcn (m)




Mike Miller <mtmiller>
Group Member
Mon 14 May 2018 09:08:34 PM UTC, comment #13: 

I stlil can't explain why the .m file would be called if the mex file already exists.  But for the case of compiling the mex file after the m file has already been executed once, I think there are two problems.

First, we weren't looking at the timestamp of the private directory, so the load path was not updated if the private subdirectory changed.  This changeset should fix that problem:

  http://hg.savannah.gnu.org/hgweb/octave/rev/c8f49ee7a687

I am able to see a difference with the _dump_load_path_ function.  Previously, after compiling the .mex file, the private function map still had the .m file.  Now it will be updated to show the .mex file instead.

Second, the function out_of_date_check in symtab.cc doesn't check to see whether a .oct or .mex file has appeared when checking whether a .m file is out of date.  I don't have a fix for that yet because I don't know whether that check is always appropriate in the out_of_date_check function or if it should be done separately (or possibly that the entire logic here should be revamped).

However, with the first changeset above, the following now works for me when it previously failed:


addpath ("/path/to/mex_private")
tst_privfcn  ## calls .m file
cd /path/to/mex_private/private
mex myprivfcn.c
cd /some/where
tst_privfcn  ## still calls .m file
clear functions
tst_privfcn  ## now calls .mex file


Previously, the clear functions command had no effect because the load path was never updated.

John W. Eaton <jwe>
Group administrator
Fri 11 May 2018 03:11:49 PM UTC, comment #12: 

Yes, those are the places to look.

John W. Eaton <jwe>
Group administrator
Fri 11 May 2018 03:08:31 PM UTC, comment #11: 

There are a number of bug reports that seem related in one way or another: bug #40824, bug #45444, bug #46281, bug #47693, bug #48925

Are libinterp/corefcn/fcn-info.cc and libinterp/corefcn/load-path.cc the place to look at for this?

Guillaume <gyom>
Wed 09 May 2018 03:23:27 PM UTC, comment #10: 

Thanks for your tests - it's a funny one so I'm glad some of you reproduced it. As I mentioned in my initial comment, I've got some more complicated examples involving mex files and private directory where it works fine so it's a strange interaction with something else I cannot figure out yet.

I don't have a simple example at hand but I' have also seen situations where private m-files functions where not detected by Octave (without any mex file involved) so there is also probably another dimension to this error.

One way to reproduce the error is through Mike's docker image:
  https://hub.docker.com/r/mtmiller/octave-snapshot/

Guillaume <gyom>
Wed 09 May 2018 01:52:15 AM UTC, comment #9: 

I'll try to figure out what is going on with this problem.

John W. Eaton <jwe>
Group administrator
Tue 08 May 2018 10:27:51 PM UTC, comment #8: 

I just updated and installed revision bc5f225bc578 and I still see the same thing on my system, the m-file is always called even if the mex file exists when Octave starts.


>> tst_privfcn
error: Calling the m-file!
error: called from
    myprivfcn at line 2 column 3
    tst_privfcn at line 4 column 3
>> cd private
>> myprivfcn
error: myprivfcn: Calling the MEX-file!
>> cd ..
>> tst_privfcn
error: Calling the m-file!
error: called from
    myprivfcn at line 2 column 3
    tst_privfcn at line 4 column 3


Mike Miller <mtmiller>
Group Member
Tue 08 May 2018 09:16:30 PM UTC, comment #7: 

Rik's test code works for me as well if I compile the MEX file before executing the test function but it fails if I start Octave without the MEX file compiled, then do


addpath ("/path/to/mex_private")
tst_privfcn  ## calls .m file
cd /path/to/mex_private/private
mex myprivfcn.c
cd /some/where
tst_privfcn  ## still calls .m file
_verbatim-

The problem isn't fixed by "clear all", so this looks like a problem with the load-path or fcn-info code not noticing that there is a new file available.

John W. Eaton <jwe>
Group administrator
Tue 08 May 2018 09:14:50 PM UTC, comment #6: 

I compiled the mex file in the shell with Octave not running. I basically followed the steps you posted in comment #3.

Here is an exact transcript of what I am doing:


$ tar -xf ~/Downloads/mex_private.tar.gz
$ cd mex_private/private/
$ mkoctfile-5.0.0 --mex myprivfcn.c
$ cd ..
$ octave-5.0.0 --norc --no-history --silent
octave:1> tst_privfcn
error: Calling the m-file!
error: called from
    myprivfcn at line 2 column 3
    tst_privfcn at line 4 column 3


Mike Miller <mtmiller>
Group Member
Tue 08 May 2018 09:00:51 PM UTC, comment #5: 

Boy, that's weird.  I have re-confirmed that I get the print out from the MEX file with 5.0.0 which is installed, and also using run-octave in a development directory.  The HG ID for this is


parent: 25360:bc5f225bc578 tip
 eliminate some global accesses to the breakpoint table


from today.

If I start with no .mex file in the private directory, then I get the output from the m-file.  With Octave still running, if I then use a shell and mkoctfile to compile the .mex file, I still get the output from them-file.  No amount of 'clear -f' or 'clear -a' will make Octave see the new .mex file.  However, if I quit Octave and re-start then the printout is from the .mex file.

So, it may be something about scanning for .mex files only once at startup.  I also tried 'rmpath mex_private; addpath mex_private' hoping to force a re-scan, but it didn't work.

If the .mex file is present, however, when you start Octave then you are seeing a different issue.

Rik <rik5>
Group administrator
Tue 08 May 2018 08:29:11 PM UTC, comment #4: 

On my system Octave prints the message from the m-file, both with the original example and with Rik's example in comment #3.

Mike Miller <mtmiller>
Group Member
Tue 08 May 2018 05:09:17 PM UTC, comment #3: 

I tested the situation in comment #2, with a mex file in a private directory which is not associated with a classdef file, and it works for me.

I've attached the entire directory structore and files that I used in mex_private.tar.gz.

To test


1) tar xf mex_private.tar.gz
2) cd mex_private/private
3) mkoctfile --mex myprivfcn.c
4) cd ..
5) octave -f
6) tst_privfcn


On my system it prints out a message from the MEX file, not the message from the m-file.


(file #44124)

Rik <rik5>
Group administrator
Tue 08 May 2018 09:22:20 AM UTC, comment #2: 

And same thing with a classdef file. The issue is actually not class related and occurs with the following:


.
├── callprivate.m
└── private
    ├── mysubfunction.c
    ├── mysubfunction.cc
    ├── mysubfunction.m
    ├── mysubfunction.mex
    └── mysubfunction.oct


where mysubfunction.* is as before while callprivate.m is:


function callprivate ()
  mysubfunction ();
endfunction


callprivate () calls the M function and not its compiled alternative.

Guillaume <gyom>
Tue 08 May 2018 08:54:02 AM UTC, comment #1: 

I tried with an oct file instead of a mex file and the same thing happens.

Guillaume <gyom>
Mon 07 May 2018 10:38:14 AM UTC, original submission:  

I observed errors by which M functions in a private directory are not overshadowed by an associated MEX file. I have got plenty of examples where it does work but it seems to fail with the toy example below.

Just compile the MEX file with mkoctfile --mex mysubfunction.c and run:


octave:1> a = myclass ();
octave:2> mymethod (a)
error: Calling the M-file!
error: called from
    mysubfunction at line 2 column 3
    mymethod at line 2 column 3



@myclass
├── myclass.m
├── mymethod.m
└── private
    ├── mysubfunction.c
    ├── mysubfunction.m
    └── mysubfunction.mex



function m = myclass
  m = class (struct ('a',1), 'myclass');
endfunction



function mymethod(a)
  mysubfunction ();
endfunction



function mysubfunction ()
  error ("Calling the M-file!");
endfunction



#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
  mexErrMsgTxt("Calling the MEX-file!");
}


Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44124:  mex_private.tar.gz added by rik5 (3KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-15 mtmiller StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2018-05-14 jwe StatusConfirmed In Progress
    2018-05-10 rik5 StatusWorks For Me Confirmed
    2018-05-08 rik5 Attached File- Added mex_private.tar.gz, #44124
        Item GroupUnexpected Error or Warning Matlab Compatibility
        StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code