bugGNU Octave - Bugs: bug #48408, save -append should overwrite a...

 
 

bug #48408: save -append should overwrite a variable with the same name

Submitter:  None
Submitted:  Tue 05 Jul 2016 12:26:43 PM UTC
   
 
Category:  Octave Function Severity:  2 - Minor
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 06 Jul 2016 03:01:52 PM UTC, comment #7: 

I do agree it is low priority, but it's worth fixing.
In terms of workaround, it is just simpler to load the entire file.

Anonymous
Tue 05 Jul 2016 10:00:10 PM UTC, comment #6: 

The easy way to update variables is not to use Matlab ;)

-append certainly works in Octave by itself, and in Matlab by itself.  If you want to swap data files between the two then just make sure you write out all the variables you need.

For example,


varnames = {'x', 'b'};
save ('-v7', 'tst.mat', varnames{:})


jwe's suggestion works fine.  But load/save are both implemented in C++ rather than as m-files.  This means it is not quite as simple to code up, even though the concept is fine.

You could always write your own wrapper around save.  Something vaguely like this:


function save_append (fname, varargin)
  s = load (fname);
  for i = 1:numel (varargin)
     varname = varargin{i};
     if (exist (varname, "var"))
       s.(varname) = __varval__ (varname);
     endif
  endfor
  save ("-v7", fname, "-struct", s);
endfunction




Rik <rik5>
Group administrator
Tue 05 Jul 2016 09:30:21 PM UTC, comment #5: 

You can do what Rik suggests in comment #3.  But I think you can store the variables obtained from the load function in a struct, modify the struct with the new variable(s) that you are appending, then write the struct out to the same file that you just loaded.

Maybe you could do better in C++.

In any case, it is also low priority for me unless someone wants to use $$ to increase my interest.

John W. Eaton <jwe>
Group administrator
Tue 05 Jul 2016 09:13:27 PM UTC, comment #4: 

I am the original submitter of the bug report.


This is annoying, and possibly not worth fixing


I don't see how else you can update variables in a file.
You have to use -append and Octave should overwrite the existing  variable.
I would say it is well worth fixing.
It might be hard to do though...

And I would also say that efficiency is a big deal.
I have no idea how this could/should be done, but maybe erasing the previous versions of the variables that should be overwritten (leaving empty spaces within the file), might be an option.

Anonymous
Tue 05 Jul 2016 04:14:39 PM UTC, comment #3: 

Mike is correct.  Matlab specifically mentions that Matlab overwrites an existing variable when using -append.

This is annoying, and possibly not worth fixing.  I'm pretty sure Octave just relies on opening the file in 'append' mode and adding the data at the end of the file.

The alternative, which Matlab seems to do, is to open the file for reading, parse through all contents, and copy all the unduplicated variables to a new file, save in the new or duplicated appended variables, close the file, and then rename the new temporary file to the old one.

If efficiency isn't a big deal, it might be possible to work at the m-file level in a separate namespace.  First, clear the namespace of all variables.  Next, load in the existing file to get all of the variables in the file.  Use evalin() or somesuch to import the variables that are to be appended.  Finally, write out all variables in the temporary namespace to the file.  But, if you have 2GB arrays, then this is going to be cumbersome.

Rik <rik5>
Group administrator
Tue 05 Jul 2016 03:47:59 PM UTC, comment #2: 

I've changed the Item Group to Matlab Compatibility, and marked it as minor since there is the obvious workaround of loading the whole file.

The sequence of commands works as long as one stays in Octave.  This leads me to believe that it is Matlab which has the issue.  My guess is that even for '-append', they may be checking whether the variable exists and replacing the variable, rather than appending it.

In Octave, I checked the size of the generated file after every save command.


x = 1
save -v7 tst.mat x
ls -l tst.mat
  -rw-rw-r-- 1 rik rik 174 Jul  5 08:38 tst.mat
b = zeros (2,2);
save -v7 -append tst.mat b
ls -l tst.mat
  -rw-rw-r-- 1 rik rik 220 Jul  5 08:38 tst.mat
b(1,1) = 1
save -v7 -append tst.mat b
ls -l tst.mat
  -rw-rw-r-- 1 rik rik 271 Jul  5 08:39 tst.mat


You might try the entire sequence under Matlab and see if the file always grows, or whether they are replacing variables.

The other thing to try is changing the b variable to something which is of a different size or type in Octave.  Repeat the first steps you did to create datafile.mat in Matlab.  But in Octave, try


b = repmat (i, [3,2]);
save -v7 -append datafile.mat b


Then try loading the file in Matlab.  If Matlab is really just using the first instance of a 'b' variable that it finds in the file then this will still be zeros(2,2).

Rik <rik5>
Group administrator
Tue 05 Jul 2016 03:33:05 PM UTC, comment #1: 

Thank you for contributing this bug report.

In Octave, the following


>> a = 23;
>> b = 24;
>> save foo.txt
>> b = 42;
>> save -append foo.txt b


results in a file that contains one copy of "a" and two copies of "b" with different values.

In Octave, loading from the file always retrieves the last value of "b", either with


>> load foo.txt
>> load foo.txt b


So the primary bug seems to be that Octave does not overwrite a variable with the same name in the file if the -append option is given. This is explicitly mentioned in Matlab's help for save at https://www.mathworks.com/help/matlab/ref/save.html.

Mike Miller <mtmiller>
Group Member
Tue 05 Jul 2016 12:26:43 PM UTC, original submission:  

In MATLAB R2015a (under Ubuntu 12.04) I created a mat file so that I could read it in Octave:

>> a=1;
>> save('datafile.mat', 'a', '-v7');


I then created a matrix and added it to the file:

>> b=zeros(2,2);
>> save('datafile.mat', 'b', '-append', '-v7');


In Octave 4.0.3 I loaded the file, changed the contents of b and saved the modified version to file.
load('datafile.mat');

> b(1,1)=1;
> save('datafile.mat', 'b', '-append', '-v7');


Back in MATLAB, I tried to load the variable from the file:

>> load('datafile.mat', 'b')
>> b

b =
     0     0
     0     0

It loads the first version of b.
If I load the full file, it loads correctly:

>> load('datafile.mat')
>> b

b =
     1     0
     0     0

Of course, one might argue that this is a MATLAB bug, but the same code in MATLAB produces the correct result.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-07-05 mtmiller Priority5 - Normal 3 - Low
        Release4.0.3 dev
    2016-07-05 rik5 Priority3 - Low 5 - Normal
        Releasedev 4.0.3
    2016-07-05 mtmiller Priority5 - Normal 3 - Low
        Release4.0.3 dev
    2016-07-05 mtmiller StatusNone Confirmed
    2016-07-05 mtmiller Operating SystemMicrosoft Windows Any
        SummaryChanging a variable and saving file with -append in Octave makes it impossible to load a particular variable in MATLAB save -append should overwrite a variable with the same name
    2016-07-05 rik5 Item GroupIncorrect Result Matlab Compatibility
        StatusConfirmed None
        Operating SystemAny Microsoft Windows
        Summarysave -append should overwrite a variable with the same name Changing a variable and saving file with -append in Octave makes it impossible to load a particular variable in MATLAB
    2016-07-05 mtmiller Severity3 - Normal 2 - Minor
        StatusNone Confirmed
        Operating SystemMicrosoft Windows Any
        SummaryChanging a variable and saving file with -append in Octave makes it impossible to load a particular variable in MATLAB save -append should overwrite a variable with the same name

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code