bugGNU Octave - Bugs: bug #58127, -append does not automatically...

 
 

bug #58127: -append does not automatically detect/check file type for save

Submitter:  Liang Tang <lt1234>
Submitted:  Mon 06 Apr 2020 06:36:51 PM UTC
   
 
Category:  Octave Function Severity:  2 - Minor
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  lt1234 Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 21 Apr 2020 05:34:45 PM UTC, comment #9: 

Don't know if it's worth it, but one thing to do in the meantime would be to issue an error message if "-append" is used WITHOUT also specifying a file type.  The code for that is in the function parse_save_options located in the file libinterp/corefcn/load-save.cc.

The function is


string_vector
load_save_system::parse_save_options (const string_vector& argv,
                                      load_save_format& fmt, bool& append,
                                      bool& save_as_floats, bool& use_zlib)
{
#if ! defined (HAVE_ZLIB)
  octave_unused_parameter (use_zlib);
#endif

  string_vector retval;
  int argc = argv.numel ();

  bool do_double = false;
  bool do_tabs = false;

  for (int i = 0; i < argc; i++)
    {
      if (argv[i] == "-append")
        {
          append = true;
        }
      else if (argv[i] == "-ascii" || argv[i] == "-a")
        {
          fmt.set_type (MAT_ASCII);
        }
      else if (argv[i] == "-double")
        {
          do_double = true;
        }
      else if (argv[i] == "-tabs")
        {
          do_tabs = true;
        }
      else if (argv[i] == "-text" || argv[i] == "-t")
        {
          fmt.set_type (TEXT);
        }
      else if (argv[i] == "-binary" || argv[i] == "-b")
        {
          fmt.set_type (BINARY);
        }
      else if (argv[i] == "-hdf5" || argv[i] == "-h")
        {
#if defined (HAVE_HDF5)
          fmt.set_type (HDF5);
#else
          err_disabled_feature ("save", "HDF5");
#endif
        }
      else if (argv[i] == "-v7.3" || argv[i] == "-V7.3" || argv[i] == "-7.3")
        {
          error ("save: Matlab file format -v7.3 is not yet implemented");
        }
#if defined (HAVE_ZLIB)
      else if (argv[i] == "-v7" || argv[i] == "-V7" || argv[i] == "-7"
               || argv[i] == "-mat7-binary")
        {
          fmt.set_type (MAT7_BINARY);
        }
#endif
      else if (argv[i] == "-mat" || argv[i] == "-m"
               || argv[i] == "-v6" || argv[i] == "-V6" || argv[i] == "-6"
               || argv[i] == "-mat-binary")
        {
          fmt.set_type (MAT5_BINARY);
        }
      else if (argv[i] == "-v4" || argv[i] == "-V4" || argv[i] == "-4"
               || argv[i] == "-mat4-binary")
        {
          fmt.set_type (MAT_BINARY);
        }
      else if (argv[i] == "-float-binary" || argv[i] == "-f")
        {
          fmt.set_type (BINARY);
          save_as_floats = true;
        }
      else if (argv[i] == "-float-hdf5")
        {
#if defined (HAVE_HDF5)
          fmt.set_type (HDF5);
          save_as_floats = true;
#else
          err_disabled_feature ("save", "HDF5");
#endif
        }
#if defined (HAVE_ZLIB)
      else if (argv[i] == "-zip" || argv[i] == "-z")
        {
          use_zlib = true;
        }
#endif
      else if (argv[i] == "-struct")
        {
          retval.append (argv[i]);
        }
      else if (argv[i][0] == '-' && argv[i] != "-")
        {
          error ("save: Unrecognized option '%s'", argv[i].c_str ());
        }
      else
        retval.append (argv[i]);
    }


It's a reasonably straightforward change, although it would cause the code to grow quite a bit.  One would need to define a bool variable at the start, something like have_fmt, as false.  Then, in every else if branch where fmt.set_type is called you would need to set have_fmt to true.  At the end of this one could check whether "append" was true, but "have_fmt" was false and issue an error in that case.

Rik <rik5>
Group administrator
Tue 21 Apr 2020 03:55:31 AM UTC, comment #8: 

Since a workaround is to use the same file type for both save commands, if you want to use the -v7 format, then you can set it to be the default with save_default_options().  Putting the command


save_default_options ("-v7");


in your .octaverc will make this happen every time you start Octave.

With that default in place, this code works just fine


a=1;b=2;
unlink ('ab.mat');
save ab.mat a
save -append ab.mat b
clear a b
load ab.mat



Rik <rik5>
Group administrator
Tue 21 Apr 2020 03:38:41 AM UTC, comment #7: 

Marking as confirmed, but changing the Summary to reflect the specific issue.

When the file type for the original save and the file type of the '-append' save match, then the operation completes succesfully.  However, if they do not match then strange errors are thrown that don't immediately point back to this root cause.

Sample code


a=1;b=2;
unlink ('ab.mat');
save -v7 ab.mat a
save -text -append ab.mat b
clear a b
load ab.mat
a
b


This code fails with


error: load: invalid element type = 8227


However, if you specify matching formats then it works correctly


a=1;b=2;
unlink ('ab.mat');
save -v7 ab.mat a
save -v7 -append ab.mat b
clear a b
load ab.mat
a
b



Rik <rik5>
Group administrator
Mon 13 Apr 2020 03:05:48 PM UTC, comment #6: 

Hi Mike,

When I look again the writing we have so far.  I do recognize that the default of two codes are different, i.e., they are not meant to be compatible by design.  Therefore it really is not meaningful to continue with matlab compatibility as the focus.  I also would agree to your issue assessment in priority (low) and severity (minor) for general user population. 

If you agree, then the only issue left is  4.1:
 
a=1;b=1
save -V7 ab.mat a
save -append ab.mat b
load ab.mat
error: load: invalid element type = 8227

Either the 2nd save needs to be rejected or the load needs to bring the variables back to the workspace.  If you disagree with this as an issue, I would recommend to close this item.  Thanks.

Liang Tang <lt1234>
Thu 09 Apr 2020 05:10:44 PM UTC, comment #5: 

The Octave default save format is -text. That's why I asked what the Matlab behavior is when the preference has been changed to use -v6 as the default. Does it still append to a v7 file in v7 format, or does it now append in v6 format? Does -append also work with a v6 file without the -v6 option?

Mike Miller <mtmiller>
Group Member
Tue 07 Apr 2020 09:36:19 PM UTC, comment #4: 

Hi Mike,

I have provided two examples in my submission to demonstrate an error which only occurs in Octave and incompatibility of the two software in handling the file version when used together with -append.   I am using both codes in their default setting as much as I can.  The matlab default save version is v7.3. 

I am not using v6.  When v6 is used, the matlab behavior is different.  There is no warning message. 

>> a=1;b=1;
>> save -V6 ab.mat a
>> save -v6 -append ab.mat b


Both v7 & 7.3 behave the same.

>> a=1;b=1;
>> save -V6 ab.mat a
>> save -v6 -append ab.mat b
>> save -V7 ab.mat a
>> save -v7 -append ab.mat b

Warning: Ignoring the version specified. The version flag is not required when using the '-append' flag.

>> save -V7.3 ab.mat a
>> save -v7.3 -append ab.mat b

Warning: Ignoring the version specified. The version flag is not required when using the '-append' flag.


The work around I have is to suppress the warning message.  I don't need any help on this end.

Liang Tang <lt1234>
Tue 07 Apr 2020 09:03:34 PM UTC, comment #3: 

Sure, I understand the problem statement. Still hoping you can provide us with some more information so we can evaluate whether this is a bug and whether to do anything about it.

If you are instead just looking for advice or help on how to work around this difference in your code, then you should probably try the help mailing list, not the bug tracker.

Mike Miller <mtmiller>
Group Member
Tue 07 Apr 2020 12:08:39 PM UTC, comment #2: 

Thanks for the comments.  I would like to add some explanations.

(1) The purpose is to exchange saved files from Octave or Matlab for later re-used in both codes.  (If there is a better approach, I would appreciate your inputs.) 
(2) Given Matlab does not read Octave default save file, to get file compatibility between Octave and Matlab, I write the same Matlab file format in both Octave and Matlab. 
(3) -v7 is just one of options in writing the files that can be exchangeable between the two codes.  Other options may be possible, but I did not try.  I need just one to work consistently.  I am using both -v7 in octave and matlab for outputting.
(4) There are two issues I ran into.

(4.1) The file saved right below in Octave cannot be reload back in Octave when there is no error in writing the file. 
a=1;b=1
save -V7 ab.mat a
save -append ab.mat b
load ab.mat
error: load: invalid element type = 8227

The few lines above save and reload correctly in Matlab. 

(4.2) Compatibility.  The few lines below writes out a file can be loaded in Octave and Matlab correctly 
a=1;b=1
b =  1
save -V7 ab.mat a
save -v7 -append ab.mat b
load ab.mat

However the same executed in Matlab, you have a warning message:
a=1;b=1
b =  1
save -V7 ab.mat a
save -v7 -append ab.mat b
Warning: Ignoring the version specified. The version flag is not required when using the '-append' flag.
load ab.mat

I can either suppress the warning message or follow Matlab's warning to remove -v7 in the second save.  If I do the later, I am back to (4.1) issue.

Liang Tang <lt1234>
Mon 06 Apr 2020 10:52:11 PM UTC, comment #1: 

It's unclear from your description whether this is because '-v7' is the default in Matlab.

What happens if you open the Preferences in Matlab and set the default "MAT-File save format" to either v6 or v7.3? Are you still able to save a file with '-v7' and then append a variable to it with '-append'?

According to Matlab's help for the save command, if the file is in v6 format, you need to use both '-append' and '-v6'. If that's true, it doesn't seem that unreasonable to require the same in Octave with '-v7'.

But it might be a useful enhancement if Octave were able to detect and use the existing file format when saving with '-append'.

Mike Miller <mtmiller>
Group Member
Mon 06 Apr 2020 06:36:51 PM UTC, original submission:  

In matlab, -v7 is optional when -append is specified in save command. If you specify both, there is a warning message for -v7 usage. In octave, two options must be used.  If -v7 is missing with -append, error occurs.

% error

a=1;b=1
save -V7 ab.mat a
save -append ab.mat b
load ab.mat
error: load: invalid element type = 8227


% good

a=1;b=1
b =  1
save -V7 ab.mat a
save -v7 -append ab.mat b
load ab.mat

Liang Tang <lt1234>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by lt1234 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-21 rik5 StatusNeed Info Confirmed
        Release5.1.0 dev
        Operating SystemMicrosoft Windows Any
        Summarysave -V7 and -append error -append does not automatically detect/check file type for save
    2020-04-07 rik5 Carbon-CopyRemoved 72865 -
    2020-04-07 rik5 Priority5 - Normal 3 - Low
    2020-04-06 mtmiller Severity3 - Normal 2 - Minor
        StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-04b1.
    Corresponding source code