bugGNU Octave - Bugs: bug #59238, load variables from file: the next...

 
 

bug #59238: load variables from file: the next saving operation produces a faulty file

Submitted by:  T Knauss <tknauss>
Submitted on:  Thu 08 Oct 2020 09:25:04 PM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  5.2.0 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 20 Oct 2020 10:12:28 AM UTC, comment #9: 

Sorry, I misunderstood what I was supposed to test.
Saving a double variable with the following commands, also leads to a file with CRLF line endings on Windows:

A = rand(4);
save A_text.txt A -text

Loading that file (again on Windows) successfully restores all matrix elements of A.

The issue seems to be with saving and loading cell strings in "-text" format if the line ending is CRLF.

An example that reproduces the issue for me on Windows is:

cell_str = {"This\nis\na\ntest."};
save cell_str_text.txt cell_str -text
cell_str2 = cell_str;
load cell_str_text.txt

The length of the string as marked in `cell_str_text.txt` is 15. But because the string was saved with CRLF line endings, 18 bytes have to be read from the file.

I see the following possible solutions:
- When saving, write the string literal in the file with LF line endings. This might be awkward when displaying the file in editors that have issues with mixed line endings in the same file.
- When saving, increase "length" by the number of additional bytes for the CR characters.
- When loading, count CRLF as one byte only.

Afaict, this is an issue in core Octave. There's not much the "symbolic" package can do about it.

Markus Mützel <mmuetzel>
Project Member
Tue 20 Oct 2020 07:23:01 AM UTC, comment #8: 

In my opinion, Octave should be allowed to use whatever line-ending  it wants to use, which means, any of CR and LF and CRLF should be allowed. You may define a line-ending that should be used, but this does not mean that it has to be used always.

The load operation on the other hand should be as flexible as possible and recognize all 3 types, which should be quite simple. This also would enable exchanging saved files between different operating systems and easy manual editing of the saved file on any operating system.

T Knauss <tknauss>
Mon 19 Oct 2020 09:45:53 PM UTC, comment #7: 

You can see from the attachments that the file is saved in '-text' format, not '-ascii'. I don't remember how Octave's load and save commands hook into user defined classes. So my earlier questions are about how 'save -text' is expected to work on Windows, and does Octave take care of all line ending translations or are user defined classes expected to handle that? It certainly looks like the load command is truncating too many characters when handing some text off to initialize a symbolic variable.

Mike Miller <mtmiller>
Project Administrator
Mon 19 Oct 2020 09:00:08 PM UTC, comment #6: 

It's still not clear to me what save() command the symbolic package is using.  It might be -ascii, or it might not.

I don't want to go down a false path and investigate the -ascii format if it is not being used.

Rik <rik5>
Project Administrator
Mon 19 Oct 2020 07:56:36 PM UTC, comment #5: 

The default line ending on Windows is CRLF. The following example produces a file with CRLF line ending on Windows:

A = rand(4);
save A_test.txt A -ascii

Having written that: It's also quite common to come across files with LF line ending on Windows. Most editors (including in the meantime also the built-in Notepad) display such files just fine.

Markus Mützel <mmuetzel>
Project Member
Tue 13 Oct 2020 09:20:45 PM UTC, comment #4: 

I don't have the symbolic package nor can I run on Windows.  But, I think I can help narrow down the issue.

What format does the symbolic use to save its variables?  That should be in the load/save routines for the package.  If it is using "-ascii" that would certainly be a problem.  If the save command does not specify anything then it will be whatever the user has set with save_default_options().  That will certainly create portability problems.  It might be best to explicitly specify "-binary" when saving since that format does not depend on whatever value a newline character is.

Rik <rik5>
Project Administrator
Mon 12 Oct 2020 09:12:06 PM UTC, comment #3: 

I think the load function truncates the variables when the file in this example has CRLF line endings. If I test with the original attached file, the variables are truncated after loading. If I strip all carriage returns first, then the load and save seem to work correctly.

Is it expected behavior that text format files have CRLF line endings with Octave on Windows? Or is this file considered malformed because it has the wrong line ending? Or if it is valid, is the bug in Octave or in the way the symbolic package creates its variables from text data?

Mike Miller <mtmiller>
Project Administrator
Fri 09 Oct 2020 11:57:58 AM UTC, comment #2: 

Yes, it also happens from CLI.

T Knauss <tknauss>
Fri 09 Oct 2020 05:32:19 AM UTC, comment #1: 

Thank you for the bug report.

Does this also happen in the command line version of octave? You have chosen "GUI" as category for this bug but I do not think that the GUI is involved here.

Torsten Lilge <ttl>
Project Member
Thu 08 Oct 2020 09:25:04 PM UTC, original submission:  

I am using "symbolic" package 2.9.0 and I have installed python 3.9 with sympy 1.5.1 (sympy 1.6.2 fails to work) on Win 10 Enterprise 1809 x64.

I discovered the failure when saving my symbolic matrices, but I think it's independent from the "symbolic" package.

Run these commands:
load transformationVars
save transformationVars2

Then compare the two files, e.g. with Beyond Compare. You'll see that the last element in many (or all?) matrices is missing.
The files are attached.

T Knauss <tknauss>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49945:  transformationVars added by tknauss (23KiB - application/octet-stream)
file #49946:  transformationVars2 added by tknauss (23KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by tknauss (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-12 mtmiller CategoryGUI => Octave Function
        StatusNone => Confirmed
    2020-10-08 tknauss Attached File- => Added transformationVars2, #49946
    2020-10-08 tknauss Attached File- => Added transformationVars, #49945

    Back to the top


    Powered by Savane 3.5