bugGNU Octave - Bugs: bug #39505, dlmwrite with coffset and roffset...

 
 

bug #39505: dlmwrite with coffset and roffset >0, inserts extra characters

Submitter:  imartinez <imartinez>
Submitted:  Wed 17 Jul 2013 06:22:35 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Invalid / Not an Octave Bug Assigned to:  None
Originator Name:  imartinez Open/Closed:  * Closed
Release:  * 3.6.3 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 19 Jul 2013 12:34:32 PM UTC, comment #17: 

Like Octave, Matlab does write the tabs on file, but you (and wordpad, and your microwave S2P program) don't see them because they're "whitespace" (if you don't know what that is look it up on google).

Friendly advice: please dump wordpad asap - use a real editor.
Hint: I find notepad++ to be very handy for editing even very large data files (and I run it in Linux under wine).

You did hit single vs. double quote issues in Octave, sorry for that; but that has been fixed now.
Double vs. single quotes is a bit of an Octave idiosyncrasy (or I'd rather say: a Matlab deficiency), you're not the first to stumble over it.

Anyway the second I press the "submit" button will be the last one I'll waste on this.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 17 Jul 2013 10:08:46 PM UTC, comment #16: 

Calm down, Philip Nienhuis.... it needs several cases to check
what is going on.

in matlab with tabs first and then with commas...

dlmwrite('matrix_test.txt',a,'-append','delimiter', '\t','roffset',0,'coffset',0);

>> dlmwrite('matrix_test.txt',b,'-append','delimiter', '\t','roffset',0,'coffset',0);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', '\t','roffset',4,'coffset',0);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', '\t','roffset',4,'coffset',4);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', ',','roffset',0,'coffset',0);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', ',','roffset',1,'coffset',0);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', ',','roffset',2,'coffset',0);
>> dlmwrite('matrix_test.txt',b,'-append','delimiter', ',','roffset',3,'coffset',0);
>>


Tabs are ok in matlab, empty lines are generate when you open the file in wordpad. This is what a microwave S2P file needs

I do recognize that with commas, it is inserting commas on the blank lines.

So, I think it still holds water at about 50%

1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
6 5 4 3 2
6 5 4 3 2
6 5 4 3 2




6 5 4 3 2
6 5 4 3 2
6 5 4 3 2




6 5 4 3 2
6 5 4 3 2
6 5 4 3 2
6,5,4,3,2
6,5,4,3,2
6,5,4,3,2
,,,,
6,5,4,3,2
6,5,4,3,2
6,5,4,3,2
,,,,
,,,,
6,5,4,3,2
6,5,4,3,2
6,5,4,3,2
,,,,
,,,,
,,,,
6,5,4,3,2
6,5,4,3,2
6,5,4,3,2

imartinez <imartinez>
Wed 17 Jul 2013 10:08:43 PM UTC, comment #15: 

I think this bug report is done.  Matlab and Octave both print out lines full of delimiters if the roffset option is used.  Along the way we at least found and fixed the '\t' interpretation issue.

Rik <rik5>
Group administrator
Wed 17 Jul 2013 09:57:57 PM UTC, comment #14: 

@comment #12 (ML r2013b prerel.):


>> dlmwrite ('tst.txt', magic (3), ',', 3, 2)
type tst.txt
csvwrite ('tst.csv', magic (3), 3, 2)
type tst.csv

,,,,
,,,,
,,,,
,,8,1,6
,,3,5,7
,,4,9,2

,,,,
,,,,
,,,,
,,8,1,6
,,3,5,7
,,4,9,2
>>


Philip Nienhuis <philipnienhuis>
Group Member
Wed 17 Jul 2013 09:48:49 PM UTC, comment #13: 

Mike, I fixed the interpretation of '\t' for dlmwrite (http://hg.savannah.gnu.org/hgweb/octave/rev/1a1e831fe6b4).  It was already being handled correctly for dlmread.

Rik <rik5>
Group administrator
Wed 17 Jul 2013 09:30:25 PM UTC, comment #12: 

I don't think there is any issue with coffset, although it is worth checking.

What file does the following code sample produce?


dlmwrite ('tst.txt', magic (3), ',', 3, 2)
type tst.txt
csvwrite ('tst.csv', magic (3), 3, 2)
type tst.csv


We need to check csvwrite because Octave uses dlmwrite internally for csvwrite.  I don't know whether it will also produce blank lines or not.

Rik <rik5>
Group administrator
Wed 17 Jul 2013 09:25:37 PM UTC, comment #11: 

(Trying again, my comment was cut off half-way....)

No I didn't read back in Octave, I just tried your original example in Octave with double quotes around \t. That gave a beautiful text file with -seemingly- empty lines - but read on.

Trying you original example in Matlab r2013b pre-release gives:


>>  a=[1 2 3 4 5;1 2 3 4 5;1 2 3 4 5]; >> dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','roffset',0,'coffset',0) Error using dlmwrite (line 113) Invalid attribute tag: on


so your example doesn't work as-is in Matlab; 'append' is not a valid user-configurable option in ML.

About your comment #4, did you literally try it in Matlab?
If I try (in ML r2013b pre-release):


>> dlmwrite('matrix_text2.txt',a,'delimiter', ',','roffset',5,'coffset',0)


I get a text file looking like


,,,,
,,,,
,,,,
,,,,
,,,,
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5


so your claim that ML writes empty lines corresponding to empty data rows in case of printable characters doesn't hold water. Not even for non-printable delimiters like tabs, my example in comment #5 gives the exact same results in ML (but with single quotes around \t in the dlmwrite argument list).

As to your comment #7, I don't follow. Matlab says:


dlmread fills empty delimited fields with zero.


All in all I think you'd better re-read the docs.
I tend to close this bug with "invalid" but I'll await Jordi's and Rik's opinions.

The "\t" versus '\t' thing is a different issue. Is this documented? If not it may be worth another bug report.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 17 Jul 2013 09:16:13 PM UTC, comment #10: 

No I didn't read back in Octave, I just tried your original example in Octave with double quotes around \t. That gave a beautiful text file with -seemingly- empty lines - but read on.

Trying you original example in Matlab r2013b pre-release gives:


>>  a=[1 2 3 4 5;1 2 3 4 5;1 2 3 4 5];
>> dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','roffset',0,'coffset',0)
Error using dlmwrite (line 113)
Invalid attribute tag: on


so your example doesn't work as-is in Matlab; 'append' is not a valid user-configurable option in ML.

About your comment #4, did you literally try it in Matlab?
If I try (in ML r2013b pre-release):


dlmwrite('matrix_text2.txt',a,'delimiter',',','roffset',5,'coffset',0)
-verbatim

I get:

+verbatim+
,,,,
,,,,
,,,,
,,,,
,,,,
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5


so your claim that ML writes blank lines for empty data rows in case of printable delimiters doesn't hold water. Not at all.
It does for tabs, but in a binary editor you'll see there are indeed tabs inserted for empty data rows, only thing is you don't see tabs directly as they are non-printable. In my comment #5 you do see them expressed as 8-space jumps.

As to comment #7, I don't follow. dlmread probably reads zeros as it returns a type double matrix, not a cell array. Read the docs about dlmread's behavior.
ML r2013b prr-release "help dlmread" says this literally (I assume I may quote small excerpts, Jordi?):


dlmread fills empty delimited fields with zero.



All in all I see nothing wrong with writing / reading empty data rows. I do think you've just misinterpreted the docs.
I tend to close this bug report with "invalid" but I'll await responses from Rik or Jordi.

The "\t" versus '\t' is another issue. Is this documented? If not, it may be worth another bug report.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 17 Jul 2013 09:01:32 PM UTC, comment #9: 

I now see in the MATLAB description that dlmread and dlmwrite both interpret the string '\t' as a tab character. I think we may simply need to handle this for compatibility the same way we're doing it in importdata.

http://hg.savannah.gnu.org/hgweb/octave/file/78f57b14535c/scripts/io/importdata.m#l303

Mike Miller <mtmiller>
Group Member
Wed 17 Jul 2013 08:44:13 PM UTC, comment #8: 

Also jumping in...

Reading through the comments and running the provided snippets, there are two potential problems here, and I'm also unclear which one the reporter is claiming as the bug.

1. OP's comment 4 seems to indicate that a positive "roffset" inserts blank lines in MATLAB rather than lines with back-to-back delimiters. I cannot verify this at the moment, and it's unclear to me from the API documentation whether the offset lines should be blank or filled with delimiters. It's also unclear whether the OP meant that the line appeared blank in a text editor but was actually filled with tab characters.

2. There is definitely some inconsistency in dlmwrite when a single-quoted string is used with what would be an escape sequence if it were double-quoted, see below.


octave:1> dlmwrite ("test.txt", ones (3), "delimiter", "\t", "roffset", 2);
octave:2> system ("cat -T test.txt");
^I^I
^I^I
1^I1^I1
1^I1^I1
1^I1^I1
octave:3> dlmwrite ("test.txt", ones (3), "delimiter", '\t', "roffset", 2);
octave:4> system ("cat -T test.txt");
\t\t
\t\t
1^I1^I1
1^I1^I1
1^I1^I1


I'm using cat -T so the actual tab characters are shown as ^I, while "\t" is two characters.

Mike Miller <mtmiller>
Group Member
Wed 17 Jul 2013 08:43:37 PM UTC, comment #7: 

Those zeros are not supposed to be there.....
They are supposed to be
,,,,,,,,,

imartinez <imartinez>
Wed 17 Jul 2013 08:34:26 PM UTC, comment #6: 

Are you reading the file back on octave??

I am writing on octave and opening the file in a text editor to check it.

if you read back on octave, it looks ok

x = dlmread ('matrix_text.txt', ',')
x =

        2        3        4        5        6
        2        3        4        5        6
        2        3        4        5        6
        0        0        0        0        0
        0        0        0        0        0
        1        2        3        4        5
        1        2        3        4        5
        1        2        3        4        5
        0        0        0        0        0
        0        0        0        0        0
        0        0        0        0        0
        0        0        0        0        0
        1        2        3        4        5
        1        2        3        4        5
        1        2        3        4        5

octave does not have problems because the extra characters are read are valid empty lines.

READ THE NEXT STATEMENT CAREFULLY

if you open the file in an EXTERNAL TEXT EDITOR (like KWRITE)

2,3,4,5,6
2,3,4,5,6
2,3,4,5,6
,,,,
,,,,
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
,,,,
,,,,
,,,,
,,,,
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5


If the file goes to an external application, then the little ,,,,,
are a problem.

it is HIS code

imartinez <imartinez>
Wed 17 Jul 2013 07:17:36 PM UTC, comment #5: 

PMFJI

Do tabs need to be entered between double quotes rather than single quotes to write tab characters? On my WinXP box with octave-3.7.5 that is indeed the case.
Maybe that is the OP's problem.

If I run his example in his (her?) original post with "\t" rather than '\t', I get (hopefully savannah leaves tab formatting intact):


1        2        3        4        5
1        2        3        4        5
1        2        3        4        5
2        3        4        5        6
2        3        4        5        6
2        3        4        5        6

1        2        3        4        5
1        2        3        4        5
1        2        3        4        5


1        2        3        4        5
1        2        3        4        5
1        2        3        4        5



1        2        3        4        5
1        2        3        4        5
1        2        3        4        5



                        1        2        3        4        5
                        1        2        3        4        5
                        1        2        3        4        5


Philip Nienhuis <philipnienhuis>
Group Member
Wed 17 Jul 2013 06:54:00 PM UTC, comment #4: 


>>a=[1 2 3 4 5;1 2 3 4 5;1 2 3 4 5];



>> b=[2 3 4 5 6;2 3 4 5 6;2 3 4 5 6]

b =

        2        3        4        5        6
        2        3        4        5        6
        2        3        4        5        6

>>dlmwrite('matrix_text.txt',b,'-append','on','delimiter', ',','roffset',0,'coffset',0);


%leave or try to leave a blank line

>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','roffset',2,'coffset',0);


2,3,4,5,6
2,3,4,5,6
2,3,4,5,6
,,,,                    (this is supposed to be blank)
,,,,                    (this is also supposed to be blank)  
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5

MATLAB:

2,3,4,5,6
2,3,4,5,6
2,3,4,5,6

                        (BLANK as expected) 
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5

imartinez <imartinez>
Wed 17 Jul 2013 06:41:09 PM UTC, comment #3: 

Bug was re-opened.  What is the actual problem?  dlmwrite seems to be respecting the documentation regarding inserting delimiters if roffset or coffset is used.  The Mathworks documentation is here (http://www.mathworks.com/help/matlab/ref/dlmwrite.html).

I changed the sample code to use a comma as the delimiter so that I can see what is going on.


a=[1 2 3 4 5;1 2 3 4 5;1 2 3 4 5];

dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','roffset',0,'coffset',0);

b=[2 3 4 5 6;2 3 4 5 6;2 3 4 5 6];

dlmwrite('matrix_text.txt',b,'-append','on','delimiter', ',','roffset',0,'coffset',0);

dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','roffset',1,'coffset',0);

dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','roffset',2,'coffset',0);

dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','newline','unix','roffset',3,'coffset',0);

dlmwrite('matrix_text.txt',a,'-append','on','delimiter', ',','newline','unix','roffset',4,'coffset',3);


If I then read the file back in with dlmread all of the test matrices are in their proper positions.

"z

x = dlmread ('matrix_text.txt', ',')
x =

   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   2   3   4   5   6   0   0   0
   2   3   4   5   6   0   0   0
   2   3   4   5   6   0   0   0
   0   0   0   0   0   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   1   2   3   4   5   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   0   0   0   0   0
   0   0   0   1   2   3   4   5
   0   0   0   1   2   3   4   5
   0   0   0   1   2   3   4   5


I'm attaching the actual file generated as matrix_text.octave.txt.  If you have access to Matlab you could run the code in this comment and compare the generated file to matrix_text.octave.txt.  If there is a difference then we can look into it, but Octave appears to be following the documentation correctly.


(file #28606)

Rik <rik5>
Group administrator
Wed 17 Jul 2013 05:23:24 PM UTC, comment #2: 

This bug report was followed in bug #39511. I have requested the reporter to continue here instead.

Jordi GutiƩrrez Hermoso <jordigh>
Group Member
Wed 17 Jul 2013 03:34:35 PM UTC, comment #1: 

Which part is a bug?  roffset and coffset are used to position the matrix being written in the destination file.  For example, roffset=1 means one entire line of delimiters will be written.  A coffset=2 means 2 delimiters will be prepended to every row written out.

Example


M = [1 2; 3 4];
dlmwrite ('tst.txt', M, ',');
dlmwrite ('tst.txt', M, '-append', 'delimiter', ',', 'roffset', 3, 'coffset', 2);
type tst.txt
tst.txt is the user-defined function defined from: ./tst.txt

1,2
3,4
,,,
,,,
,,,
,,1,2
,,3,4

M2 = dlmread ('tst.txt')
M2 =

   1   2   0   0
   3   4   0   0
   0   0   0   0
   0   0   0   0
   0   0   0   0
   0   0   1   2
   0   0   3   4

Notice how the second matrix portion has been offset from the first as specified by roffset and coffset.

Rik <rik5>
Group administrator
Wed 17 Jul 2013 06:22:35 AM UTC, original submission:  

I think this is a bug... I ran a few things on the command line

dlmwrite with roffset and coffset =0 seems ok

with roffset > 0

it is inserting tabs characters..

code

>> a=[1 2 3 4 5;1 2 3 4 5;1 2 3 4 5];


>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','roffset',0,'coffset',0);


>> b=[2 3 4 5 6;2 3 4 5 6;2 3 4 5 6];


>>dlmwrite('matrix_text.txt',b,'-append','on','delimiter', '\t','roffset',0,'coffset',0);


>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','roffset',1,'coffset',0);


>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','roffset',2,'coffset',0);


>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','newline','unix','roffset',3,'coffset',0);


>>dlmwrite('matrix_text.txt',a,'-append','on','delimiter', '\t','newline','unix','roffset',3,'coffset',3);


file looks like:

1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
\t\t\t\t
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
\t\t\t\t
\t\t\t\t
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
\t\t\t\t
\t\t\t\t
\t\t\t\t
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
\t\t\t\t\t\t\t
\t\t\t\t\t\t\t
\t\t\t\t\t\t\t
                        1 2 3 4 5
                        1 2 3 4 5
                        1 2 3 4 5

imartinez <imartinez>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #28606:  matrix_text.octave.txt added by rik5 (243B - text/plain)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by imartinez (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-07-17 rik5 StatusConfirmed Invalid / Not an Octave Bug
        Open/ClosedOpen Closed
    2013-07-17 rik5 Attached File- Added matrix_text.octave.txt, #28606
    2013-07-17 jordigh Open/ClosedClosed Open
    2013-07-17 jordigh StatusInvalid / Not an Octave Bug Confirmed
        Dependencies- Depends on bugs #39511
    2013-07-17 rik5 StatusNone Invalid / Not an Octave Bug
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code