bugGNU Octave - Bugs: bug #53899, [octave forge] (io) Octave crashes...

 
 

bug #53899: [octave forge] (io) Octave crashes when writing large dbf files with dbfwrite

Submitted by:  None
Submitted on:  Mon 14 May 2018 01:51:51 PM UTC  
 
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  3 - Low Item Group:  Segfault, Bus Error, etc.
Status:  In Progress Assigned to:  Philip Nienhuis <philipnienhuis>
Originator Name:  Matthew Parkan Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  other
Operating System:  Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 30 Sep 2018 11:38:08 AM UTC, comment #7:

Thanks for the patch Matthew, congrats with your PhD thesis.

Some preliminary questions & remarks about the patch:

1. Why is "fclose all" required? behind the users's back that may also close other files that need to stay open.

2. Why does the .dbf file need to be opened specifically in ieee-le mode and then add arch descriptors to every write stmt?

3. 2nd hunk, writing dates:

  • AFAIU uint8 needs no ieee-le arch, it's just one byte.
  • I suppose we'd need to subtract using ".-" rather than "-" to avoid broadcasting (-risk);
  • The next 4-byte sequences are just zero-filled placeholders, no need for ieee-le either.

I'll look into the data writing part later.
At first sight the vectorizing trick looks neat! but I'm always wary for overflow somewhere deep down, esp. in formatted I/O, so I'd rather break it up into some proper (yet big) buffer size. A cumsum of fldlngs may come in handy there.

Status => back into "In Progress"

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 18 Sep 2018 12:23:24 PM UTC, comment #6:

Hi Philip,

sorry for the very late reply, but I had to write my phd in the meantime...

After some more testing of dbfwrite, I think I've determined the problem. It seems that the crash is caused by the call to datevec(date) rather than by write speed. I've replaced it by:

While trying to find the bug, I also made a few changes to the code which might improve performance when writing large files (vectorizing data writing)

I've attached the updated version of dbfwrite and the corresponding patch to this comment.

(file #45033, file #45034)

Matthew Parkan <mparkan>
Sat 19 May 2018 02:51:13 PM UTC, comment #5:

@Matthew,

Could you please test with attached dbfwrite.m?

I've come up with a simple buffer approach, see L.150 and up.
The buffer size can be easily adapted; I chose 4 K (matching approx. low level buffer size), maybe multiples of 4K work faster & better.

(file #44180)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 17 May 2018 09:24:26 PM UTC, comment #4:

For the moment I've patched dbfwrite.m along your suggestion in comment #1.

It may be that it is the single-byte write at the top of the inner loop is the straw that broke the camel's back. I'll look at this again at some later stage.

I'll set status to "Ready for test" and I'm lowering priority until I have more time for it, or someone beats me to it.

Thanks for reporting and suggesting the provisional fix.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 17 May 2018 05:32:11 PM UTC, comment #3:

Hi Philip,

Indeed, the file is written to a SSD.

It seems that the problem is coming from writing the data records not the header of the DBF file.

I also think that it could be solved by avoiding the repeated fwrite calls in the double "for" loop. Vectorizing the code, as suggested in the "FIXME" comment, is likely a more robust solution than introducing pauses at each iteration.

N.B. I am using the latest shapewrite/read function from the mapping package repo

Anonymous
Tue 15 May 2018 06:51:20 PM UTC, comment #2:

Just FTR, is this when writing to a spinning HD or to an SSD ?
I'm asking because we've seen reports before about problem relating to write actions to SSD (in relation to zip & maybe gzip).

I never experienced this (yet) and I used to write huge shapefiles as well with tens of attributes and >50,000 shape entries. My systems (Linux and Windows) have SSDs but as of these days quite mediocre Core i5 CPUs. Maybe your system is much faster and stumbles over the low level stuff and HW.

In several functions in the io package I had to introduce delays along the line you sketch to overcome (what I interpret to be) lagging low-level/HW responses so I'm not at all surprised. The big question is how long such a delay should be in this case, as it is clearly system-dependent.

I do have plans to try collecting several writes into one bigger write action to help speed up dbfwrite.m, by reducing overhead in the low-level routines (= beyond Octave's control). Perhaps that would also solve this bug.

BTW hint: there's a much better shapewrite (and -read) in the mapping package repo.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Mon 14 May 2018 05:33:20 PM UTC, comment #1:

After some more testing, it seems the crash issue is not related to the size of the input but to the writing speed. It looks like including a small pause after each iteration in the data writing part of dbfwrite (after line 172) solves the problem:

Anonymous
Mon 14 May 2018 01:51:51 PM UTC, original submission:

Octave 4.4.0 crashes when trying to write a large structure (50 fields and ~28'000 entries resulting in a ~15 MB DBF file) to a DBF file with dbfwrite (io package v.2.4.11).

The same code works in Matlab r2017b.

Note: dbfwrite is called by the shapewrite function (from the mapping package).

Thanks for your help.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45033:  dbfwrite.m added by mparkan (7KiB - text/plain)
file #45034:  dbfwrite.patch added by mparkan (5KiB - application/octet-stream)
file #44180:  dbfwrite.m added by philipnienhuis (6KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mparkan (Updated the item)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-09-30 philipnienhuis Item GroupOther => Segfault, Bus Error, etc.
        StatusReady For Test => In Progress
    2018-09-28 philipnienhuis Assigned toNone => philipnienhuis
    2018-09-18 mparkan Attached File- => Added dbfwrite.m, #45033
        Attached File- => Added dbfwrite.patch, #45034
    2018-05-19 philipnienhuis Attached File- => Added dbfwrite.m, #44180
    2018-05-17 philipnienhuis Priority5 - Normal => 3 - Low
        StatusNeed Info => Ready For Test
        Release4.4.0 => other
        Operating SystemMicrosoft Windows => Any
    2018-05-15 philipnienhuis StatusNone => Need Info

    Back to the top


    Powered by Savane 3.3