bugGNU Octave - Bugs: bug #53899, [octave forge] (io) Octave crashes...

 
 

bug #53899: [octave forge] (io) Octave crashes when writing large dbf files with dbfwrite

Submitted by:  None
Submitted on:  Mon 14 May 2018 01:51:51 PM UTC  
 
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  3 - Low Item Group:  Other
Status:  Ready For Test Assigned to:  None
Originator Name:  Matthew Parkan Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  other
Operating System:  Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 19 May 2018 02:51:13 PM UTC, comment #5:

@Matthew,

Could you please test with attached dbfwrite.m?

I've come up with a simple buffer approach, see L.150 and up.
The buffer size can be easily adapted; I chose 4 K (matching approx. low level buffer size), maybe multiples of 4K work faster & better.

(file #44180)

Philip Nienhuis <philipnienhuis>
Project Member
Thu 17 May 2018 09:24:26 PM UTC, comment #4:

For the moment I've patched dbfwrite.m along your suggestion in comment #1.

It may be that it is the single-byte write at the top of the inner loop is the straw that broke the camel's back. I'll look at this again at some later stage.

I'll set status to "Ready for test" and I'm lowering priority until I have more time for it, or someone beats me to it.

Thanks for reporting and suggesting the provisional fix.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 17 May 2018 05:32:11 PM UTC, comment #3:

Hi Philip,

Indeed, the file is written to a SSD.

It seems that the problem is coming from writing the data records not the header of the DBF file.

I also think that it could be solved by avoiding the repeated fwrite calls in the double "for" loop. Vectorizing the code, as suggested in the "FIXME" comment, is likely a more robust solution than introducing pauses at each iteration.

N.B. I am using the latest shapewrite/read function from the mapping package repo

Anonymous
Tue 15 May 2018 06:51:20 PM UTC, comment #2:

Just FTR, is this when writing to a spinning HD or to an SSD ?
I'm asking because we've seen reports before about problem relating to write actions to SSD (in relation to zip & maybe gzip).

I never experienced this (yet) and I used to write huge shapefiles as well with tens of attributes and >50,000 shape entries. My systems (Linux and Windows) have SSDs but as of these days quite mediocre Core i5 CPUs. Maybe your system is much faster and stumbles over the low level stuff and HW.

In several functions in the io package I had to introduce delays along the line you sketch to overcome (what I interpret to be) lagging low-level/HW responses so I'm not at all surprised. The big question is how long such a delay should be in this case, as it is clearly system-dependent.

I do have plans to try collecting several writes into one bigger write action to help speed up dbfwrite.m, by reducing overhead in the low-level routines (= beyond Octave's control). Perhaps that would also solve this bug.

BTW hint: there's a much better shapewrite (and -read) in the mapping package repo.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 14 May 2018 05:33:20 PM UTC, comment #1:

After some more testing, it seems the crash issue is not related to the size of the input but to the writing speed. It looks like including a small pause after each iteration in the data writing part of dbfwrite (after line 172) solves the problem:

Anonymous
Mon 14 May 2018 01:51:51 PM UTC, original submission:

Octave 4.4.0 crashes when trying to write a large structure (50 fields and ~28'000 entries resulting in a ~15 MB DBF file) to a DBF file with dbfwrite (io package v.2.4.11).

The same code works in Matlab r2017b.

Note: dbfwrite is called by the shapewrite function (from the mapping package).

Thanks for your help.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44180:  dbfwrite.m added by philipnienhuis (6KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-19 philipnienhuis Attached File- => Added dbfwrite.m, #44180
    2018-05-17 philipnienhuis Priority5 - Normal => 3 - Low
        StatusNeed Info => Ready For Test
        Release4.4.0 => other
        Operating SystemMicrosoft Windows => Any
    2018-05-15 philipnienhuis StatusNone => Need Info

    Back to the top


    Powered by Savane 3.3