bugGNU Octave - Bugs: bug #53899, [octave forge] (io) Octave crashes...


bug #53899: [octave forge] (io) Octave crashes when writing large dbf files with dbfwrite

Submitted by:  None
Submitted on:  Mon 14 May 2018 01:51:51 PM UTC  
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  4 Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  Philip Nienhuis <philipnienhuis>
Originator Name:  Matthew Parkan Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  other
Operating System:  Any

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Wed 28 Nov 2018 06:02:25 PM UTC, comment #10:


OF-mapping is waiting for a new geometry package release (that's a dependency of mapping).
I've bugged its maintainer but he's probably quite busy ATM.

Anyway I need to clean up a little bit in the mapping package - some Matlab-compatible functions I figured would be in mapping landed in geometry instead. Sometime these weeks I'll fix that.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 28 Nov 2018 10:43:19 AM UTC, comment #9:

Hi Philip,

thanks for the updates!

You are right about the fclose('all') and little endian specifiers.

Is any new release for the mapping package planned? A release with the updated shaperead/shapewrite would be really helpful at this point.

Have a nice day.

Matthew Parkan <mparkan>
Sun 28 Oct 2018 02:03:46 PM UTC, comment #8:

I've pushed your patch after a few edits here, crediting you and adding you into the copyright.

As announced I've amended the code for writing the data blob to do it in ~100 MB chunks, after I found that trying to write a heterogeneous cell array of 10,000,000x10 (code to create it attached, make_big_aa.m) gave dbf files that were missing the last 6 % or so from the data. That is, on my Core i5 8 GB RAM system with an SSD.

Thanks to your vectorized code (somewhat amended by me) writing time (tic...toc) of that big array went down from 4182.68 seconds to 532.211 seconds, not bad :-)

After that I've made various other improvements in the code.
A.o., I enclosed most of the writing code in an unwind_protect block to be able to properly close the file in case of errors. I suppose the motive behind you adding an "fclose (all)" statement was that you were plagued by zombie file handles that couldn't be accessed anymore after dbfwrite "crashed", let alone having no option to get those files deleted until after restarting Octave.

Again thanks very much for your contribution.

Closing report.

(file #45317)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sun 30 Sep 2018 11:38:08 AM UTC, comment #7:

Thanks for the patch Matthew, congrats with your PhD thesis.

Some preliminary questions & remarks about the patch:

1. Why is "fclose all" required? behind the users's back that may also close other files that need to stay open.

2. Why does the .dbf file need to be opened specifically in ieee-le mode and then add arch descriptors to every write stmt?

3. 2nd hunk, writing dates:

  • AFAIU uint8 needs no ieee-le arch, it's just one byte.
  • I suppose we'd need to subtract using ".-" rather than "-" to avoid broadcasting (-risk);
  • The next 4-byte sequences are just zero-filled placeholders, no need for ieee-le either.

I'll look into the data writing part later.
At first sight the vectorizing trick looks neat! but I'm always wary for overflow somewhere deep down, esp. in formatted I/O, so I'd rather break it up into some proper (yet big) buffer size. A cumsum of fldlngs may come in handy there.

Status => back into "In Progress"

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 18 Sep 2018 12:23:24 PM UTC, comment #6:

Hi Philip,

sorry for the very late reply, but I had to write my phd in the meantime...

After some more testing of dbfwrite, I think I've determined the problem. It seems that the crash is caused by the call to datevec(date) rather than by write speed. I've replaced it by:

While trying to find the bug, I also made a few changes to the code which might improve performance when writing large files (vectorizing data writing)

I've attached the updated version of dbfwrite and the corresponding patch to this comment.

(file #45033, file #45034)

Matthew Parkan <mparkan>
Sat 19 May 2018 02:51:13 PM UTC, comment #5:


Could you please test with attached dbfwrite.m?

I've come up with a simple buffer approach, see L.150 and up.
The buffer size can be easily adapted; I chose 4 K (matching approx. low level buffer size), maybe multiples of 4K work faster & better.

(file #44180)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 17 May 2018 09:24:26 PM UTC, comment #4:

For the moment I've patched dbfwrite.m along your suggestion in comment #1.

It may be that it is the single-byte write at the top of the inner loop is the straw that broke the camel's back. I'll look at this again at some later stage.

I'll set status to "Ready for test" and I'm lowering priority until I have more time for it, or someone beats me to it.

Thanks for reporting and suggesting the provisional fix.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Thu 17 May 2018 05:32:11 PM UTC, comment #3:

Hi Philip,

Indeed, the file is written to a SSD.

It seems that the problem is coming from writing the data records not the header of the DBF file.

I also think that it could be solved by avoiding the repeated fwrite calls in the double "for" loop. Vectorizing the code, as suggested in the "FIXME" comment, is likely a more robust solution than introducing pauses at each iteration.

N.B. I am using the latest shapewrite/read function from the mapping package repo

Tue 15 May 2018 06:51:20 PM UTC, comment #2:

Just FTR, is this when writing to a spinning HD or to an SSD ?
I'm asking because we've seen reports before about problem relating to write actions to SSD (in relation to zip & maybe gzip).

I never experienced this (yet) and I used to write huge shapefiles as well with tens of attributes and >50,000 shape entries. My systems (Linux and Windows) have SSDs but as of these days quite mediocre Core i5 CPUs. Maybe your system is much faster and stumbles over the low level stuff and HW.

In several functions in the io package I had to introduce delays along the line you sketch to overcome (what I interpret to be) lagging low-level/HW responses so I'm not at all surprised. The big question is how long such a delay should be in this case, as it is clearly system-dependent.

I do have plans to try collecting several writes into one bigger write action to help speed up dbfwrite.m, by reducing overhead in the low-level routines (= beyond Octave's control). Perhaps that would also solve this bug.

BTW hint: there's a much better shapewrite (and -read) in the mapping package repo.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Mon 14 May 2018 05:33:20 PM UTC, comment #1:

After some more testing, it seems the crash issue is not related to the size of the input but to the writing speed. It looks like including a small pause after each iteration in the data writing part of dbfwrite (after line 172) solves the problem:

Mon 14 May 2018 01:51:51 PM UTC, original submission:

Octave 4.4.0 crashes when trying to write a large structure (50 fields and ~28'000 entries resulting in a ~15 MB DBF file) to a DBF file with dbfwrite (io package v.2.4.11).

The same code works in Matlab r2017b.

Note: dbfwrite is called by the shapewrite function (from the mapping package).

Thanks for your help.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #45317:  make_big_aa.m added by philipnienhuis (799B - application/octet-stream)
file #45033:  dbfwrite.m added by mparkan (7KiB - text/plain)
file #45034:  dbfwrite.patch added by mparkan (5KiB - application/octet-stream)
file #44180:  dbfwrite.m added by philipnienhuis (6KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by mparkan (Updated the item)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-28 philipnienhuis StatusIn Progress => Fixed
        Open/ClosedOpen => Closed
    2018-10-28 philipnienhuis Attached File- => Added make_big_aa.m, #45317
        Priority3 - Low => 4
    2018-09-30 philipnienhuis Item GroupOther => Segfault, Bus Error, etc.
        StatusReady For Test => In Progress
    2018-09-28 philipnienhuis Assigned toNone => philipnienhuis
    2018-09-18 mparkan Attached File- => Added dbfwrite.m, #45033
        Attached File- => Added dbfwrite.patch, #45034
    2018-05-19 philipnienhuis Attached File- => Added dbfwrite.m, #44180
    2018-05-17 philipnienhuis Priority5 - Normal => 3 - Low
        StatusNeed Info => Ready For Test
        Release4.4.0 => other
        Operating SystemMicrosoft Windows => Any
    2018-05-15 philipnienhuis StatusNone => Need Info

    Back to the top

    Powered by Savane 3.4