bugGNU nano - Bugs: bug #50406, reading in a large file is...


You are not allowed to post comments on this tracker with your current authentication level.

bug #50406: reading in a large file is relatively slow

Submitted by:  Benno Schulenberg <bens>
Submitted on:  Sun 26 Feb 2017 12:13:28 PM UTC  
Severity:  2 - Minor Status:  Wont Fix
Assigned to:  None Open/Closed:  Closed

( Jump to the original submission)

Sat 30 Nov 2019 05:01:58 PM UTC, comment #7: 

With the attached two patches I've tested to see how much speedup could be had by using getline() instead of getc().  It makes a difference of roughly ten to twelve percent.  I don't think that is worth dropping support for Mac files (DOS files could still be supported, but would require some work, which I didn't want to do for the test -- it would also reduce the advantage of the getline() case a little).

So... let's leave it as it is.  The speedup from using the unlocked getc() was significant.  It's good enough now.

(file #47951, file #47952)

Benno Schulenberg <bens>
Project Administrator
Thu 12 Jul 2018 10:55:52 AM UTC, comment #6: 

A significant speed-up has been achieved by commit b2ff5746, from a patch by Devin Hussey.  But... maybe there is more to be had?  As the original submission suggested: would using getline() maybe be faster?  (Why is there no getline_unlocked()?)

Benno Schulenberg <bens>
Project Administrator
Sat 07 Jul 2018 08:09:21 PM UTC, comment #5: 

With file #44288, on my system the loading time of the million file goes down from 1.2 to .75 seconds.  Such a 37% reduction of loading time is worth having.

Benno Schulenberg <bens>
Project Administrator
Mon 04 Jun 2018 01:11:47 AM UTC, comment #4: 

Here is the fastest of 5 runs for each patch:
1.574s - git without cc2b19c8
0.585s - git with cc2b19c8
0.344s - git with cc2b19c8 and huge buffer (file #42893)
0.328s - git with cc2b19c8 and 4k buffer (attached)

Cutting the time in half is a huge gain.

The previous patch used a huge buffer to load the file because that was easiest to demonstrate how to do it and if it would improve load time. As stated below, I left it up to someone else to implement a smaller buffer.

The new patch uses a buffer of filesize or 4096, which ever is smaller. And for some reason, it is faster than the huge buffer. Changing buffer to 8k gives about the same results as 4k, usually a couple milliseconds worse.

(file #44288)

Brand Huntsman <brand>
Sun 03 Jun 2018 06:44:59 PM UTC, comment #3: 

With commit cc2b19c8, the loading time of the "million" file is reduced from 3.5 to 1.2 seconds (on my current system).  (The 3.5 seconds is for nano-2.9.5.)  Measured in this way:

   time  src/nano --ignore million

and typing ^X during loading.

Loading the entire file into memory and then reading it in from there is a rather wasteful use of memory.  On 8GB systems this may be acceptable, but on 1GB systems or less...

But in patch #9641, Devin Hussey is busy with a reasonably sized intermediate buffer.  But I doubt much can be gained over the above commit -- except on systems where each getc() call is really expensive.

Benno Schulenberg <bens>
Project Administrator
Fri 12 Jan 2018 12:26:04 PM UTC, comment #2: 

The attached patch decreased load time from 2 seconds to 0.7 seconds, with file cached for both reads. It is just a starting point if someone else wants to clean it up and test.

(file #42893)

Brand Huntsman <brand>
Fri 12 Jan 2018 11:16:35 AM UTC, comment #1: 

What about allocating a buffer the size of the entire file, fread the file into buffer, iterate over the buffer instead of calling fgetc and then free the buffer.

A slightly more complex approach would be to allocate a small buffer, say 4k. At head of loop, if index > 4095, fread next chunk of file into buffer. Initialize index to 4096 so the first loop iteration fills the buffer. Larger buffer sizes should perform even better, but don't allocate a buffer larger than the file.

Brand Huntsman <brand>
Sun 26 Feb 2017 12:13:28 PM UTC, original submission:  

To reproduce, create a file with a million lines of average length.  For example by running in nano's source tree:

   for x in $(seq 45); do cat src/*.c >>million; done;

Then open this file with 'nano --ignore' and count the seconds.  It takes about twelve seconds (on a slow Atom processor).  Then open the same file with Pico.  Takes about ten seconds.  That's comparable, but Pico is faster.

Opening the same file with joe is very fast: it takes a quarter of second.  Nice editor (ne) is a bit slower: just under one second.  Vim (when it has been started before) takes about one second.  Emacs first asks if you really want to open this large file, and after saying yes, it takes about three seconds to load the file.

It would be nice if nano would be at least as fast as emacs, or at the very least faster than Pico.

Currently nano assembles a file byte by byte into its linestructs.  It should be possible to speed this up by reading whole lines instead.  The complication is: we will have to scan each read line for \r characters, to detect whether it is a Mac or DOS file.  That will make things messy.

Benno Schulenberg <bens>
Project Administrator


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by brand (Posted a comment)
  • -email is unavailable- added by jahboater
  • -email is unavailable- added by sworddragon
  • -email is unavailable- added by bens (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-11-30 bens Attached File- => Added 0001-drop-support-for-both-DOS-and-Mac-format.patch, #47951
        Attached File- => Added 0002-use-getline-instead-of-getc.patch, #47952
        Severity3 - Normal => 2 - Minor
        StatusNone => Wont Fix
        Open/ClosedOpen => Closed
    2018-06-04 brand Attached File- => Added improve-file-read-time-using-a-4k-buffer.patch, #44288
    2018-01-12 brand Attached File- => Added 0001-improve-file-read-time.patch, #42893
    2017-05-23 jahboater Carbon-Copy- => Added jahboater
    2017-02-26 sworddragon Carbon-Copy- => Added sworddragon

    Back to the top

    Powered by Savane 3.5