bugGNU nano - Bugs: bug #50406, reading in a large file is...

 
 

bug #50406: reading in a large file is relatively slow

Submitted by:  Benno Schulenberg <bens>
Submitted on:  Sun 26 Feb 2017 12:13:28 PM UTC  
 
Severity:  3 - Normal Status:  None
Assigned to:  None Open/Closed:  Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 12 Jul 2018 10:55:52 AM UTC, comment #6:

A significant speed-up has been achieved by commit b2ff5746, from a patch by Devin Hussey. But... maybe there is more to be had? As the original submission suggested: would using getline() maybe be faster? (Why is there no getline_unlocked()?)

Benno Schulenberg <bens>
Project Administrator
Sat 07 Jul 2018 08:09:21 PM UTC, comment #5:

With file #44288, on my system the loading time of the million file goes down from 1.2 to .75 seconds. Such a 37% reduction of loading time is worth having.

Benno Schulenberg <bens>
Project Administrator
Mon 04 Jun 2018 01:11:47 AM UTC, comment #4:

Here is the fastest of 5 runs for each patch:
1.574s - git without cc2b19c8
0.585s - git with cc2b19c8
0.344s - git with cc2b19c8 and huge buffer (file #42893)
0.328s - git with cc2b19c8 and 4k buffer (attached)

Cutting the time in half is a huge gain.

The previous patch used a huge buffer to load the file because that was easiest to demonstrate how to do it and if it would improve load time. As stated below, I left it up to someone else to implement a smaller buffer.

The new patch uses a buffer of filesize or 4096, which ever is smaller. And for some reason, it is faster than the huge buffer. Changing buffer to 8k gives about the same results as 4k, usually a couple milliseconds worse.

(file #44288)

Brand Huntsman <brand>
Sun 03 Jun 2018 06:44:59 PM UTC, comment #3:

With commit cc2b19c8, the loading time of the "million" file is reduced from 3.5 to 1.2 seconds (on my current system). (The 3.5 seconds is for nano-2.9.5.) Measured in this way:

time src/nano --ignore million

and typing ^X during loading.

Loading the entire file into memory and then reading it in from there is a rather wasteful use of memory. On 8GB systems this may be acceptable, but on 1GB systems or less...

But in patch #9641, Devin Hussey is busy with a reasonably sized intermediate buffer. But I doubt much can be gained over the above commit -- except on systems where each getc() call is really expensive.

Benno Schulenberg <bens>
Project Administrator
Fri 12 Jan 2018 12:26:04 PM UTC, comment #2:

The attached patch decreased load time from 2 seconds to 0.7 seconds, with file cached for both reads. It is just a starting point if someone else wants to clean it up and test.

(file #42893)

Brand Huntsman <brand>
Fri 12 Jan 2018 11:16:35 AM UTC, comment #1:

What about allocating a buffer the size of the entire file, fread the file into buffer, iterate over the buffer instead of calling fgetc and then free the buffer.

A slightly more complex approach would be to allocate a small buffer, say 4k. At head of loop, if index > 4095, fread next chunk of file into buffer. Initialize index to 4096 so the first loop iteration fills the buffer. Larger buffer sizes should perform even better, but don't allocate a buffer larger than the file.

Brand Huntsman <brand>
Sun 26 Feb 2017 12:13:28 PM UTC, original submission:

To reproduce, create a file with a million lines of average length. For example by running in nano's source tree:

for x in $(seq 45); do cat src/*.c >>million; done;

Then open this file with 'nano --ignore' and count the seconds. It takes about twelve seconds (on a slow Atom processor). Then open the same file with Pico. Takes about ten seconds. That's comparable, but Pico is faster.

Opening the same file with joe is very fast: it takes a quarter of second. Nice editor (ne) is a bit slower: just under one second. Vim (when it has been started before) takes about one second. Emacs first asks if you really want to open this large file, and after saying yes, it takes about three seconds to load the file.

It would be nice if nano would be at least as fast as emacs, or at the very least faster than Pico.

Currently nano assembles a file byte by byte into its linestructs. It should be possible to speed this up by reading whole lines instead. The complication is: we will have to scan each read line for \r characters, to detect whether it is a Mac or DOS file. That will make things messy.

Benno Schulenberg <bens>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by brand (Posted a comment)
  • -email is unavailable- added by jahboater
  • -email is unavailable- added by sworddragon
  • -email is unavailable- added by bens (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-06-04 brand Attached File- => Added improve-file-read-time-using-a-4k-buffer.patch, #44288
    2018-01-12 brand Attached File- => Added 0001-improve-file-read-time.patch, #42893
    2017-05-23 jahboater Carbon-Copy- => Added jahboater
    2017-02-26 sworddragon Carbon-Copy- => Added sworddragon

    Back to the top


    Powered by Savane 3.3