bugGNU nano - Bugs: bug #50406, reading in a large file is...

 
 

bug #50406: reading in a large file is relatively slow

Submitted by:  Benno Schulenberg <bens>
Submitted on:  Sun 26 Feb 2017 12:13:28 PM UTC  
 
Severity: 3 - NormalStatus: None
Assigned to: NoneOpen/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 12 Jan 2018 12:26:04 PM UTC, comment #2:

The attached patch decreased load time from 2 seconds to 0.7 seconds, with file cached for both reads. It is just a starting point if someone else wants to clean it up and test.

(file #42893)

Brand Huntsman <brand>
Fri 12 Jan 2018 11:16:35 AM UTC, comment #1:

What about allocating a buffer the size of the entire file, fread the file into buffer, iterate over the buffer instead of calling fgetc and then free the buffer.

A slightly more complex approach would be to allocate a small buffer, say 4k. At head of loop, if index > 4095, fread next chunk of file into buffer. Initialize index to 4096 so the first loop iteration fills the buffer. Larger buffer sizes should perform even better, but don't allocate a buffer larger than the file.

Brand Huntsman <brand>
Sun 26 Feb 2017 12:13:28 PM UTC, original submission:

To reproduce, create a file with a million lines of average length. For example by running in nano's source tree:

for x in $(seq 45); do cat src/*.c >>million; done;

Then open this file with 'nano --ignore' and count the seconds. It takes about twelve seconds (on a slow Atom processor). Then open the same file with Pico. Takes about ten seconds. That's comparable, but Pico is faster.

Opening the same file with joe is very fast: it takes a quarter of second. Nice editor (ne) is a bit slower: just under one second. Vim (when it has been started before) takes about one second. Emacs first asks if you really want to open this large file, and after saying yes, it takes about three seconds to load the file.

It would be nice if nano would be at least as fast as emacs, or at the very least faster than Pico.

Currently nano assembles a file byte by byte into its linestructs. It should be possible to speed this up by reading whole lines instead. The complication is: we will have to scan each read line for \r characters, to detect whether it is a Mac or DOS file. That will make things messy.

Benno Schulenberg <bens>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by brand (Posted a comment)
  • -unavailable- added by jahboater
  • -unavailable- added by sworddragon
  • -unavailable- added by bens (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 12 Jan 2018 12:26:04 PM UTCbrandAttached File-=>Added 0001-improve-file-read-time.patch, #42893
    Tue 23 May 2017 07:51:10 PM UTCjahboaterCarbon-Copy-=>Added jahboater
    Sun 26 Feb 2017 09:04:35 PM UTCsworddragonCarbon-Copy-=>Added sworddragon

    Back to the top


    Powered by Savane 3.1-cleanup1