GNU nano - Bugs: bug #50406, reading in a large file is...
You are not allowed to post comments on this tracker with your current authentication level.
bug #50406: reading in a large file is relatively slow
Submitter: | Benno Schulenberg <bens> | ||
Submitted: | Sun 26 Feb 2017 12:13:28 PM UTC | ||
Severity: | 2 - Minor | Status: | Wont Fix |
Assigned to: | None | Open/Closed: | Closed |
Jump to the original submission
Sat 30 Nov 2019 05:01:58 PM UTC, comment #7: |
Benno Schulenberg <bens> |
Thu 12 Jul 2018 10:55:52 AM UTC, comment #6: A significant speed-up has been achieved by commit b2ff5746, from a patch by Devin Hussey. But... maybe there is more to be had? As the original submission suggested: would using getline() maybe be faster? (Why is there no getline_unlocked()?) |
Benno Schulenberg <bens> |
Sat 07 Jul 2018 08:09:21 PM UTC, comment #5: With file #44288, on my system the loading time of the million file goes down from 1.2 to .75 seconds. Such a 37% reduction of loading time is worth having. |
Benno Schulenberg <bens> |
Mon 04 Jun 2018 01:11:47 AM UTC, comment #4: Here is the fastest of 5 runs for each patch:
|
Brand Huntsman <brand> |
Sun 03 Jun 2018 06:44:59 PM UTC, comment #3: With commit cc2b19c8, the loading time of the "million" file is reduced from 3.5 to 1.2 seconds (on my current system). (The 3.5 seconds is for nano-2.9.5.) Measured in this way:
|
Benno Schulenberg <bens> |
Fri 12 Jan 2018 12:26:04 PM UTC, comment #2: The attached patch decreased load time from 2 seconds to 0.7 seconds, with file cached for both reads. It is just a starting point if someone else wants to clean it up and test. |
Brand Huntsman <brand> |
Fri 12 Jan 2018 11:16:35 AM UTC, comment #1: What about allocating a buffer the size of the entire file, fread the file into buffer, iterate over the buffer instead of calling fgetc and then free the buffer.
|
Brand Huntsman <brand> |
Sun 26 Feb 2017 12:13:28 PM UTC, original submission:
To reproduce, create a file with a million lines of average length. For example by running in nano's source tree:
|
Benno Schulenberg <bens> |
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 9 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2019-11-30 | bens | Attached File | - | Added 0001-drop-support-for-both-DOS-and-Mac-format.patch, #47951 | |
Attached File | - | Added 0002-use-getline-instead-of-getc.patch, #47952 | |||
Severity | 3 - Normal | 2 - Minor | |||
Status | None | Wont Fix | |||
Open/Closed | Open | Closed | |||
2018-06-04 | brand | Attached File | - | Added improve-file-read-time-using-a-4k-buffer.patch, #44288 | |
2018-01-12 | brand | Attached File | - | Added 0001-improve-file-read-time.patch, #42893 | |
2017-05-23 | jahboater | Carbon-Copy | - | Added jahboater | |
2017-02-26 | sworddragon | Carbon-Copy | - | Added sworddragon |
Powered by Savane 3.13-02a9.
Corresponding source code
With the attached two patches I've tested to see how much speedup could be had by using getline() instead of getc(). It makes a difference of roughly ten to twelve percent. I don't think that is worth dropping support for Mac files (DOS files could still be supported, but would require some work, which I didn't want to do for the test -- it would also reduce the advantage of the getline() case a little).
So... let's leave it as it is. The speedup from using the unlocked getc() was significant. It's good enough now.
(file #47951, file #47952)