taskGNU Astronomy Utilities - Tasks: task #14100, Script to configure and build in...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14100: Script to configure and build in tmpfs (RAM)

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sat 23 Jul 2016 11:35:20 PM UTC
   
 
Should Start On:  Sat 23 Jul 2016 03:00:00 PM UTC Should be Finished on:  Sat 23 Jul 2016 03:00:00 PM UTC
Category:  Installation Priority:  5 - Normal
Item Group:  Enhancement Status:  Done
Privacy:  Public Assigned to:  makhlaghi
Percent Complete:  100% Open/Closed:  Closed
Effort:  0.00

Jump to the original submission

Thu 04 Aug 2016 11:28:27 PM UTC, comment #7: 

After some further modifications, the branch is now merged with master and pushed to the main repository. There is now a new file in the source file and tar-ball called 'tmpfs-config-make' which the user can run with './tmpfs-config-make'.

This script is by no means final, I feel it is just ready for more thorough testing with the rest of Gnuastro. Further issues related to this script can be either submitted as bugs (if there is a problem) or task for possible improvements. Thanks again to Mosè for the great test and improvements in the script.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 28 Jul 2016 11:35:48 PM UTC, comment #6: 

This is really interesting/puzzlng!

I was expecting the process to be much faster when done on an SSD and even more on the RAM. I guess there is some bottle-neck somewhere.

So until we find the bottleneck (or see that there is no solution!), I will remove the speed effect in the text and mainly focus on the large number of read/writes and their effect on the longevity of HDD/SSD for people who develop a lot.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 28 Jul 2016 11:08:18 PM UTC, comment #5: 

How can I run configure in a clean workspace without bootstrapping?  I noticed the problem with the "build" link not being created just because I was in a clean workspace.  You should always start from there when you want to test new features ;-)

Regarding the type of my disk, I have a HDD (5400 rpm, not even the fastest one).

Mosè Giordano <giordano>
Group Member
Thu 28 Jul 2016 10:38:05 PM UTC, comment #4: 

Great test, thank you! I also tried it out. But I didn't do the bootstrapping because this script is only concerned with configure and make. Both were run after a 'make distclean'.


$ time (./configure &&make -j8)
...
real        0m31.752s
user        0m38.003s
sys        0m3.927s

-------------

$ time (./parallel-build-in-tmpfs)
...
real        0m29.958s
user        0m36.473s
sys        0m3.487s


For me it was just slightly (<2 sec) faster! I then thought it might be because of the source file reading speed bottle-neck. Since the source wasn't in the RAM. So I unpacked a 'make dist' tarball in the /dev/shm and ran configure and make there. The result wasn't any different (within errors) than the './parallel-build-in-tmpfs' result above:


real        0m30.685s
user        0m37.317s
sys        0m3.510s


I then checked the memory usage before and after the processes involving /dev/shm (to make sure that tmpfs was actually in the RAM, not SWAP) and I saw that indeed the files are in the RAM. So I am also confused!

The main thing I was concerned about when coming up with this script was the burden of a large number of read/writes on the SSD drive. I hadn't actually objectively tested the running time like the test above. I guess this shows how important prejudice is ;-)!

I am curious if you are also using an SSD or HDD? Since your running time is so similar to mine, I guess you are also using an SSD, right? Probably on a HDD the difference might be more. I can't test on a HDD now.

At least I feel it is good that the writings of small files are done in the RAM not on the SSD. But I am still confused how the running time difference is so small.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 28 Jul 2016 06:57:51 PM UTC, comment #3: 

I just pushed to my gitlab repository a fix: https://gitlab.com/mose/gnuastro/commit/620aaeef1b19da6c89e9a3924c9d4349b12af4a0

Honestly, I didn't find a boost in compilation time (actually it was the contrary):

% mount|grep /dev/shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

% git clean -fxd; GNULIB_SRCDIR=../gnulib ./bootstrap --copy&&time (./configure &&make -j8)
[...]
( ./configure && make -j8; )  32,51s user 3,19s system 116% cpu 30,670 total

% git clean -fxd; GNULIB_SRCDIR=../gnulib ./bootstrap --copy&&time ./parallel-build-in-tmpfs
[...]
./parallel-build-in-tmpfs  33,13s user 3,25s system 115% cpu 31,493 total


What am I doing wrong?

Mosè Giordano <giordano>
Group Member
Mon 25 Jul 2016 10:43:52 AM UTC, comment #2: 

The script will now make a symbolic link called "build" in the top source directory for easy access to the built files:

https://gitlab.com/makhlaghi/gnuastro/commit/2de6a041ed550123c51eac8af5a47330687880b3

If you get the chance to try it out, any feedback would be very welcome ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 23 Jul 2016 11:38:38 PM UTC, comment #1: 

Sorry, I forgot to mention that the name of the branch is 'parallel-build-in-tmpfs'.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 23 Jul 2016 11:35:20 PM UTC, original submission:  

The text below is taken from the comments of the suggested script explaining the purpose of this suggestion. The script has been put in my fork of Gnuastro (link below) for ideas, criticisms, and suggestions prior to merging it with master.

https://gitlab.com/makhlaghi/gnuastro/branches

------

The configure and build process involves the creation, reading, and modification of a large number of files (input/output, or I/O). Therefore I/O issues can directly affect the work of developers who need to configure and build Gnuastro numerous times. Some such issues are listed below:

  • I/O can be slow in on-volatile (non-RAM) memory like traditional hard disks (HDDs) and even SSDs.


  • I/O will cause wear and tear on both the HDDs (mechanical failures) and SSDs (decreasing the lifetime).


  • Having the built files mixed with the source files can greatly affect backing up (synchronization) of source files (since it involves the management of a large number of small files that are changed a lot.


One solution to these problems is the tmpfs file system (see https://en.wikipedia.org/wiki/Tmpfs). Any file in tmpfs is actually stored in the RAM, not on HDDs or SSDs. Therefore I/O can be much faster and will also not harm the non-volatile memory devices. However, due to the volatile nature of RAM, files in the tmpfs filesystem will be permanently lost after a power-off.

The modern GNU C library (and thus the Linux kernel) define the '/dev/shm' directory for this purpose (POSIX shared memory). Therefore this script takes '/dev/shm' as the default here (the value to TMPDIR) this should be fine for all modern Linux based OSs. This script will make a 'gnuastro' directory in TMPDIR, change to that directory and configure and build gnuastro in there to have fast and non-wear configures and builds.

After this script is run, change directory to $TMPDIR/gnuastro and run other 'make' commands (for example 'make pdf', 'make check', or 'make install') from there. In Emacs, the command to be run with the 'M-x compile' command can be changed to 'cd /dev/shm/gnuastro; make -kj8' (to build on 8 threads) instead of the default 'make -k'. In this manner, the 'M-x recompile' commands to be run later will also build in the RAM while you modify the source in your non-volatile HDD or SSD.

To further speed up the build process, this script will build ('make') Gnuastro on multiple threads (value to the 'NUM_THREADS' variable bellow). This script can also be used in any other source file that uses the GNU build system.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by giordano (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-08-04 makhlaghi StatusReady For Test Done
        Percent Complete30% 100%
        Open/ClosedOpen Closed
    2016-07-25 makhlaghi Percent Complete20% 30%

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code