taskGNU Astronomy Utilities - Tasks: task #14100, Script to configure and build in...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14100: Script to configure and build in tmpfs (RAM)

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Sat 23 Jul 2016 11:35:20 PM UTC  
 
Should Start On: Sat 23 Jul 2016 03:00:00 PM UTCShould be Finished on: Sat 23 Jul 2016 03:00:00 PM UTC
Category: InstallationPriority: 5 - Normal
Item Group: EnhancementStatus: Done
Privacy: PublicPercent Complete: 100%
Assigned to: Mohammad Akhlaghi <makhlaghi>Open/Closed: Closed
Effort: 0.00

(Jump to the original submission Jump to the original submission)

Thu 04 Aug 2016 11:28:27 PM UTC, comment #7:

After some further modifications, the branch is now merged with master and pushed to the main repository. There is now a new file in the source file and tar-ball called 'tmpfs-config-make' which the user can run with './tmpfs-config-make'.

This script is by no means final, I feel it is just ready for more thorough testing with the rest of Gnuastro. Further issues related to this script can be either submitted as bugs (if there is a problem) or task for possible improvements. Thanks again to Mosè for the great test and improvements in the script.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Thu 28 Jul 2016 11:35:48 PM UTC, comment #6:

This is really interesting/puzzlng!

I was expecting the process to be much faster when done on an SSD and even more on the RAM. I guess there is some bottle-neck somewhere.

So until we find the bottleneck (or see that there is no solution!), I will remove the speed effect in the text and mainly focus on the large number of read/writes and their effect on the longevity of HDD/SSD for people who develop a lot.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Thu 28 Jul 2016 11:08:18 PM UTC, comment #5:

How can I run configure in a clean workspace without bootstrapping? I noticed the problem with the "build" link not being created just because I was in a clean workspace. You should always start from there when you want to test new features ;-)

Regarding the type of my disk, I have a HDD (5400 rpm, not even the fastest one).

Mosè Giordano <giordano>
Project Member
Thu 28 Jul 2016 10:38:05 PM UTC, comment #4:

Great test, thank you! I also tried it out. But I didn't do the bootstrapping because this script is only concerned with configure and make. Both were run after a 'make distclean'.

For me it was just slightly (<2 sec) faster! I then thought it might be because of the source file reading speed bottle-neck. Since the source wasn't in the RAM. So I unpacked a 'make dist' tarball in the /dev/shm and ran configure and make there. The result wasn't any different (within errors) than the './parallel-build-in-tmpfs' result above:

I then checked the memory usage before and after the processes involving /dev/shm (to make sure that tmpfs was actually in the RAM, not SWAP) and I saw that indeed the files are in the RAM. So I am also confused!

The main thing I was concerned about when coming up with this script was the burden of a large number of read/writes on the SSD drive. I hadn't actually objectively tested the running time like the test above. I guess this shows how important prejudice is ;-)!

I am curious if you are also using an SSD or HDD? Since your running time is so similar to mine, I guess you are also using an SSD, right? Probably on a HDD the difference might be more. I can't test on a HDD now.

At least I feel it is good that the writings of small files are done in the RAM not on the SSD. But I am still confused how the running time difference is so small.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Thu 28 Jul 2016 06:57:51 PM UTC, comment #3:

I just pushed to my gitlab repository a fix: https://gitlab.com/mose/gnuastro/commit/620aaeef1b19da6c89e9a3924c9d4349b12af4a0

Honestly, I didn't find a boost in compilation time (actually it was the contrary):

What am I doing wrong?

Mosè Giordano <giordano>
Project Member
Mon 25 Jul 2016 10:43:52 AM UTC, comment #2:

The script will now make a symbolic link called "build" in the top source directory for easy access to the built files:

https://gitlab.com/makhlaghi/gnuastro/commit/2de6a041ed550123c51eac8af5a47330687880b3

If you get the chance to try it out, any feedback would be very welcome ;-).

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Sat 23 Jul 2016 11:38:38 PM UTC, comment #1:

Sorry, I forgot to mention that the name of the branch is 'parallel-build-in-tmpfs'.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Sat 23 Jul 2016 11:35:20 PM UTC, original submission:

The text below is taken from the comments of the suggested script explaining the purpose of this suggestion. The script has been put in my fork of Gnuastro (link below) for ideas, criticisms, and suggestions prior to merging it with master.

https://gitlab.com/makhlaghi/gnuastro/branches

------

The configure and build process involves the creation, reading, and modification of a large number of files (input/output, or I/O). Therefore I/O issues can directly affect the work of developers who need to configure and build Gnuastro numerous times. Some such issues are listed below:

  • I/O can be slow in on-volatile (non-RAM) memory like traditional hard disks (HDDs) and even SSDs.
  • I/O will cause wear and tear on both the HDDs (mechanical failures) and SSDs (decreasing the lifetime).
  • Having the built files mixed with the source files can greatly affect backing up (synchronization) of source files (since it involves the management of a large number of small files that are changed a lot.

One solution to these problems is the tmpfs file system (see https://en.wikipedia.org/wiki/Tmpfs). Any file in tmpfs is actually stored in the RAM, not on HDDs or SSDs. Therefore I/O can be much faster and will also not harm the non-volatile memory devices. However, due to the volatile nature of RAM, files in the tmpfs filesystem will be permanently lost after a power-off.

The modern GNU C library (and thus the Linux kernel) define the '/dev/shm' directory for this purpose (POSIX shared memory). Therefore this script takes '/dev/shm' as the default here (the value to TMPDIR) this should be fine for all modern Linux based OSs. This script will make a 'gnuastro' directory in TMPDIR, change to that directory and configure and build gnuastro in there to have fast and non-wear configures and builds.

After this script is run, change directory to $TMPDIR/gnuastro and run other 'make' commands (for example 'make pdf', 'make check', or 'make install') from there. In Emacs, the command to be run with the 'M-x compile' command can be changed to 'cd /dev/shm/gnuastro; make -kj8' (to build on 8 threads) instead of the default 'make -k'. In this manner, the 'M-x recompile' commands to be run later will also build in the RAM while you modify the source in your non-volatile HDD or SSD.

To further speed up the build process, this script will build ('make') Gnuastro on multiple threads (value to the 'NUM_THREADS' variable bellow). This script can also be used in any other source file that uses the GNU build system.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by giordano (Posted a comment)
  • -unavailable- added by makhlaghi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 04 Aug 2016 11:28:27 PM UTCmakhlaghiStatusReady For Test=>Done
      Percent Complete30%=>100%
      Open/ClosedOpen=>Closed
    Mon 25 Jul 2016 10:43:52 AM UTCmakhlaghiPercent Complete20%=>30%

    Back to the top


    Powered by Savane 3.1-cleanup1