bugGNU Octave - Bugs: bug #45646, files in source distribution...

 
 

bug #45646: files in source distribution should not be deleted on "clean" or "distclean"

Submitter:  Mike Miller <mtmiller>
Submitted:  Wed 29 Jul 2015 12:41:16 AM UTC
   
 
Category:  Configuration and Build System Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Other
Status:  Fixed Assigned to:  mtmiller
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 06 Jul 2016 12:04:25 AM UTC, comment #18: 

That's good enough for me.  Closing this report.

Rik <rik5>
Group administrator
Tue 05 Jul 2016 11:54:30 PM UTC, comment #17: 

I pushed a change

http://hg.savannah.gnu.org/hgweb/octave/rev/21c12b837089

to only delete built documentation on 'make maintainer-clean'.

I can now build in-tree from a source distribution with multiple sequences of './configure && make' and 'make distclean' and get repeatable results and identical content to the original source tarball, with the only anomaly being octave.jar.

The octave.jar file is built from some .class files compiled from .java sources. The .java sources are distributed, as is the octave.jar file, but not the .class files. This is analogous to distributing an executable and a bunch of sources, but not the *.o files that were linked into the executable. Make has to regenerate these intermediate files because they are missing, which causes the end product to be overwritten.

Mike Miller <mtmiller>
Group Member
Sat 02 Jul 2016 05:38:17 PM UTC, comment #16: 

I agree that it is not ideal to use an internal name like "clean-aminfo".  On the other hand, Automake and the whole Autotools suite are ancient and change very slowly.  It might be 5 years before that name gets changed, and in the meantime we would have everything working correctly.

I vote to make the change.

Rik <rik5>
Group administrator
Sat 02 Jul 2016 04:43:44 AM UTC, comment #15: 

I can prevent the built-in automake rule that deletes the doc files by defining a target "clean-aminfo" that overrides the automake one. But this is an internal name, not mentioned in the user manual, doesn't seem like a good idea to depend on that name/behavior. Other than that there is no way to prevent 'make clean' from deleting the dvi/pdf/ps/html files.

Mike Miller <mtmiller>
Group Member
Sat 02 Jul 2016 04:24:28 AM UTC, comment #14: 

I pushed a change that fixes which files get deleted and preserved for the clean and distclean targets:

http://hg.savannah.gnu.org/hgweb/octave/rev/659731a4563f

The general principle I applied is for 'make clean' to delete files that are created by 'make all', and 'make distclean' to delete files created by './configure'.

The exceptions are the source files that we build during 'make all', but that we want to be part of the source distribution. So now all of the libinterp and liboctave files that were mentioned here before are not deleted on clean or distclean.

There is still a small set of files that are distributed but are deleted on 'make clean' (if the user does an in-source-tree build) that I don't have a good way to fix at the moment:


doc/interpreter/octave.dvi
doc/interpreter/octave.html
doc/interpreter/octave.pdf
doc/interpreter/octave.ps
doc/liboctave/liboctave.dvi
doc/liboctave/liboctave.html
doc/liboctave/liboctave.pdf
doc/liboctave/liboctave.ps
scripts/java/octave.jar


The documentation files are deleted because of a built-in automake rule. The octave.jar file needs to be deleted because it seems to always get rebuilt even if it exists in $(srcdir).

Mike Miller <mtmiller>
Group Member
Mon 10 Aug 2015 04:27:09 AM UTC, comment #13: 

The problem seems to be the line in config.h that is


#include "oct-conf-post.h"


config.h is in the build directory and it assumes that oct-conf-post.h is also in the build directory.  The line that generates this in config.h is in configure.ac:


AH_BOTTOM([#include "oct-conf-post.h"])


If there is a way to substitute a variable, such as $srcdir, in the configure.ac code then this can be easily solved.  I tried, but wasn't successful.  We could also have a rule to copy over oct-conf-post.h from the source to the build directory but first checking to see if they were the same.  Something has to be done, but I don't have good ideas right now.

Rik <rik5>
Group administrator
Sun 09 Aug 2015 02:47:18 PM UTC, comment #12: 

After Rik's change (comment #6) I get this (cloned repo/fresh checkout) on Mageia-4 Linux (just did the usual
./bootstrap
in the src dir (= my <development_subdir>/dev) followed by
../dev/configure ; make -j7
in the build dir:

:
:
  CC       cloexec.lo
In file included from ../../dev/libgnu/base64.c:42:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
In file included from ../../dev/libgnu/areadlink-with-size.c:21:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
Makefile:2145: recipe for target 'areadlink-with-size.lo' failed
make[4]: *** [areadlink-with-size.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
Makefile:2145: recipe for target 'base64.lo' failed
make[4]: *** [base64.lo] Error 1
In file included from ../../dev/libgnu/md5.c:22:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
In file included from ../../dev/libgnu/c-ctype.c:18:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
Makefile:2145: recipe for target 'md5.lo' failed
compilation terminated.
make[4]: *** [md5.lo] Error 1
Makefile:2145: recipe for target 'c-ctype.lo' failed
make[4]: *** [c-ctype.lo] Error 1
In file included from ../../dev/libgnu/canonicalize.c:17:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
Makefile:2145: recipe for target 'canonicalize.lo' failed
make[4]: *** [canonicalize.lo] Error 1
In file included from ../../dev/libgnu/cloexec.c:20:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
In file included from ../../dev/libgnu/bitrotate.c:1:0:
../config.h:3479:27: fatal error: oct-conf-post.h: No such file or directory
 #include "oct-conf-post.h"
                           ^
compilation terminated.
Makefile:2145: recipe for target 'cloexec.lo' failed
make[4]: *** [cloexec.lo] Error 1


Copying / symlinking oct-conf-post.h to libgnu/ (OK a dirty hack) fixes the build for Linux, but cross-building in mxe-octave doesn't work as "make all dist" doesn't copy it (of course) to libgnu/ in the dist archive.
Manually adding it to the dist archive before building in mxe works, of course.

I wonder why a mere rename of an include file + its references break the build process.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 06 Aug 2015 06:22:23 PM UTC, comment #11: 

@jwe: Yes, you've got it right.  'make clean' by itself removes files that were present in the original tarball.  However it only has this issue for in-tree builds.  VPATH builds with a separate build directory appear to be fine from my testing.

This bug has been present for a long time but I don't think it's been a big issue because the normal usage pattern is


tar xf octave.tar.gz
cd octave
./configure
make
make install
cd ..
rm -rf octave



Rik <rik5>
Group administrator
Thu 06 Aug 2015 06:13:54 PM UTC, comment #10: 

Maybe I misunderstand Rik's comment #6 then, because it seems to say that make clean by itself (not make maintainer-clean) will remove some files that are included in the tarball distribution.

John W. Eaton <jwe>
Group administrator
Thu 06 Aug 2015 05:08:56 PM UTC, comment #9: 

@jwe: Yes, your "rules" in comment #7 match the guidance frome GNU that I published in comment #1.


Rik <rik5>
Group administrator
Thu 06 Aug 2015 04:38:25 PM UTC, comment #8: 

Yes, that is my understanding of the clean targets.

This bug was originally about files, e.g. libinterp/parse-tree/oct-parse.cc, that are deleted on make clean (because they are in BUILT_SOURCES, and all of BUILT_SOURCES is added to CLEANFILES). This goes back to previous versions too, in 3.8 if you unpack the offical tar release, build it, clean it, you can't build again without installing bison.

One problem I've further noticed after reporting this is that we have some files that we like to distribute (so they should not be deleted on distclean), but if you do an out-of-tree build they are copied into the build tree, so they should be deleted.

For example, DOCSTRINGS are copied (with timestamp preserved) into the build directory. Thanks to Rik's change, they are not deleted on distclean (because they are distributed) but that leaves leftover files for out-of-tree builds now.

So we may need more special handling for those types of files.

Mike Miller <mtmiller>
Group Member
Thu 06 Aug 2015 03:50:48 PM UTC, comment #7: 

I've always thought that the clean rules could be written something like this:


clean:
        rm -f some files that are built by make
        rm -f perhaps leaving some that are not
        rm -f included in a tarball distribution

distclean: clean
        rm -f some other files so that
        rm -f we are back to the set of files
        rm -f from a tarball distribution

maintainer-clean: distclean
        rm -f also remove files that need special
        rm -f tools to rebuild that only maintainers
        rm -f are expected to have


Is that not correct?

John W. Eaton <jwe>
Group administrator
Thu 06 Aug 2015 03:42:36 PM UTC, comment #6: 

I checked in a changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/9c7dd58b57e4) that handles oct-conf-post.h.  At this point 'make distclean' preserves all of the original files.  The list of files that are removed from the original tarball with 'make clean' is now


./libinterp/parse-tree/oct-parse.cc
./libinterp/parse-tree/lex.cc
./libinterp/parse-tree/oct-gperf.h
./libinterp/parse-tree/oct-parse.h
./libinterp/corefcn/oct-tex-lexer.cc
./libinterp/corefcn/oct-tex-parser.cc
./libinterp/corefcn/oct-tex-symbols.cc
./libinterp/corefcn/graphics-props.cc
./libinterp/builtin-defun-decls.h
./scripts/java/octave.jar
./doc/liboctave/liboctave.html/*
./doc/liboctave/liboctave.pdf
./doc/liboctave/liboctave.dvi
./doc/liboctave/liboctave.ps
./doc/interpreter/octave.html/*
./doc/interpreter/octave.ps
./doc/interpreter/octave.dvi
./doc/interpreter/octave.pdf



Rik <rik5>
Group administrator
Wed 29 Jul 2015 09:40:41 PM UTC, comment #5: 

@Mike: oct-conf.post.h is generated from oct-conf.post.in.h by configure


AC_CONFIG_FILES([
  oct-conf-post.h:oct-conf-post.in.h


However, I don't see that there are any substitutions between in.h and .h.  Maybe there was at one point, but processing this through configure no longer seems necessary.  If that is the case we could remove it from configure.ac and use EXTRA_DIST to distribute it.  There is no need to clean it, even at the maintainer-clean level because it is not a generated file.

Rik <rik5>
Group administrator
Wed 29 Jul 2015 09:31:50 PM UTC, comment #4: 

I fixed ChangeLog by moving it to MAINTAINERCLEANFILES here (http://hg.savannah.gnu.org/hgweb/octave/rev/517f5a006279).

Rik <rik5>
Group administrator
Wed 29 Jul 2015 09:08:57 PM UTC, comment #3: 

@Mike: I think I answered my own question.  I unpacked a tarball and if I do an in-tree build then I see what you are saying.  I alse notice that the HTML documentation and the manual itself (octave.pdf, octave.ps, octave.dvi) are also removed with a simple 'make clean'

Rik <rik5>
Group administrator
Wed 29 Jul 2015 08:58:04 PM UTC, comment #2: 

@Mike: I'm unable to reproduce the list of files being deleted in the source tree for 'make clean'.  I am using a VPATH build.  Were you doing an in-tree build?

Rik <rik5>
Group administrator
Wed 29 Jul 2015 08:51:13 PM UTC, comment #1: 

Good catch, although we need to think carefully about what files belong in which cleanup batch.

Here is the guidance from GNU Autotools


As the GNU Standards aren’t always explicit as to which files should be removed by which rule, we’ve adopted a heuristic that we believe was first formulated by François Pinard:

    If make built it, and it is commonly something that one would want to rebuild (for instance, a .o file), then mostlyclean should delete it.
    Otherwise, if make built it, then clean should delete it.
    If configure built it, then distclean should delete it.
    If the maintainer built it (for instance, a .info file), then maintainer-clean should delete it. However maintainer-clean should not delete anything that needs to exist in order to run ‘./configure && make’.


I fixed the removal of the documentation files (.texi and DOCSTRINGS) with distclean target in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/4300fe896004).

Rik <rik5>
Group administrator
Wed 29 Jul 2015 12:41:16 AM UTC, original submission:  

The following files are part of the source distribution tarball, but are deleted when the "clean" target is executed. This leaves the user with fewer files than were there when the source was unpacked:

  * libinterp/builtin-defun-decls.h
  * libinterp/corefcn/graphics-props.cc
  * libinterp/corefcn/oct-tex-lexer.cc
  * libinterp/corefcn/oct-tex-parser.cc
  * libinterp/corefcn/oct-tex-symbols.cc
  * libinterp/parse-tree/lex.cc
  * libinterp/parse-tree/oct-gperf.h
  * libinterp/parse-tree/oct-parse.cc
  * libinterp/parse-tree/oct-parse.h
  * scripts/java/octave.jar

And the following set of files are similarly deleted when the "distclean" target is executed, again leaving the user with fewer files than we gave them in the source tarball:

  * ChangeLog
  doc/interpreter/.texi
  * libinterp/DOCSTRINGS
  * oct-conf-post.h
  * scripts/DOCSTRINGS

Ideally, the "distclean" target should return the user to the same set of files that were there when the tarball was unpacked. I plan to look at this, just documenting what I've found so far.

Mike Miller <mtmiller>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-07-06 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2015-07-29 rik5 StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code