bugGNU Octave - Bugs: bug #49126, build-env-features.cc missing all...

 
 

bug #49126: build-env-features.cc missing all defines (e.g. HAVE_CHOLMOD) due to dependency on gawk features

Submitter:  Carlo de Falco <cdf>
Submitted:  Mon 19 Sep 2016 10:39:56 AM UTC
   
 
Category:  Configuration and Build System Severity:  5 - Blocker
Priority:  5 - Normal Item Group:  Installation Failure
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.2.0-rc1 Operating System:  * Mac OS
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 20 Sep 2016 09:08:50 AM UTC, comment #16: 

Re: comment #14

jwe, I have installed GNU awk into /usr/local/bin
after discovering this issue that's why you were
not able to reproduce it, previously there was only the
one in /usr/bin

c.

Carlo de Falco <cdf>
Group Member
Mon 19 Sep 2016 11:45:21 PM UTC, comment #15: 

Done

http://hg.savannah.gnu.org/hgweb/octave/rev/b3268dbf9781

So the build should no longer need gawk (but will use it over other awks if it is found in PATH).

The sed vs gsed issue I mentioned on the mailing list remains.

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 11:40:22 PM UTC, comment #14: 

It seems to fix the problem for me on the mac mini system that we are using for buildbot.  I was initially confused because I wasn't seeing a problem even before applying the patch, but that was apparently because my PATH had /usr/local/bin first so I was getting awk == gawk.  I was able to duplicate the problem by putting /usr/bin first, and then verify that the patch fixes the problem.

So, yes, please push this change.

Thanks.

John W. Eaton <jwe>
Group administrator
Mon 19 Sep 2016 11:11:24 PM UTC, comment #13: 

I think it's a safe change, but I would like some confirmation from Carlo or another macOS user first that this change actually fixes the problem, since I don't have the exact same awk on hand. There are so many possible variations that it could be this change or there could be something else going on.

I think I remember right that Debian used to carry a "nawk" program, which I think was the same as the BSD awk that is still used on all *BSD and macOS systems, but I can't find it in the Debian repositories anymore, so I'd like independent confirmation on macOS.

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 11:05:00 PM UTC, comment #12: 

I already wrote to the maintainers-list my preference for sed/awk/perl etc.

But I certainly think you can safely make the change proposed in comment #11.  As written, the regular expression is hard to understand by humans, let alone awk.  Your patch makes it clearer what is supposed to be happening.

Rik <rik5>
Group administrator
Mon 19 Sep 2016 08:39:12 PM UTC, comment #11: 

As far as I can tell the only GNU-specific feature in this script is the regular expression syntax


/(OCTAVE_|)HAVE_/


This is not an error with gawk --posix, but is with mawk on my system.

Can you test whether the attached patch fixes this when building without gawk installed but with gsed installed (the script uses both)?

This is a pretty trivial change, if that's all there is to it (of course there are sed issues discussed elsewhere). The biggest problem is that it's not easy to catch these kinds of things, and they tend to creep in again because most of us just use the GNU tools and have no easy way to test with all possible variants.


(file #38556)

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:44:27 PM UTC, comment #10: 

Yes, I definitely get build failures with the --posix options on both, so this is reproducible on any system. I think that it should be possible to make the scripts portable, but I'm not sure about pattern alternation (a\|b) as I mentioned on the mailing list. I can certainly help work on this to some degree.

As far as the scripts that use sed instead of $SED, I think the intent is that they are portable so it doesn't matter what $SED is. I think that's the approach that both gnulib and automake try to take, to only write sed commands that will work with any sed so it doesn't matter as long as some sed can be called.

As I said I lean towards portability, but other of our co-maintainers may say it's not worth the effort, what's the harm in requiring GNU awk and sed?

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:31:06 PM UTC, comment #9: 

Mike,

That's a good point.

We should also consider that many third party scripts
which are part of the build system (e.g. from gnulib and automake) use sed, so we acnnot give up sed alltogether ...

Also note that not all those third party scripts allow
choosing gsed instead of sed, for example

  build-aux/ylwrap

uses just 'sed' instead of $SED ...

c.


Carlo de Falco <cdf>
Group Member
Mon 19 Sep 2016 03:27:51 PM UTC, comment #8: 

Thank you (smh), now testing with AWK="gawk --posix" SED="gsed --posix".

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:23:37 PM UTC, comment #7: 

Re: needing both sed and awk, I haven't look too closely at it, but they definitely do have different features. I think simple text replacement is much more easily expressed with sed, and awk is much better at text processing that requires tracking state or printing large blocks of text in addition to text substitution or generating new text.

In short, I think everything could be done with awk, but it may make the things that are currently done with sed a little more awkward. Just my impression from years of using both tools and knowing their strengths and weaknesses, I would use each in different contexts.

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:22:29 PM UTC, comment #6: 

if you want to test portability of SED scripts
you could invoke gsed with the '--posix' option


Carlo de Falco <cdf>
Group Member
Mon 19 Sep 2016 03:18:03 PM UTC, comment #5: 

I get the same file if I configure with AWK=mawk on Debian.

I think I lean towards portability, jwe and others can weight in on what they think. Now that I have an awk that I can test with that makes things easier. I don't have a sed program that I can test with to make sure our usage of sed is portable.

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:10:31 PM UTC, comment #4: 

Indeed it seems to be an issue related
to awk/gawk the attached libinterp/build-env-features.cc
produced after setting AWK=gawk looks much better.

So again, do we really need both tools and do we really
need GNU version of both tools?

Thanks,
c.




(file #38553)

Carlo de Falco <cdf>
Group Member
Mon 19 Sep 2016 03:08:13 PM UTC, comment #3: 

Thanks, that confirms that macOS awk cannot produce the right output file, the way the script is written now (see libinterp/build-env-features.sh).

See also my post to the list about sed vs gsed (http://lists.gnu.org/archive/html/octave-maintainers/2016-09/msg00193.html), same applies for awk vs gawk, we need to decide whether we value portability of these scripts to any version of sed/awk, or features for easier maintenance and testing. I can't say, I don't write most of these scripts.

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 03:01:07 PM UTC, comment #2: 

Hi Mike,

Here you are, I'll try to rebuild using

  AWK=gawk

and report back.
But do we really need to use both awk AND sed?

c.




$ cat libinterp/build-env-features.cc
// DO NOT EDIT!  Generated automatically from config.h by Make."

#if defined (HAVE_CONFIG_H)
#  include "config.h"
#endif

#include "build-env.h"
#include "ov.h"

namespace octave
{
  namespace build_env
  {
    octave_scalar_map
    features (void)
    {
      static bool initialized = false;

      static octave_scalar_map m;

      if (! initialized)
        {
          static octave_value ov_true = (true);
          static octave_value ov_false = (false);


          initialized = true;
        }

      return m;
    }
  };
};
mathicsepc32:octave_default_build carlo$



Carlo de Falco <cdf>
Group Member
Mon 19 Sep 2016 02:50:27 PM UTC, comment #1: 

Good testing, thank you. I see that you passed SED=gsed, but not AWK=gawk. I also see that configure tests for gawk before awk, and didn't find it.

The script that puts the "features" into the Octave executable is a awk/gawk script, and maybe the awk features that we are using are not portable. Can you also attach or provide a link to the file libinterp/build-env-features.cc?

Mike Miller <mtmiller>
Group Member
Mon 19 Sep 2016 10:39:56 AM UTC, original submission:  

Octave 4.2 built on OSX El Capitan with the following options:

brew_prefix=/usr/local
../octave/configure \
    SED=gsed \
    CC=clang CXX=clang++ \
    F77=gfortran \
    LDFLAGS="-L${brew_prefix}/lib" \
    --prefix=/opt/octave/4.2.0 \
    --enable-dependency-tracking \
    --enable-link-all-dependencies \
    --with-x=no \
    --without-OSMesa \
    --disable-java \
    --with-blas="openblas"


Here [1] and [2] you can find the files config.log and fntests.log.

configure does not complain about
suitesparse components (e.g. cholmod or amd) not being found, and even
reports the flags used to link to them, yet the tests for Octave functions
depending on those libraries are skipped by 'make check'.

[1] http://www1.mate.polimi.it/~carlo/config.log
[2] http://www1.mate.polimi.it/~carlo/fntests.log

Carlo de Falco <cdf>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38556:  awk.diff added by mtmiller (678B - text/x-diff)
file #38553:  build-env-features.cc added by cdf (37KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by cdf (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-09-19 mtmiller StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2016-09-19 mtmiller Attached File- Added awk.diff, #38556
    2016-09-19 mtmiller Summarybuild-env-featuers.cc missing all defines (e.g. HAVE_CHOLMOD) due to dependency on gawk features build-env-features.cc missing all defines (e.g. HAVE_CHOLMOD) due to dependency on gawk features
    2016-09-19 mtmiller StatusNone Confirmed
        Summaryconfigure finds suitesparse components but flags HAVE_CHOLMOD, HAVE_AMD, etc. are build-env-featuers.cc missing all defines (e.g. HAVE_CHOLMOD) due to dependency on gawk features
    2016-09-19 cdf Attached File- Added build-env-features.cc, #38553
        StatusConfirmed None
        Summaryconfigure finds suitesparse components but build-env-features HAVE_CHOLMOD, HAVE_AMD, etc. wrong due to awk vs gawk configure finds suitesparse components but flags HAVE_CHOLMOD, HAVE_AMD, etc. are
    2016-09-19 mtmiller StatusNone Confirmed
        Summaryconfigure finds suitesparse components but flags HAVE_CHOLMOD, HAVE_AMD, etc. are configure finds suitesparse components but build-env-features HAVE_CHOLMOD, HAVE_AMD, etc. wrong due to awk vs gawk

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code