bugManeage - Bugs: bug #63054, Pyerfa crash, setting...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #63054: Pyerfa crash, setting setuptools_scm as a prerequisite of pyerfa

Submitter:  Raul Infante-Sainz <infantesainz>
Submitted:  Tue 13 Sep 2022 08:43:28 AM UTC
   
 
Category:  Software Severity:  3 - Normal
Item Group:  Crash Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Thu 20 Jul 2023 01:35:37 PM UTC, comment #11: 

Indeed, this was not a stable fix! When we tried a clean build from Commit 8161194d686 (one after Commit 644a236b7 of the previous comment). Astropy crashed, complaining about 'extension-helpers'!

We had to manually delete 'extension-helpers' as below and re-configure Maneage so it builds to the end successfully!


$ rm .local/version-info/python/extension-helper*
$ ./project configure -e


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 May 2023 09:58:19 AM UTC, comment #10: 

Until we find a more robust solution, I just merged the suggestion (to add 'setuptools_scm' as a dependency of 'pyerfa' and 'extension-helpers') to the main 'maneage' branch (as part of some other small fixes) in Commit 644a236b7.

From the discussions we had here, this is not a stable fix so I am not closing this bug. But since it decreases the number of occurrences, it was merged with that commit.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 03 Oct 2022 07:53:01 AM UTC, comment #9: 

Practical warning!: commit 394eee2 worked fine for me on each machine provided that I did a completely fresh build.

Trying to uninstall the python modules; or the python modules and python itself; either by rm's in .local/version-info/... alone, or by version-info plus .local/bin/... plus _.local/{lib,include}/... for the python-related files alone; failed in all cases for me - each time there was something old that caused a failure in the rebuild. So we probably could create  a developers' task for uninstalling all of python. For the moment, the only practical way I found to test was to do a completely fresh build.

Boud Roukema <boud>
Group Member
Mon 03 Oct 2022 07:35:13 AM UTC, comment #8: 

Although commit 2b02ded seems to work as a workaround (a very crude workaround), there's a more systematic solution based on #python community recommendations. This is based on 'python setup.py bdist_wheel'; I'll add comments about this in a few minutes at task #16268 [1]. So I recommend not to use 2b02ded (it's not elegant and no longer needed).

The (modular) bdist_wheel commit that I now recommend, and that worked for me on Debian 11.5 and CentOS 7, is d5775fa. The more practical commit for testing is 394eee2, which first includes cherry-picked 7286ad6 (cherry-picked ID dfa4758; for ncurses for python; bug #61271); 67daeee (cherry-picked ID f53729f; pyyaml update to 6.0 to solve bug #61811); and ab7303c (cherry-picked ID f161328; for extension-helpers and 'packaging'; also in bug #61811); then d577fa itself is the commit for bdist_wheel; and finally 'astropy' is enabled so that commit 394eee2 functions as a test. My earlier (horrible) pyerfa workaound is not included in 394eee2. See the branch 'bdist_wheel' [2] for all these commits, e.g. the modular commit d5775fa, or the testable combined commit 394eee2.

Testing on Darwin/xnu (and other systems) would be nice. :)

[1] https://savannah.nongnu.org/task/?16268

[2] https://codeberg.org/boud/maneage_dev/src/branch/bdist_wheel



Boud Roukema <boud>
Group Member
Tue 27 Sep 2022 07:59:36 AM UTC, comment #7: 

Commit e6c09c0ec [1] solves the pyerfa bug for me. This is effectively ab7303c [2] that fixes 'extension-helpers' and 'packaging' in bug #61811 [3], followed by 2b02ded that forcefully adds the 'packaging' .egg file to sys.path , and followed by enabling gnuastro installation in order to have a full test through to verify.mk checksums. [The 'pyyaml' bug (fixable with 67daeee) gives the expected error in the log file, without preventing successful configuring and making.]

In other words, 2b02ded does not solve the pyerfa bug for me unless it is merged with ab7303c; 2b02ded and ab7303c merged together do solve the pyerfa bug for me (provided that gnuastro is enabled in TARGETS.conf).


[1] https://codeberg.org/boud/maneage_dev/commit/e6c09c0ecf8f96106435353f3a37ad90ea0afa8f

[2] https://codeberg.org/boud/maneage_dev/commit/ab7303cd70e48830c22bd0f4ae7f70411afc3246

[3] https://savannah.nongnu.org/bugs/index.php?61811#comment8

[4] https://codeberg.org/boud/maneage_dev/commit/2b02dedd38d833463811a64400757bf9ba96b5b5

Boud Roukema <boud>
Group Member
Sun 25 Sep 2022 03:17:13 PM UTC, comment #6: 

I have now reproduced the bug: 'pyerfa' fails with an error that 'packaging' is missing, even though 'packaging' is a dependency of 'pyerfa' in 'python.mk'.

Commit 1333c1b1 proposed below by Raúl is not enough for my run: I got the above fatal error while including setuptools_scm as a prerequisite for pyerfa .

I'm currently trying a workaround, since 'packaging' was installed on this particular Maneage system about an hour before 'pyerfa' needed it, so this is probably something subtle related to pyerfa's 'setup.py' file and the various python 'os', 'sys' and 'setuptools' packages work...

Boud Roukema <boud>
Group Member
Sat 24 Sep 2022 08:09:51 PM UTC, comment #5: 

See my two merge requests at bug #61811 for overlapping discussion.

Rather than adding 'setuptools_scm' for all python packages, or rather than waiting until someone reports a problem for a package, a systematic way to benefit from the work of the Debian Astro group would be, for each of our python packages, to look at

https://tracker.debian.org/PACKAGENAME

after checking/searching for the matching debian PACKAGENAME; once the tracker web page is found, go to 'browse' (left-hand column), 'debian/' and find the file 'control', and see what prerequisites Debian Astro think are needed (a few packages are not on salsa.debian.org, but a 'browse' source button should be available somewhere on the tracker). 'Build-Depends:' are for building; 'Depends:' are for running.

Adding 'setuptools_scm' as a prerequisite for 'pyyaml' made no difference for me, and it's not listed in the Debian Astro control file [1].

With core maneage 4318670, I have not detected any errors compiling pyerfa.

[1] https://salsa.debian.org/python-team/packages/pyyaml/-/blob/debian/master/debian/control

Boud Roukema <boud>
Group Member
Fri 16 Sep 2022 06:43:46 AM UTC, comment #4: 

Thanks Mohammad. I have pushed a commit so you can easily try it. Just don't forget to add 'astropy' as a final Python target to be built.

https://gitlab.com/maneage/project-dev/-/commit/a9eac928

In my case, none of the problems described in bug #61811 happened. However, a new one appeared: 'numpy' complaining about not finding 'cython'! So, it would be a new case for bug #61811. Let me know how it goes in your case.

I will try to check more things in the mean time. Maybe adding 'setuptools_scm' as a prerequisite of all problematic packages? Let's see.

Raul Infante-Sainz <infantesainz>
Group Member
Thu 15 Sep 2022 06:37:59 PM UTC, comment #3: 

That sounds like a good idea.

Maybe after adding this, you can try a clean build in a different build directory on one of your systems that had that problem before. I remember almost always having the problem on my system, so after checking and committing, just let me know and I'll also do a clean build here to confirm and hopefully merge it in.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 15 Sep 2022 05:52:12 PM UTC, comment #2: 

maybe you are right! I have been looking at that. The only thing I have found is this line in the 'setup.py' file of 'extension-helpers':


setup(use_scm_version={'write_to': os.path.join('extension_helpers', 'version.py')})


I tried to reproduce that problem but since it is not a reproducible bug I couldn't. In any case I can also add 'setuptools_scm' as a prerequisite of 'extension-helpers' and let the users try if they get the same problem. What do you think?

Raul Infante-Sainz <infantesainz>
Group Member
Thu 15 Sep 2022 04:24:49 PM UTC, comment #1: 

Thanks a lot for sharing this discovery!

Do you think this is related to bug #61811 (jinja2, extension-helpers, pybind11 not found, even though installed)? Because pyerfa was also mentioned there. If those software have a similar problem, we may be able to fix them in a similar way ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 13 Sep 2022 08:43:28 AM UTC, original submission:  

Until now, 'setuptools_scm' was not a prerequisite of 'pyerfa'. It is strange that we didn't notice this until now because 'pyerfa' is also a requisite of 'astropy'. In any case, I faced an installation crash after updating the Maneage branch of one project and I found this issue.

With the commit below, 'setuptools_scm' has been set as a prerequisite of 'pyerfa'. Now the installation of 'pyerfa' is done without any problem. Check it and merge if you consider that everything is good.

https://gitlab.com/maneage/project-dev/-/commit/1333c1b1

Raul Infante-Sainz <infantesainz>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by infantesainz (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code