bugGNU Astronomy Utilities - Bugs: bug #65578, Numeric user ID too large when...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #65578: Numeric user ID too large when generating the tarball

Submitter:  Raul Infante-Sainz <infantesainz>
Submitted:  Wed 10 Apr 2024 02:46:06 PM UTC
   
 
Category:  Development Severity:  3 - Normal
Item Group:  Output not reasonable Status:  In Progress
Privacy:  Public Assigned to:  glorenzetti
Open/Closed:  Open

Thu 11 Apr 2024 01:24:25 PM UTC, comment #5: 

Checked on Raul's computer, apparently the latest workaround doesn't work, and for two reasons:

  • TAR_OPTIONS are not recognized (neither on a setup with gnu bash, gnu make, gnu tar!)
  • setting --owner=root requires extra perms


That means that my attempt is not portable, and that even --owner=root may have portability issues.

Giacomo Lorenzetti <glorenzetti>
Group Member
Thu 11 Apr 2024 11:40:22 AM UTC, comment #4: 

This workaround works for me: https://gitlab.cefca.es/glorenzetti/gnuastro/-/commit/9f6aa5e454097d459634c0e5e9cc4d8f62c678f9

It is an old workaround, as pointed out here: https://mail.gnu.org/archive/html/bug-automake/2023-06/msg00002.html

And I don't know how portable it is.

Raul, would you mind to check if it works also on Mac?

Giacomo Lorenzetti <glorenzetti>
Group Member
Thu 11 Apr 2024 09:31:16 AM UTC, comment #3: 

Thanks Mohammad.

Unfortunately the "fastest solution" won't work either: on gnutar -o is also an alias for --old-archive, that is '--format=v7'. Incidentally, this also mean that we aren't using the same format, apparently.
Here is branch that forces the same format (ustar, as per convo in Maneage), regardless of the tar used: https://gitlab.cefca.es/glorenzetti/gnuastro/-/commit/b9f950d79fa1a64468b99f9b9be770461c5c702e
(Note: this doesn't fix Raul's issue, it's just for homogeneous reasons)

Speaking of the issue, let me stress what triggered it: the output of 'id -u' and 'id -g' on Raul's computer is O(10⁹), while on a typical Linux the result is O(10³) (usually both start from 1000, adding +1 for a new user, unless different settings). Testing on personal MacOSes, the result is ~500.
To have userIDs that big is strange, and even the comment in the autogenerated aclocal.m4 is along the same lines:

>     [# The POSIX 1988 'ustar' format is defined with fixed-size fields.
>      # There is notably a 21 bits limit for the UID and the GID.  In fact,
>      # the 'pax' utility can hang on bigger UID/GID (see automake bug #8343
>      # and bug #13588).
>      am_max_uid=2097151 # 2^21 - 1
>      am_max_gid=$am_max_uid
>      # The $UID and $GID variables are not portable, so we need to resort
>      # to the POSIX-mandated id(1) utility.  Errors in the 'id' calls
>      # below are definitely unexpected, so allow the users to see them
>      # (that is, avoid stderr redirection).

I'm investigating why those numbers are that big.

Rearding the usage of userIDs in the metadata, the idea is interesting and will probably solve also the issue (I guess that tar wouldn't crash if it avoids reading IDs...). The issue is then how to put those in automake, I'm looking into it.

Giacomo Lorenzetti <glorenzetti>
Group Member
Wed 10 Apr 2024 04:15:01 PM UTC, comment #2: 

Thanks Giacomo for looking into this bug so fast and the nice links to follow up on.

Since this problem is just about the user IDs, in the case of a distribution tarball, it is very bad to keep this metadata (which I assume can have security implications as well). Can add options to the 'tar' command in Automake? If so, we should use the '--owner=root --group=root' options in tar to just set the root's user ID on each file. On GNU/Linux, the root's user ID is 0. Raul, can you check what the root's user ID is on macOS?

In parallel (while we are investigating this), Raul's fastest solution to this problem is to use GNU Tar and not BSD Tar (which is available by default on macOS), right?

On a more fundamental point, we had some discussions on the best Tar command for archival purposes in Maneage, see this discussion, and in particular the points that Antonio Diaz Diaz (author of Lzip) raised here: https://savannah.nongnu.org/task/index.php?15699 .

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 10 Apr 2024 03:48:35 PM UTC, comment #1: 

Apparently bsdtar -o (and we use -chof...) is the same as
'--old-archive', i.e. '--format=ustar'. ustar is among those who
doesn't support ids grater than ~2M (see
https://www.gnu.org/software/tar/manual/html_chapter/Formats.html).
We can force tar to use something more recent, such as pax, as showed in this branch: https://gitlab.cefca.es/glorenzetti/gnuastro/-/commit/7bfcdd9f91b971fb6234f0a6fc06aef69811df17

Is this a desired change? Apparently not, GNU suggests to keep the old one, due to compatibilty issues: https://www.gnu.org/software/automake/manual/1.10/html_node/Options.html#index-Option_002c-tar_002dv7

On the other hand, the values of 'id -u' and 'id -g' on your computer appear too big, that I'm unable to replicate on a personal macOS (I obtain something around ~500, instead of ~1 billion).
Perhaps is worth investigating why those IDs have those values, if it's due to a initial configuration of the computer.

Giacomo Lorenzetti <glorenzetti>
Group Member
Wed 10 Apr 2024 02:46:06 PM UTC, original submission:  

I can not create the tarball of Gnuastro (as in .tar.gz) on my macOS (Monterey 12.3) laptop. I do the following steps:

cd /path/to/gnuastro-source
./developer-build
cd build
make dist

The above lines should generate the tarball of Gnuastro. But it is not generated. It runs and at some point it stops after showing several times 'Numeric user ID too large:'.

[...]
tardir=gnuastro-X.XX.XX-XXXXX && ${TAR-tar} chof - "$tardir" --format gnu | eval GZIP= gzip --best -c >gnuastro-X.XX.XX-XXXXX.tar.gz
: Numeric user ID too large: Numeric group ID too large:[...]
[...]


Raul Infante-Sainz <infantesainz>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by glorenzetti (Updated the item)
  • -email is unavailable- added by infantesainz (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-04-10 glorenzetti StatusNone In Progress
        Assigned toNone glorenzetti

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code