bugGNU gettext - Bugs: bug #49654, msgfmt and build determinism

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #49654: msgfmt and build determinism

Submitter:  John Darrington <jmd>
Submitted:  Sun 20 Nov 2016 06:54:44 AM UTC
   
 
Category:  Programmer tools Severity:  3 - Normal
Item Group:  None Status:  Fix Released
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Jump to the original submission

Thu 09 May 2019 07:25:30 PM UTC, comment #12: 

The feature is contained in the gettext-0.20 release.

Bruno Haible <haible>
Group administrator
Tue 22 Aug 2017 11:00:12 AM UTC, comment #11: 


> Implementing any of these options means the test will only work with the new gettext, it won't work with the old one anymore.


This is true for the po2test.awk suggestion. But the first of my 3 suggestions (use a modified de.po file, without POT-Creation-Date: header line) should work with both the old and new msgfmt, and does not require modifications of the C code.

Bruno Haible <haible>
Group administrator
Tue 22 Aug 2017 10:12:53 AM UTC, comment #10: 


> The glibc/intl/tst-gettext test needs to be updated.
> - Either use as input instead of de.po a modified PO file, possibly &amp;quot;grep -v 'POT-Creation-Date:' de.po&amp;quot;
> - Or modify the po2test.awk script so that it eliminates the POT-Creation-Date: header line.


Implementing any of these options means the test will only work with the new gettext, it won't work with the old one anymore.

> - Or modify the positive_gettext_test / negative_gettext_test functions in tst-gettext.c to allow for the difference in POT-Creation-Date line.


That indeed will work, that said given it's C code, it's not trivial to just ignore this difference.

Aurelien Jarno <aurel32>
Mon 21 Aug 2017 10:48:03 PM UTC, comment #9: 


> This change breaks the assumption that any msgstr entry from the po file is put unchanged into the mo file.


You are right, this assumption does not hold any more for the header entry.

> This breaks at least the glibc intl/tst-gettext test which checks that.


The glibc/intl/tst-gettext test needs to be updated.
- Either use as input instead of de.po a modified PO file, possibly "grep -v 'POT-Creation-Date:' de.po"
- Or modify the po2test.awk script so that it eliminates the POT-Creation-Date: header line.
- Or modify the positive_gettext_test / negative_gettext_test functions in tst-gettext.c to allow for the difference in POT-Creation-Date line.

Bruno Haible <haible>
Group administrator
Mon 21 Aug 2017 08:53:43 AM UTC, comment #8: 

This change breaks the assumption that any msgstr entry from the po file is put unchanged into the mo file. This breaks at least the glibc intl/tst-gettext test which checks that.

Aurelien Jarno <aurel32>
Thu 15 Dec 2016 09:54:25 PM UTC, comment #7: 

Thanks a lot! I am adding your patch, with the necessary corrections (mostly comments, and message_list_delete_header_field could produce a non-NUL-terminated string in one case).

Bruno Haible <haible>
Group administrator
Thu 15 Dec 2016 05:25:25 PM UTC, comment #6: 

Here is a patch which I think does what Bruno suggested.



(file #39242)

John Darrington <jmd>
Sun 20 Nov 2016 07:13:09 PM UTC, comment #5: 

Daiki,

> Bruno, are you willing to provide a patch for this?


No, I'd leave that to you. I'm trying to focus on libiconv and libsigsegv in the moment.

> I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in


There are several ways in which PO files can be integrated into a release, compiled, and then installed on a user's system. For example:
- The maintainer installs PO files manually, "make" merges them with the POT file, "make install" installs the .mo file. (This is what Makefile.in.in currently does.)
- "make" fetches the PO file from the Translation project and msgmerges it. "make install" installs the .mo file.
- You may omit msgmerge and thereby put old, unused messages into the .mo file.
- Some of the gettext 'hello' examples use a Makefile.am instead of Makefile.in.in.

In summary, there is considerable variation in the way the PO files are transmitted and msgmerged. But the common thing is that they use 'xgettext' (or some similar tool), 'msgmerge' (needed to filter out old, unused messages) and 'msgfmt'. I would therefore not try to fix a general problem with the tools by modifying Makefile.in.in.

Bruno Haible <haible>
Group administrator
Sun 20 Nov 2016 05:10:46 PM UTC, comment #4: 

I too fully agree with Bruno's suggestion.  It sounds like a good idea to strip the timestamp when producing .mo files.  Bruno, are you willing to provide a patch for this?

On the other hand, I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in, as I did in p11-kit:
https://github.com/p11-glue/p11-kit/commit/4965a8b2f150ea6c8dadd7dd22aab718f2814591

Daiki Ueno <ueno>
Group administrator
Sun 20 Nov 2016 04:39:19 PM UTC, comment #3: 

I agree with Bruno's analysis.

John Darrington <jmd>
Sun 20 Nov 2016 12:56:47 PM UTC, comment #2: 
Bruno Haible <haible>
Group administrator
Sun 20 Nov 2016 12:42:06 PM UTC, comment #1: 

The flow of this timestamp is as follows:
(1) xgettext puts it in, when it creates the .pot file.
(2) msgmerge preserves it, when it merges with a translator's .po file.
(3) msgfmt preserves it, when it compiles a .po file to a .mo file.

The POT-Creation-Date is important information for a translator, so that she knows whether to review the entire file or not.

Therefore IMO it's not in the steps (1) or (2) that we should change something, but in step (3): msgfmt should eliminate the POT-Creation-Date from the binary output that it produces (regardless whether it is .mo or other).

This would be work in
  write-mo.c
  write-java.c
  write-csharp.c
  write-resources.c
  write-tcl.c
  write-qt.c
  write-desktop.c
  write-xml.c

Use of msgunfmt on a given .mo file would then not be able to restore the lost POT-Creation-Date of course; the test suite of msgunfmt will have to be adjusted.

Note: A first step towards reproducible builds was already taken in version 0.19.8 (see NEWS):
  - msgfmt now produces little-endian .mo files by default.

Bruno Haible <haible>
Group administrator
Sun 20 Nov 2016 06:54:44 AM UTC, original submission:  


Xgettext puts a line the generated .pot file of the form:
"POT-Creation-Date: 2016-11-20 07:47+0100\n"
which then gets propagated to all .po and .mo files.

This is problematic because it means that each time one builds a
project which uses xgettext one gets a different result, which raises concerns about verifibility and security.

Some currently affected projects are listed at https://tests.reproducible-builds.org/debian/issues/unstable/different_pot_creation_date_in_gettext_mo_files_issue.html"

Can I suggest that you either :


Thanks.

John Darrington <jmd>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #39242:  msgfmt-remove-pot-creation-date.patch added by jmd (11KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by aurel32 (Posted a comment)
  • -email is unavailable- added by ueno (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by jmd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-04 haible StatusFixed Fix Released
    2016-12-15 haible StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2016-12-15 haible CategoryBuild Programmer tools
    2016-12-15 haible StatusConfirmed In Progress
        Assigned toNone haible
    2016-12-15 jmd Attached File- Added msgfmt-remove-pot-creation-date.patch, #39242
    2016-11-27 haible CategoryNone Build
    2016-11-20 haible StatusNone Confirmed
    2016-11-20 haible Summaryxgettext and build determinism msgfmt and build determinism

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code