bugGNU gettext - Bugs: bug #49654, msgfmt and build determinism

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #49654: msgfmt and build determinism

Submitted by:  John Darrington <jmd>
Submitted on:  Sun 20 Nov 2016 06:54:44 AM UTC  
 
Category: Programmer toolsSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Bruno Haible <haible>
Open/Closed: Closed

(Jump to the original submission Jump to the original submission)

Tue 22 Aug 2017 11:00:12 AM UTC, comment #11:

> Implementing any of these options means the test will only work with the new gettext, it won't work with the old one anymore.


This is true for the po2test.awk suggestion. But the first of my 3 suggestions (use a modified de.po file, without POT-Creation-Date: header line) should work with both the old and new msgfmt, and does not require modifications of the C code.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Tue 22 Aug 2017 10:12:53 AM UTC, comment #10:

> The glibc/intl/tst-gettext test needs to be updated.
> - Either use as input instead of de.po a modified PO file, possibly &amp;quot;grep -v 'POT-Creation-Date:' de.po&amp;quot;
> - Or modify the po2test.awk script so that it eliminates the POT-Creation-Date: header line.


Implementing any of these options means the test will only work with the new gettext, it won't work with the old one anymore.

> - Or modify the positive_gettext_test / negative_gettext_test functions in tst-gettext.c to allow for the difference in POT-Creation-Date line.


That indeed will work, that said given it's C code, it's not trivial to just ignore this difference.

Aurelien Jarno <aurel32>
Mon 21 Aug 2017 10:48:03 PM UTC, comment #9:

> This change breaks the assumption that any msgstr entry from the po file is put unchanged into the mo file.


You are right, this assumption does not hold any more for the header entry.

> This breaks at least the glibc intl/tst-gettext test which checks that.


The glibc/intl/tst-gettext test needs to be updated.
- Either use as input instead of de.po a modified PO file, possibly "grep -v 'POT-Creation-Date:' de.po"
- Or modify the po2test.awk script so that it eliminates the POT-Creation-Date: header line.
- Or modify the positive_gettext_test / negative_gettext_test functions in tst-gettext.c to allow for the difference in POT-Creation-Date line.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Mon 21 Aug 2017 08:53:43 AM UTC, comment #8:

This change breaks the assumption that any msgstr entry from the po file is put unchanged into the mo file. This breaks at least the glibc intl/tst-gettext test which checks that.

Aurelien Jarno <aurel32>
Thu 15 Dec 2016 09:54:25 PM UTC, comment #7:

Thanks a lot! I am adding your patch, with the necessary corrections (mostly comments, and message_list_delete_header_field could produce a non-NUL-terminated string in one case).

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu 15 Dec 2016 05:25:25 PM UTC, comment #6:

Here is a patch which I think does what Bruno suggested.

(file #39242)

John Darrington <jmd>
Sun 20 Nov 2016 07:13:09 PM UTC, comment #5:

Daiki,

> Bruno, are you willing to provide a patch for this?


No, I'd leave that to you. I'm trying to focus on libiconv and libsigsegv in the moment.

> I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in


There are several ways in which PO files can be integrated into a release, compiled, and then installed on a user's system. For example:
- The maintainer installs PO files manually, "make" merges them with the POT file, "make install" installs the .mo file. (This is what Makefile.in.in currently does.)
- "make" fetches the PO file from the Translation project and msgmerges it. "make install" installs the .mo file.
- You may omit msgmerge and thereby put old, unused messages into the .mo file.
- Some of the gettext 'hello' examples use a Makefile.am instead of Makefile.in.in.

In summary, there is considerable variation in the way the PO files are transmitted and msgmerged. But the common thing is that they use 'xgettext' (or some similar tool), 'msgmerge' (needed to filter out old, unused messages) and 'msgfmt'. I would therefore not try to fix a general problem with the tools by modifying Makefile.in.in.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 20 Nov 2016 05:10:46 PM UTC, comment #4:

I too fully agree with Bruno's suggestion. It sounds like a good idea to strip the timestamp when producing .mo files. Bruno, are you willing to provide a patch for this?

On the other hand, I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in, as I did in p11-kit:
https://github.com/p11-glue/p11-kit/commit/4965a8b2f150ea6c8dadd7dd22aab718f2814591

Daiki Ueno <ueno>
Project Administrator
Sun 20 Nov 2016 04:39:19 PM UTC, comment #3:

I agree with Bruno's analysis.

John Darrington <jmd>
Sun 20 Nov 2016 12:56:47 PM UTC, comment #2:

References:
https://tests.reproducible-builds.org/debian/issues/unstable/different_pot_creation_date_in_gettext_mo_files_issue.html
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792687

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 20 Nov 2016 12:42:06 PM UTC, comment #1:

The flow of this timestamp is as follows:
(1) xgettext puts it in, when it creates the .pot file.
(2) msgmerge preserves it, when it merges with a translator's .po file.
(3) msgfmt preserves it, when it compiles a .po file to a .mo file.

The POT-Creation-Date is important information for a translator, so that she knows whether to review the entire file or not.

Therefore IMO it's not in the steps (1) or (2) that we should change something, but in step (3): msgfmt should eliminate the POT-Creation-Date from the binary output that it produces (regardless whether it is .mo or other).

This would be work in
write-mo.c
write-java.c
write-csharp.c
write-resources.c
write-tcl.c
write-qt.c
write-desktop.c
write-xml.c

Use of msgunfmt on a given .mo file would then not be able to restore the lost POT-Creation-Date of course; the test suite of msgunfmt will have to be adjusted.

Note: A first step towards reproducible builds was already taken in version 0.19.8 (see NEWS):
- msgfmt now produces little-endian .mo files by default.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun 20 Nov 2016 06:54:44 AM UTC, original submission:

Xgettext puts a line the generated .pot file of the form:
"POT-Creation-Date: 2016-11-20 07:47+0100\n"
which then gets propagated to all .po and .mo files.

This is problematic because it means that each time one builds a
project which uses xgettext one gets a different result, which raises concerns about verifibility and security.

Some currently affected projects are listed at https://tests.reproducible-builds.org/debian/issues/unstable/different_pot_creation_date_in_gettext_mo_files_issue.html"

Can I suggest that you either :

Thanks.

John Darrington <jmd>

 

Attached Files
file #39242:  msgfmt-remove-pot-creation-date.patch added by jmd (11KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by aurel32 (Posted a comment)
  • -unavailable- added by ueno (Posted a comment)
  • -unavailable- added by haible (Posted a comment)
  • -unavailable- added by jmd (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 15 Dec 2016 10:27:03 PM UTChaibleStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Thu 15 Dec 2016 09:54:43 PM UTChaibleCategoryBuild=>Programmer tools
    Thu 15 Dec 2016 09:54:25 PM UTChaibleStatusConfirmed=>In Progress
      Assigned toNone=>haible
    Thu 15 Dec 2016 05:25:25 PM UTCjmdAttached File-=>Added msgfmt-remove-pot-creation-date.patch, #39242
    Sun 27 Nov 2016 12:20:45 PM UTChaibleCategoryNone=>Build
    Sun 20 Nov 2016 12:56:47 PM UTChaibleStatusNone=>Confirmed
    Sun 20 Nov 2016 12:42:06 PM UTChaibleSummaryxgettext and build determinism=>msgfmt and build determinism

    Back to the top


    Powered by Savane 3.1-cleanup1