bugGNU gettext - Bugs: bug #49654, msgfmt and build determinism

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #49654: msgfmt and build determinism

Submitted by:  John Darrington <jmd>
Submitted on:  Sun Nov 20 06:54:44 2016  
 
Category: Programmer toolsSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Bruno Haible <haible>
Open/Closed: Closed

(Jump to the original submission Jump to the original submission)

Thu Dec 15 21:54:25 2016, comment #7:

Thanks a lot! I am adding your patch, with the necessary corrections (mostly comments, and message_list_delete_header_field could produce a non-NUL-terminated string in one case).

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu Dec 15 17:25:25 2016, comment #6:

Here is a patch which I think does what Bruno suggested.

(file #39242)

John Darrington <jmd>
Sun Nov 20 19:13:09 2016, comment #5:

Daiki,

> Bruno, are you willing to provide a patch for this?


No, I'd leave that to you. I'm trying to focus on libiconv and libsigsegv in the moment.

> I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in


There are several ways in which PO files can be integrated into a release, compiled, and then installed on a user's system. For example:
- The maintainer installs PO files manually, "make" merges them with the POT file, "make install" installs the .mo file. (This is what Makefile.in.in currently does.)
- "make" fetches the PO file from the Translation project and msgmerges it. "make install" installs the .mo file.
- You may omit msgmerge and thereby put old, unused messages into the .mo file.
- Some of the gettext 'hello' examples use a Makefile.am instead of Makefile.in.in.

In summary, there is considerable variation in the way the PO files are transmitted and msgmerged. But the common thing is that they use 'xgettext' (or some similar tool), 'msgmerge' (needed to filter out old, unused messages) and 'msgfmt'. I would therefore not try to fix a general problem with the tools by modifying Makefile.in.in.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun Nov 20 17:10:46 2016, comment #4:

I too fully agree with Bruno's suggestion. It sounds like a good idea to strip the timestamp when producing .mo files. Bruno, are you willing to provide a patch for this?

On the other hand, I think it could also be worked around by disabling the auto-msgmerge rule in po/Makefile.in.in, as I did in p11-kit:
https://github.com/p11-glue/p11-kit/commit/4965a8b2f150ea6c8dadd7dd22aab718f2814591

Daiki Ueno <ueno>
Project Administrator
Sun Nov 20 16:39:19 2016, comment #3:

I agree with Bruno's analysis.

John Darrington <jmd>
Sun Nov 20 12:56:47 2016, comment #2:

References:
https://tests.reproducible-builds.org/debian/issues/unstable/different_pot_creation_date_in_gettext_mo_files_issue.html
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792687

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun Nov 20 12:42:06 2016, comment #1:

The flow of this timestamp is as follows:
(1) xgettext puts it in, when it creates the .pot file.
(2) msgmerge preserves it, when it merges with a translator's .po file.
(3) msgfmt preserves it, when it compiles a .po file to a .mo file.

The POT-Creation-Date is important information for a translator, so that she knows whether to review the entire file or not.

Therefore IMO it's not in the steps (1) or (2) that we should change something, but in step (3): msgfmt should eliminate the POT-Creation-Date from the binary output that it produces (regardless whether it is .mo or other).

This would be work in
write-mo.c
write-java.c
write-csharp.c
write-resources.c
write-tcl.c
write-qt.c
write-desktop.c
write-xml.c

Use of msgunfmt on a given .mo file would then not be able to restore the lost POT-Creation-Date of course; the test suite of msgunfmt will have to be adjusted.

Note: A first step towards reproducible builds was already taken in version 0.19.8 (see NEWS):
- msgfmt now produces little-endian .mo files by default.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Sun Nov 20 06:54:44 2016, original submission:

Xgettext puts a line the generated .pot file of the form:
"POT-Creation-Date: 2016-11-20 07:47+0100\n"
which then gets propagated to all .po and .mo files.

This is problematic because it means that each time one builds a
project which uses xgettext one gets a different result, which raises concerns about verifibility and security.

Some currently affected projects are listed at https://tests.reproducible-builds.org/debian/issues/unstable/different_pot_creation_date_in_gettext_mo_files_issue.html"

Can I suggest that you either :

Thanks.

John Darrington <jmd>

 

Attached Files
file #39242:  msgfmt-remove-pot-creation-date.patch added by jmd (11kB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by ueno (Posted a comment)
  • -unavailable- added by haible (Posted a comment)
  • -unavailable- added by jmd (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu Dec 15 22:27:03 2016haibleStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Thu Dec 15 21:54:43 2016haibleCategoryBuild=>Programmer tools
    Thu Dec 15 21:54:25 2016haibleStatusConfirmed=>In Progress
      Assigned toNone=>haible
    Thu Dec 15 17:25:25 2016jmdAttached File-=>Added msgfmt-remove-pot-creation-date.patch, #39242
    Sun Nov 27 12:20:45 2016haibleCategoryNone=>Build
    Sun Nov 20 12:56:47 2016haibleStatusNone=>Confirmed
    Sun Nov 20 12:42:06 2016haibleSummaryxgettext and build determinism=>msgfmt and build determinism

    Back to the top


    Powered by Savane 3.1-cleanup1