bugGNU gettext - Bugs: bug #59658, xgettext and msginit output isn't...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #59658: xgettext and msginit output isn't reproducible

Submitter:  Miguel Ángel Arruga Vivas <m1gu3l>
Submitted:  Fri 11 Dec 2020 01:19:33 PM UTC
Votes:  1  
 
Category:  Programmer tools Severity:  3 - Normal
Item Group:  None Status:  In Progress
Privacy:  Public Assigned to:  m1gu3l
Open/Closed:  Open
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

( Jump to the original submission )

Sat 07 Aug 2021 06:22:59 PM UTC, comment #10: 

I wonder if this is related to the following Battle for Wesnoth bug? https://github.com/wesnoth/wesnoth/issues/5989
A commenter seems to think some of the lack of reproducibility there is due to msgmerge...

Eric Gallager <egallager>
Fri 18 Dec 2020 05:55:10 PM UTC, comment #9: 

comment #8:

> 3) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/
> [...]
> https://reproducible-builds.org/docs/plans/


I'm very thankful of your comments and your pointers, as I haven't read until now most of the pages you're pointing out.  Nonetheless, your interpretation of these documents and their objectives isn't the same as mine.  Disagreement is a great thing, as we can learn new things from other points of view and understand each other a bit more.

My personal interpretation of those documents regarding the issue at hand is that the ability of "recreate bit-by-bit identical copies of all specified artifacts" is the final objective, and divergences from that are only accepted as a temporal workaround until everybody is on board---as freedom only comes with agreement, not by imposition.

Despite this I want to clarify that my main concern is a general design issue: I don't consider acceptable the modification of the input processing outside of the purpose of the tool in hand, only to cope with an unrelated problem and because "it fits there".  Sincerely, I wouldn't be happy if the attached patch which implements your proposal for msginit was "the contribution" that closed this bug report.

From my point of view, the behavior of msginit shouldn't change for this bug report, as shouldn't have changed msgfmt in that sense.  The timestamp used for the output has to be acknowledged as an input, and workarounds like dateshift are just that, not at all a sign of good design.  SOURCE_DATE_EPOCH, regardless of reproducible-builds.org, provides an already standard way of controlling exactly that input.

> 4)
> > they are symptoms, not the issue.  The impossibility of controlling one of the implicit inputs (the time) is the issue here.
>
> I disagree here, on two reasons:
> * In https://reproducible-builds.org/docs/timestamps/ the reproducible-builds.org people state that the preferred way to handle timestamps is to omit them from the output. This is what I propose (1) for the -email is unavailable- files. The SOURCE_DATE_EPOCH is a second-choice mechanism, which can be useful when you don't want to go into the details.

That link (at least on my computer) contains on the second paragraph of the header "timestamps are best avoided":

If a date is required to give users an idea on when the software was made, it is better to use a date that is relevant to the source code instead of the build

But you, and I agree on that, insist on the usefulness of the timestamp generated by xgettext---which was the main issue at hand on both bug reports as the msginit header year wasn't something to look at before rolling the hill up---, therefore their preferred way isn't omitting the timestamp but providing a meaningful value based on the source date and not the build date.

The responsibility of providing the right date is on the shoulders of the person executing the software, the responsibility of the software is to allow the control that through some mechanism or any other mean, which currently is an implicit and non-easily controllable input from the system clock.  sed, awk or even switching bits by hand can be done, but xgettext and msginit are the only ones who can avoid the need of any further cleanup after them when the builder wants to do exactly that.

> * The SOURCE_DATE_EPOCH mechanism is dangerous:

Dangerous?  Human response to danger is fear, which isn't a reason but a feeling, therefore I'm trying to answer from that point of view, because those feelings may be there and they should be acknowledged and treated as such.  I try to understand them along the responsibility of maintaining a project so useful and long lived, and as user I have to thank you for all your dedication and contributions towards making software accessible to everybody regardless their language.  For that reason, I'd like to move the terms of the discourse to advantages/benefits/disadvantages/risks related to them, and analyze the problem with that in mind.

> It can produce output that is not meaningful. I do not want an xgettext or msginit program that produces "POT-Creation-Date: 1970-01-01 00:00+0000" or "Automatically generated, 1971.", because that would lead to confusion and trouble for translators.

This is obviously a risk, but that would be reported as a bug by any translator to the creator of the POT, as it's their responsibility to provide a useful date---xgettext can only provide a mechanism for that.  Paraphrasing from the other thread, if any project maintainer wants to enforce the usage of the system clock, SOURCE_DATE_EPOCH can be unset right before the desired invocation, therefore this risk has a low final impact from my point of view.

Furthermore, the date retrieved from the system clock isn't meaningful either, as it doesn't follow the code but the invocation.  A recreation of an old POT file doesn't provide a sensible value here, so there is an advantage to the end users being able to control it.

> Everyone can modify the current time on a per-process basis, using tools such as dateshift http://www.linuxcertif.com/man/1/dateshift/ or time-warp.so https://www.thanassis.space/tricks.html . So, while everyone can generate bogus POT files, it should not be that easy.
>
> I want a feature that is not so easy to fool into producing bogus output, even if it covers only a special case of what you consider a general issue.

A system clock without battery nor ntp produces a similar output.  I guess you're here confusing the responsibilities of the one who executes the code from the one who creates it: the first one has the responsibility of providing the inputs, the second of processing the inputs and generating the output, again for the usage of the former.  To call "bogus" the desired output based on the input provided by the user may be a bit loaded.  The environment variables, as any command line parameter, are completely controlled by the caller: only a person who doesn't know how the execution process works can be fooled that way, which again is out of the scope of gettext.

> 5)
> > nobody defines SOURCE_DATE_EPOCH on their environment without a concrete objective, even less by default.
>
> They are defined (or will get defined over time) on many distros.

That isn't true on any interactive session on any current system, and I'd say that you don't have anything to worry in that sense in the near future.  Be sure that I'd raise as many bug reports as needed to anybody who has the nefarious idea of defining that environment variable by default on any kind of session used for actual user interaction.

I think that source of confusion here is because distros following reproducible-builds.org ideas define that variable only under their build processes for their distributable packages.

> See here e.g. for Fedora https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57

There is a big issue with that macro: the changelog date of the spec isn't a meaningful one because it isn't related to the source date and it's allowed to have a common spec for several versions of the same software.  The variable must be set to the maximum of either the source date or the spec changelog latest date, not only the latter.  Clearly any distro using that flag doesn't comply with the spirit of SOURCE_DATE_EPOCH spec.

> and here for RedHat https://bugzilla.redhat.com/show_bug.cgi?id=1793722 .

It's a shame that this was directly closed as NOTABUG even when it clearly is. The reaction wasn't very polite neither.  Sad. :(

> I'm sure that if I searched for the build recipes on openSUSE, I would find similar things.

Sure, because they use the same rpm flag: https://en.opensuse.org/openSUSE:Reproducible_Builds

> If these distros produce POT files in their source rpms, I don't want them to contain "POT-Creation-Date: 1970-01-01 00:00+0000".

That isn't the case in any of these projects.  They'd produce POT files with the latest .spec changelog entry date, which also is quite bad but nowhere close to your example.

I understand your worry about this point, but this isn't the forum to discuss their choices and neither your view point nor mine cannot be enforced over them.  As you said, they could already use any of the currently available workarounds to give a bogus but fixed date, but they are trying to give a meaningful one, even when it's clearly wrong for many workflows using .spec files.

> 6)
> > their suggested mechanism for git projects is:
> > SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
>
> OK, good to see that there is even a simpler format string directive for producing a pretty date.
>
> The difference between what you have in mind and what I prefer is that I prefer a setting in the Makevars file, that can be set by a package maintainer, and that a distro cannot abuse.

Distributions don't generate POT files for projects except for their software, so even in that case they will be well aware of the issue as they probably want to have their software translated, and they probably don't want to receive continuous complaints from their translators.

Wrapping up, I see two main options that would solve the issue:
1. The implementation of SOURCE_DATE_EPOCH that I propose.
2. The usage of a command line parameter or another variable like git (e.g. XGETTEXT_POT_CREATION_DATE).

A third option of adding code only to Makevars without any other change wouldn't really be backwards compatible nor practical for projects that call xgettext by other means, which I think is enough disadvantage to discard it.

1. provides an uniform interface, which I see as a benefit because it eases the cognitive burden of the computer user.  A disadvantage for this option is that RPM currently is misusing that variable.  The package maintainer/builder also has the advantage of only having to set this variable for several processes, e.g. help2man.

2. it provides an specific interface which can be extensively adapted and documented.  I'd be happy of hearing more about the benefits of this option, but I'm only able to see the disadvantages, which aren't as hard as with the third option: Makevars has to be changed if the option comes from the command line (and old software builds would need a wrapper anyway), or the cognitive burden of the caller must increase with another environment variable which really doesn't add nor require anything extra, unlike GIT_{COMMIT,AUTHOR}_DATE.

My position still stays with 1 because the advantages seem bigger than the disadvantages, risks and possible issues, speaking too as a maintainer of only 5 translations, but I'm open to hear your (rational) point of view, as well as I'd be thankful to read about any advantage and/or disadvantage that I haven't take into account, because I hope that we can finally agree on a way forward to transform this report into a nice new line in the NEWS file. :-)

(file #50513)

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member
Tue 15 Dec 2020 05:59:16 PM UTC, comment #8: 

3) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/ .

What are the sources: tarballs or git repos? I don't really mind if you count only the git repo as the source. Then the POT file and  the PO files became built files. And since they contain POT-Creation-Date field, people need to except this field to be different on each build.

This is OK even for the reproducible-builds.org people, since in https://reproducible-builds.org/docs/plans/ section "Providing a comparison protocol" they mention that while identical results are "ideal", "Other technologies might require removing cryptographic signatures or ignore specific parts." So, "diff -r" that removes POT-Creation-Date lines is OK for them.

Ultimately, it comes down to the question "can we trust these generated files?". The harder problem is with binary object code, because a small change in the source can lead to a big change in the output. But when comparing POT files and the difference is only in the POT-Creation-Date line, everyone will answer the question "can we trust these?" with yes.

4)

> they are symptoms, not the issue.  The impossibility of controlling one of the implicit inputs (the time) is the issue here.


I disagree here, on two reasons:

  • In https://reproducible-builds.org/docs/timestamps/ the reproducible-builds.org people state that the preferred way to handle timestamps is to omit them from the output. This is what I propose (1) for the -email is unavailable- files. The SOURCE_DATE_EPOCH is a second-choice mechanism, which can be useful when you don't want to go into the details.
  • The SOURCE_DATE_EPOCH mechanism is dangerous: It can produce output that is not meaningful. I do not want an xgettext or msginit program that produces "POT-Creation-Date: 1970-01-01 00:00+0000" or "Automatically generated, 1971.", because that would lead to confusion and trouble for translators.

Everyone can modify the current time on a per-process basis, using tools such as dateshift http://www.linuxcertif.com/man/1/dateshift/ or time-warp.so https://www.thanassis.space/tricks.html . So, while everyone can generate bogus POT files, it should not be that easy.

I want a feature that is not so easy to fool into producing bogus output, even if it covers only a special case of what you consider a general issue.

5)

> nobody defines SOURCE_DATE_EPOCH on their environment without a concrete objective, even less by default.


They are defined (or will get defined over time) on many distros. See here e.g. for Fedora https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57 and here for RedHat https://bugzilla.redhat.com/show_bug.cgi?id=1793722 . I'm sure that if I searched for the build recipes on openSUSE, I would find similar things.

If these distros produce POT files in their source rpms, I don't want them to contain "POT-Creation-Date: 1970-01-01 00:00+0000".

6)

> their suggested mechanism for git projects is:
> SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)


OK, good to see that there is even a simpler format string directive for producing a pretty date.

The difference between what you have in mind and what I prefer is that I prefer a setting in the Makevars file, that can be set by a package maintainer, and that a distro cannot abuse.

Bruno Haible <haible>
Project Administrator
Tue 15 Dec 2020 02:36:48 AM UTC, comment #7: 

As a general comment: v3 patch only define a macro and include the header into xgettext.c and msginit.c, no other modification is needed and all the handling has been moved to gnulib (patch included too for the sake of completeness.)  The tests remain mostly the same, with the removal of the sleep at xgettext-17, and the fixed timezone..

From comment #6:

> 1) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/ .
>
> The "artifacts" defined there is what gets installed through "make install" and packaged as an installable package (rpm, deb, whatever) by the distro.

That definition of artifacts mentions distribution packages explicitly, nowhere "installed".  Even rpm and deb can create source packages too, but nowadays GNU tarballs are artifacts of that kind by themselves, as they are generated from their corresponding VCS used for the actual development by make dist.

> The "source code" defined there is a tarball, because GNU provides its software through releases and tarballs, and for the vast majority of packages, this is what the distros take as their inputs.

That may be a common case, but at least most of my gettext compilations start from a git checkout, even when the one I usually use is the one compiled by the distro or from a release when it isn't possible.  Isn't that your case too?

Indeed the GPL is very clear in that respect:

The “source code” for a work means the preferred form of the work for making modifications to it.

I'm not aware of any current project that generates and modify manually POT files, nor it would be the scope of xgettext in that case.  From the same paragraph of the GPL:

“Object code” means any non-source form of a work.

It can easily be interpreted that POT files emitted by xgettext, exactly the same as -email is unavailable- by msginit, and unlike po files from translators, are "object code" for the GPL because they are not used for modifications of the work itself but subproducts of its generation, just like the .o files.

Note that tarballs of GNU software releases include object code, on purpose, to not require maintainer dependencies as autotools, grammar parsers, vala, texinfo, etc. for the compilation of that version.

> In this area, the -email is unavailable- and -email is unavailable- files are an issue, and the solution I propose is that 'msginit --no-translator' omits the PO-Revision-Date from the output.

As I said, they are symptoms, not the issue.  The impossibility of controlling one of the implicit inputs (the time) is the issue here.

> 2) "As a software developer I'd like to be sure that anybody can reproduce the distributed tarball from the sources used to generate it"
>
> I guess you mean "as a software developer of a package that uses GNU gettext for its po/ directory". [...] The "sources" in this case are a version control repository (most likely).
>
> In the current state of things, with issue 1) fixed, two tarballs produced by different people will contain identical files, except for [...].

Sorry, they are identical or they are not: the return codes of diff are different for exactly that reason.  The issue 1 fixed means that they are identical given the same inputs: sources, build environment (including SOURCE_DATE_EPOCH and TZ) and build instructions.

> Do we want to have it differently by default? No. The POT-Creation-Date values in the .pot and .po files are important for the translator workflow.

As I said, I'm well aware of this, but nobody defines SOURCE_DATE_EPOCH on their environment without a concrete objective, even less by default.

> Do we want the possibility for tarball builders to force a "POT-Creation-Date: 1970-01-01 00:00:00" in all .pot and .po files? No. If such a tarball gets in the hands of a translator, her workflow is negatively impacted.

Of course it would impact the workflow negatively, but that date can be modified anyway with sed -e 's,(POT-Creation-Date:)[^\\]*,\\1 1970-01-01 00:00+0000'.  I cannot understand the reason behind your concern, as their suggested mechanism for git projects is:

SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

> Should the POT-Creation-Date be replaced by something else that is not a date? Since non-linear versioning control systems are in  use (e.g. git with branches), people are sometimes using a 'git describe' output for the version number. But for a translator, a version number "3.4-rc1" definitely makes more sense than "3.4-217-ce814b" or - even worse - just "ce814b".

This is already encoded in the project version, something of the output that already can be controlled unlike POT-Creation-Date.  The proposed mechanism with git gets the best POT-Creation-Date you can expect based on the current format---julian/gregorian based dates are "good enough" too, but that's another social+cultural+political issue and not the one at hand.

> Should the POT-Creation-Date be replaced by a date that is not the current date but instead reflects the latest modification to the version control history? I am open to such a thing.

These patches allow exactly this without modifying the current build scripts for any project.  Please try this with the patches applied, either v2 or v3+gnulib:

cd /a/git/project/po
ts=`git log -1 --pretty=%ct`
gt=/compiled/gettext-tools/src
rm -f project.pot
PATH="$gt:$PATH" SOURCE_DATE_EPOCH=$ts make update-po
mv project.pot project.pot.v1
PATH="$gt:$PATH" SOURCE_DATE_EPOCH=$ts make update-po
diff project.pot project.pot.v1 || echo "Oops, something's broken"

(file #50464, file #50465)

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member
Sun 13 Dec 2020 12:41:22 PM UTC, comment #6: 

We need to separate the issues.

1) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/ .

The "artifacts" defined there is what gets installed through "make install" and packaged as an installable package (rpm, deb, whatever) by the distro. This includes the .gmo files; it does not include a POT file nor any .po files.

The "source code" defined there is a tarball, because GNU provides its software through releases and tarballs, and for the vast majority of packages, this is what the distros take as their inputs.

In this area, the -email is unavailable- and -email is unavailable- files are an issue, and the solution I propose is that 'msginit --no-translator' omits the PO-Revision-Date from the output.

2) "As a software developer I'd like to be sure that anybody can reproduce the distributed tarball from the sources used to generate it"

I guess you mean "as a software developer of a package that uses GNU gettext for its po/ directory". (Because for GNU gettext itself, the tarballs are authenticated through the signature of the file on ftp.gnu.org, and thus there is no need to create another tarball for comparison.)

The "sources" in this case are a version control repository (most likely).

In the current state of things, with issue 1) fixed, two tarballs produced by different people will contain identical files, except for POT-Creation-Date values in the .pot and .po files. No differences in the *.gmo files. Someone who compares such different tarballs can do so with a textual compare ("diff -r"). No need for special tools like 'diffeoscope'.

To me, that is good enough.

Do we want to have it differently by default? No. The POT-Creation-Date values in the .pot and .po files are important for the translator workflow.

Do we want the possibility for tarball builders to force a "POT-Creation-Date: 1970-01-01 00:00:00" in all .pot and .po files? No. If such a tarball gets in the hands of a translator, her workflow is negatively impacted.

Should the POT-Creation-Date be replaced by something else that is not a date? Since non-linear versioning control systems are in  use (e.g. git with branches), people are sometimes using a 'git describe' output for the version number. But for a translator, a version number "3.4-rc1" definitely makes more sense than "3.4-217-ce814b" or - even worse - just "ce814b".

Should the POT-Creation-Date be replaced by a date that is not the current date but instead reflects the latest modification to the version control history? I am open to such a thing. It could be enabled by the package maintainer in the 'Makevars' file.
The code to produce the date could be something like:

git_checkout_date=`git log -n 1 --date=iso --format=fuller | sed -n -e 's/^CommitDate: //p'`
pretty_date=`LC_ALL=C date +"%e %B %Y" --date="$git_checkout_date"`

Bruno Haible <haible>
Project Administrator
Sun 13 Dec 2020 01:41:19 AM UTC, comment #5: 

From comment #2:

> This is a bit complex. I claim there is a simpler solution.
>
> In https://savannah.gnu.org/bugs/?49654 I wrote:
> "The POT-Creation-Date is important information for a translator, so that she knows whether to review the entire file or not."
>
> Similarly, the PO-Revision-Date is important information for a translator, so that she knows whether she updated the file after she got a new POT file.
>
> But in this case (en@quot catalog), there is no translator. Therefore no PO-Revision-Date is needed.

IMHO this doesn't address the main issue: the output of xgettext and msginit cannot be easily generated again after one minute (xgettext) or after one year (msginit) in the scenario you suggest---msginit also replaces the YEAR template from the source header.  These tags are the symptom, not the actual issue.

As a translator I consider these tags very useful, although from that POV the date of the last update to the VCS is usually more meaningful and reproducible than a timestamp of the make dist/update-po invocation on the maintainer machine.  Only walking with that shoes I'd even bring back the old behavior of msgfmt, perhaps as a flag, keeping POT-Creation-Date inside .mo files as far as its contents are reproducible, to retain that traceability in the binary file too; msgfmt-19 could easily be extended to cover that use case too (e.g. --keep-pot-creation-date to keep the default since 0.20 or --remove-pot-creation-date to revert to the old default.)

As a software developer I'd like to be sure that anybody can reproduce the distributed tarball from the sources used to generate it: MO files generated by msgfmt should be as reproducible as POT files generated by xgettext and PO files generated msginit.

The usage of SOURCE_DATE_EPOCH satisfies both points of view (with extra advantages) and doesn't trim any input data as should be expected.

From comment #3:

> Can you submit a Gnulib module for SOURCE_DATE_EPOCH handling?

I'm glad you ask.  I though about this while working on it, as any time/localtime call introduce that kind of reproducibility problems.  Reading gnulib's manual I see that at least one project has to use it.  GCC has some code for it, which I'm using to discover missing points from my implementation: errno, long vs long long...

Nonetheless, I guess that a library interface should allow client code to identify issues with the variable as suggested by the specification, what do you think about this interface?

/* Read the environment variable SOURCE_DATE_EPOCH into the provided
   time_t object.

   Return 0 when the value has been read successfully, or number of
   characters left to read (up to INT_MAX) at SOURCE_DATE_EPOCH if the
   number isn't well formed.

   Return -1 when the variable isn't defined or -2 when the variable
   is defined but empty.

   See https://reproducible-builds.org/specs/source-date-epoch/  */
int source_date_epoch_time (time_t *);

/* Like source_date_epoch_time but read into struct tm.  */
int source_date_epoch_tm (struct tm *);

It allows the client code to ignore any error like this:

if (source_date_epoch_time (&now) != 0)
  /* oops: SOURCE_DATE_EPOCH is not there, perhaps call time (&now),
     show a message, check errno...  */

Or to handle them:

switch (source_date_epoch_time (&now))
{
  case -2:
    /* Do the thing when it is defined but empty, fall through?  */
  case -1:
    /* Do the thing when it isn't defined.  */
    break;
  case 0:
    /* Do the thing when it is defined and OK.  */
    break;
  default:
    /* Exit with non-zero as suggested by the spec.  */
    exit (1);
}

Also, I've noticed that the common idiom is to define TZ together with SOURCE_DATE_EPOCH, as gettext's copy of help2man does, even though the spec says that it must be treated always as UTC.  If that assumption is made, the modifications to po_strftime can be removed and we can forget about any issue regarding the localtime/gmtime because they will return the same value.  If not, I suggest keeping the interface modification from the latest patch.

From comment #4:

> > Is this time requirement acceptable?
> A time requirement of 1 or 2 seconds is acceptable. A time requirement of 61 seconds is not; this is excessive.

That's the main issue with wall clock tests, they need an annoying amount of time.  It would be worse for the msginit test, as a call to 'sleep 1y' would have been indeed even less acceptable. :-)

Jokes aside, the test can be trimmed to the main functionality as msginit-5, instead of depending on the wall clock to ensure it isn't used when SOURCE_DATE_EPOCH is provided, but I preferred to err on the safe side and provide the test with the full behavior just in case.

I'm preparing the patches based on these ideas, I hope they'll be available soon.

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member
Sat 12 Dec 2020 02:11:23 PM UTC, comment #4: 

> Is this time requirement acceptable?

A time requirement of 1 or 2 seconds is acceptable. A time requirement of 61 seconds is not; this is excessive.

The gnulib test 'test-lock' often takes a long time. But this is because on platforms with buggy locks, within just 5 or 10 seconds, it would not crash. It really needs a minute or so to distinguish good and buggy implementations of locks. And locks are a fundamental element of the code; SOURCE_DATE_EPOCH is not.

Bruno Haible <haible>
Project Administrator
Sat 12 Dec 2020 02:05:53 PM UTC, comment #3: 

Independently of GNU gettext: The SOURCE_DATE_EPOCH handling that you implemented (especially w.r.t. the time zone) seems to be non-trivial. Since https://reproducible-builds.org/specs/source-date-epoch/ applies to many programs used in a build process, a Gnulib module with this functionality would be useful.

Can you submit a Gnulib module for SOURCE_DATE_EPOCH handling?

Bruno Haible <haible>
Project Administrator
Sat 12 Dec 2020 02:00:55 PM UTC, comment #2: 

> this still leads to unreproducible builds when LINGUAS contain automatically generated PO files (e.g en@quot) and the POT file is generated/updated at build time (e.g. building from a vcs checkout where the POT file isn't present or up to date), as the contents of that field are used to fill PO-Revision-Date when the PO file is created with the --no-translator option.


Indeed, yes. Well spotted.

> The attached patch implements the default use-case of SOURCE_DATE_EPOCH


This is a bit complex. I claim there is a simpler solution.

In https://savannah.gnu.org/bugs/?49654 I wrote:
"The POT-Creation-Date is important information for a translator, so that she knows whether to review the entire file or not."

Similarly, the PO-Revision-Date is important information for a translator, so that she knows whether she updated the file after she got a new POT file.

But in this case (en@quot catalog), there is no translator. Therefore no PO-Revision-Date is needed.

Now, let's look at the steps:
1. xgettext produces a POT file with a POT-Creation-Date (current date) and with a dummy PO-Revision-Date (YEAR-MO-DA HO:MI+ZONE).
2. -email is unavailable- is created through

msginit -i gettext-runtime.pot --no-translator -l en@quot -o - 2>/dev/null \
| sed -f en@quot.insert-header \
| msgconv -t UTF-8 \
| msgfilter quot

3. -email is unavailable- is created through

rm -f en@quot.gmo
msgmerge --for-msgfmt -o en@quot.1po en@quot.po gettext-runtime.pot
msgfmt -c --statistics --verbose -o en@quot.gmo en@quot.1po
rm -f en@quot.1po

Should the PO-Revision-Date be present after step 1? Yes, since the same POT file is used by the translators.

Should the PO-Revision-Date be present after step 3? No, since it is a binary file that will be installed and should therefore not contain such a date.

Should the PO-Revision-Date be present after step 2? No, since this file is present in tarballs, and the date of production of the tarball is irrelevant.

Where in step 2 should the PO-Revision-Date be removed? The msginit step seems to be the right place, since the information that it's not for a translator is present there (command-line option '--no-translator'). No need to add another step in the pipe.

So, instead of the patch that you propose, I would prefer a simpler approach: If msginit is invoked with option '--no-translator', it omits the PO-Revision-Date from the output.

Bruno Haible <haible>
Project Administrator
Sat 12 Dec 2020 12:18:24 AM UTC, comment #1: 

The v2 adds changes a bit the interface of po_strftime in order to provide the UTC field, fixes the missing call to time/localtime on msginit and adds two new tests to the test suite: msginit-5 and xgettext-17.

xgettext-17 takes, unless skipped, at least one minute to complete in order to ensure that the output differs when SOURCE_DATE_EPOCH isn't provided but stays the same when the value is the same, even when the wall clock has changed enough.  Is this time requirement acceptable?  It also checks that the output is controlled by the variable and the program doesn't emit a fixed timestamp; it could be reduced to only that use case if deemed appropriate.

(file #50444)

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member
Fri 11 Dec 2020 01:19:33 PM UTC, original submission:  

This is an extension of the problem reported and solved by https://savannah.gnu.org/bugs/?49654

Version 0.20 excluded POT-Creation-Date from generated .mo files, but this still leads to unreproducible builds when LINGUAS contain automatically generated PO files (e.g en@quot) and the POT file is generated/updated at build time (e.g. building from a vcs checkout where the POT file isn't present or up to date), as the contents of that field are used to fill PO-Revision-Date when the PO file is created with the --no-translator option.

The attached patch implements the default use-case of SOURCE_DATE_EPOCH, but it doesn't signal any error from the variable (allowed by the spec) and simply continues with the usual local timestamp process.  Its value is used to fill POT-Creation-Date for POT files generated by xgettext, and PO-Revision-Date for PO files generated by msginit.

Perhaps the test suite should be extended and/or modified to use more this feature.  What do you think?

Miguel Ángel Arruga Vivas <m1gu3l>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50513:  0001-msginit-Do-not-use-POT-Creation-Date.patch added by m1gu3l (1KiB - text/x-patch - Bruno's proposal: Do not insert PO-Revision-Date with --no-translator)
file #50466:  0001-source-date-epoch-New-module.patch added by m1gu3l (15KiB - text/x-patch - gnulib v1.1 (removed leftover line))
file #50464:  v3-0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-time.patch added by m1gu3l (11KiB - text/x-patch - v3 gettext + v1 gnulib)
file #50444:  0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-timesta.patch added by m1gu3l (16KiB - text/x-patch - Patch v2 (new interface + msginit-5, xgettext-17))

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by egallager (Posted a comment)
  • -email is unavailable- added by egallager (Voted in favor of this item)
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by m1gu3l (Submitted the item)
  •  

    There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-08-07 egallager Carbon-Copy- Added egallager
    2020-12-22 m1gu3l Summaryxgettext and msginit don't honor SOURCE_DATE_EPOCH xgettext and msginit output isn't reproducible
    2020-12-18 m1gu3l Attached File- Added 0001-msginit-Do-not-use-POT-Creation-Date.patch, #50513
    2020-12-15 m1gu3l Attached File- Added 0001-source-date-epoch-New-module.patch, #50466
    2020-12-15 m1gu3l Attached File#50465 Removed
    2020-12-15 m1gu3l Attached File- Added v3-0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-time.patch, #50464
        Attached File- Added 0001-source-date-epoch-New-module.patch, #50465
    2020-12-12 haible StatusReady For Test In Progress
    2020-12-12 m1gu3l StatusIn Progress Ready For Test
    2020-12-12 m1gu3l Attached File- Added 0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-timesta.patch, #50444
    2020-12-11 haible Summaryxgettext and msgint don't honor SOURCE_DATE_EPOCH xgettext and msginit don't honor SOURCE_DATE_EPOCH
    2020-12-11 m1gu3l Attached File- Added 0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-timesta.patch, #50438

    Back to the top


    Powered by Savane 3.9