GNU gettext - Bugs: bug #59658, xgettext and msginit output isn't...
You are not allowed to post comments on this tracker with your current authentication level.
bug #59658: xgettext and msginit output isn't reproducible
Submitter: | Miguel Ángel Arruga Vivas <m1gu3l> | ||
Submitted: | Fri 11 Dec 2020 01:19:33 PM UTC | ||
Votes: | 1 | ||
Category: | Programmer tools | Severity: | 3 - Normal |
Item Group: | None | Status: | In Progress |
Privacy: | Public | Assigned to: | m1gu3l |
Open/Closed: | Open |
( Jump to the original submission )
Sat 07 Aug 2021 06:22:59 PM UTC, comment #10: |
Eric Gallager <egallager> |
Fri 18 Dec 2020 05:55:10 PM UTC, comment #9:
My personal interpretation of those documents regarding the issue at hand is that the ability of "recreate bit-by-bit identical copies of all specified artifacts" is the final objective, and divergences from that are only accepted as a temporal workaround until everybody is on board---as freedom only comes with agreement, not by imposition.
Despite this I want to clarify that my main concern is a general design issue: I don't consider acceptable the modification of the input processing outside of the purpose of the tool in hand, only to cope with an unrelated problem and because "it fits there". Sincerely, I wouldn't be happy if the attached patch which implements your proposal for msginit was "the contribution" that closed this bug report.
From my point of view, the behavior of msginit shouldn't change for this bug report, as shouldn't have changed msgfmt in that sense. The timestamp used for the output has to be acknowledged as an input, and workarounds like dateshift are just that, not at all a sign of good design. SOURCE_DATE_EPOCH, regardless of reproducible-builds.org, provides an already standard way of controlling exactly that input.
That link (at least on my computer) contains on the second paragraph of the header "timestamps are best avoided":
But you, and I agree on that, insist on the usefulness of the timestamp generated by xgettext---which was the main issue at hand on both bug reports as the msginit header year wasn't something to look at before rolling the hill up---, therefore their preferred way isn't omitting the timestamp but providing a meaningful value based on the source date and not the build date.
The responsibility of providing the right date is on the shoulders of the person executing the software, the responsibility of the software is to allow the control that through some mechanism or any other mean, which currently is an implicit and non-easily controllable input from the system clock. sed, awk or even switching bits by hand can be done, but xgettext and msginit are the only ones who can avoid the need of any further cleanup after them when the builder wants to do exactly that.
Dangerous? Human response to danger is fear, which isn't a reason but a feeling, therefore I'm trying to answer from that point of view, because those feelings may be there and they should be acknowledged and treated as such. I try to understand them along the responsibility of maintaining a project so useful and long lived, and as user I have to thank you for all your dedication and contributions towards making software accessible to everybody regardless their language. For that reason, I'd like to move the terms of the discourse to advantages/benefits/disadvantages/risks related to them, and analyze the problem with that in mind.
This is obviously a risk, but that would be reported as a bug by any translator to the creator of the POT, as it's their responsibility to provide a useful date---xgettext can only provide a mechanism for that. Paraphrasing from the other thread, if any project maintainer wants to enforce the usage of the system clock, SOURCE_DATE_EPOCH can be unset right before the desired invocation, therefore this risk has a low final impact from my point of view.
Furthermore, the date retrieved from the system clock isn't meaningful either, as it doesn't follow the code but the invocation. A recreation of an old POT file doesn't provide a sensible value here, so there is an advantage to the end users being able to control it.
A system clock without battery nor ntp produces a similar output. I guess you're here confusing the responsibilities of the one who executes the code from the one who creates it: the first one has the responsibility of providing the inputs, the second of processing the inputs and generating the output, again for the usage of the former. To call "bogus" the desired output based on the input provided by the user may be a bit loaded. The environment variables, as any command line parameter, are completely controlled by the caller: only a person who doesn't know how the execution process works can be fooled that way, which again is out of the scope of gettext.
That isn't true on any interactive session on any current system, and I'd say that you don't have anything to worry in that sense in the near future. Be sure that I'd raise as many bug reports as needed to anybody who has the nefarious idea of defining that environment variable by default on any kind of session used for actual user interaction.
I think that source of confusion here is because distros following reproducible-builds.org ideas define that variable only under their build processes for their distributable packages.
There is a big issue with that macro: the changelog date of the spec isn't a meaningful one because it isn't related to the source date and it's allowed to have a common spec for several versions of the same software. The variable must be set to the maximum of either the source date or the spec changelog latest date, not only the latter. Clearly any distro using that flag doesn't comply with the spirit of SOURCE_DATE_EPOCH spec.
It's a shame that this was directly closed as NOTABUG even when it clearly is. The reaction wasn't very polite neither. Sad. :(
Sure, because they use the same rpm flag: https://en.opensuse.org/openSUSE:Reproducible_Builds
That isn't the case in any of these projects. They'd produce POT files with the latest .spec changelog entry date, which also is quite bad but nowhere close to your example.
I understand your worry about this point, but this isn't the forum to discuss their choices and neither your view point nor mine cannot be enforced over them. As you said, they could already use any of the currently available workarounds to give a bogus but fixed date, but they are trying to give a meaningful one, even when it's clearly wrong for many workflows using .spec files.
Distributions don't generate POT files for projects except for their software, so even in that case they will be well aware of the issue as they probably want to have their software translated, and they probably don't want to receive continuous complaints from their translators.
Wrapping up, I see two main options that would solve the issue:
A third option of adding code only to Makevars without any other change wouldn't really be backwards compatible nor practical for projects that call xgettext by other means, which I think is enough disadvantage to discard it.
1. provides an uniform interface, which I see as a benefit because it eases the cognitive burden of the computer user. A disadvantage for this option is that RPM currently is misusing that variable. The package maintainer/builder also has the advantage of only having to set this variable for several processes, e.g. help2man.
2. it provides an specific interface which can be extensively adapted and documented. I'd be happy of hearing more about the benefits of this option, but I'm only able to see the disadvantages, which aren't as hard as with the third option: Makevars has to be changed if the option comes from the command line (and old software builds would need a wrapper anyway), or the cognitive burden of the caller must increase with another environment variable which really doesn't add nor require anything extra, unlike GIT_{COMMIT,AUTHOR}_DATE.
My position still stays with 1 because the advantages seem bigger than the disadvantages, risks and possible issues, speaking too as a maintainer of only 5 translations, but I'm open to hear your (rational) point of view, as well as I'd be thankful to read about any advantage and/or disadvantage that I haven't take into account, because I hope that we can finally agree on a way forward to transform this report into a nice new line in the NEWS file. :-) |
Miguel Ángel Arruga Vivas <m1gu3l>![]() ![]() |
Tue 15 Dec 2020 05:59:16 PM UTC, comment #8: 3) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/ .
What are the sources: tarballs or git repos? I don't really mind if you count only the git repo as the source. Then the POT file and the PO files became built files. And since they contain POT-Creation-Date field, people need to except this field to be different on each build.
This is OK even for the reproducible-builds.org people, since in https://reproducible-builds.org/docs/plans/ section "Providing a comparison protocol" they mention that while identical results are "ideal", "Other technologies might require removing cryptographic signatures or ignore specific parts." So, "diff -r" that removes POT-Creation-Date lines is OK for them.
Ultimately, it comes down to the question "can we trust these generated files?". The harder problem is with binary object code, because a small change in the source can lead to a big change in the output. But when comparing POT files and the difference is only in the POT-Creation-Date line, everyone will answer the question "can we trust these?" with yes.
4)
Everyone can modify the current time on a per-process basis, using tools such as dateshift http://www.linuxcertif.com/man/1/dateshift/ or time-warp.so https://www.thanassis.space/tricks.html . So, while everyone can generate bogus POT files, it should not be that easy.
I want a feature that is not so easy to fool into producing bogus output, even if it covers only a special case of what you consider a general issue.
5)
If these distros produce POT files in their source rpms, I don't want them to contain "POT-Creation-Date: 1970-01-01 00:00+0000".
6)
The difference between what you have in mind and what I prefer is that I prefer a setting in the Makevars file, that can be set by a package maintainer, and that a distro cannot abuse. |
Bruno Haible <haible>![]() |
Tue 15 Dec 2020 02:36:48 AM UTC, comment #7: As a general comment: v3 patch only define a macro and include the header into xgettext.c and msginit.c, no other modification is needed and all the handling has been moved to gnulib (patch included too for the sake of completeness.) The tests remain mostly the same, with the removal of the sleep at xgettext-17, and the fixed timezone..
From comment #6:
That definition of artifacts mentions distribution packages explicitly, nowhere "installed". Even rpm and deb can create source packages too, but nowadays GNU tarballs are artifacts of that kind by themselves, as they are generated from their corresponding VCS used for the actual development by make dist.
That may be a common case, but at least most of my gettext compilations start from a git checkout, even when the one I usually use is the one compiled by the distro or from a release when it isn't possible. Isn't that your case too?
Indeed the GPL is very clear in that respect:
I'm not aware of any current project that generates and modify manually POT files, nor it would be the scope of xgettext in that case. From the same paragraph of the GPL:
It can easily be interpreted that POT files emitted by xgettext, exactly the same as -email is unavailable- by msginit, and unlike po files from translators, are "object code" for the GPL because they are not used for modifications of the work itself but subproducts of its generation, just like the .o files.
Note that tarballs of GNU software releases include object code, on purpose, to not require maintainer dependencies as autotools, grammar parsers, vala, texinfo, etc. for the compilation of that version.
As I said, they are symptoms, not the issue. The impossibility of controlling one of the implicit inputs (the time) is the issue here.
Sorry, they are identical or they are not: the return codes of diff are different for exactly that reason. The issue 1 fixed means that they are identical given the same inputs: sources, build environment (including SOURCE_DATE_EPOCH and TZ) and build instructions.
As I said, I'm well aware of this, but nobody defines SOURCE_DATE_EPOCH on their environment without a concrete objective, even less by default.
Of course it would impact the workflow negatively, but that date can be modified anyway with sed -e 's,(POT-Creation-Date:)[^\\]*,\\1 1970-01-01 00:00+0000'. I cannot understand the reason behind your concern, as their suggested mechanism for git projects is:
This is already encoded in the project version, something of the output that already can be controlled unlike POT-Creation-Date. The proposed mechanism with git gets the best POT-Creation-Date you can expect based on the current format---julian/gregorian based dates are "good enough" too, but that's another social+cultural+political issue and not the one at hand.
These patches allow exactly this without modifying the current build scripts for any project. Please try this with the patches applied, either v2 or v3+gnulib:
|
Miguel Ángel Arruga Vivas <m1gu3l>![]() ![]() |
Sun 13 Dec 2020 12:41:22 PM UTC, comment #6: We need to separate the issues.
1) What the "reproducible builds" initiative is about: See https://reproducible-builds.org/docs/definition/ .
The "artifacts" defined there is what gets installed through "make install" and packaged as an installable package (rpm, deb, whatever) by the distro. This includes the .gmo files; it does not include a POT file nor any .po files.
The "source code" defined there is a tarball, because GNU provides its software through releases and tarballs, and for the vast majority of packages, this is what the distros take as their inputs.
In this area, the -email is unavailable- and -email is unavailable- files are an issue, and the solution I propose is that 'msginit --no-translator' omits the PO-Revision-Date from the output.
2) "As a software developer I'd like to be sure that anybody can reproduce the distributed tarball from the sources used to generate it"
I guess you mean "as a software developer of a package that uses GNU gettext for its po/ directory". (Because for GNU gettext itself, the tarballs are authenticated through the signature of the file on ftp.gnu.org, and thus there is no need to create another tarball for comparison.)
The "sources" in this case are a version control repository (most likely).
In the current state of things, with issue 1) fixed, two tarballs produced by different people will contain identical files, except for POT-Creation-Date values in the .pot and .po files. No differences in the *.gmo files. Someone who compares such different tarballs can do so with a textual compare ("diff -r"). No need for special tools like 'diffeoscope'.
To me, that is good enough.
Do we want to have it differently by default? No. The POT-Creation-Date values in the .pot and .po files are important for the translator workflow.
Do we want the possibility for tarball builders to force a "POT-Creation-Date: 1970-01-01 00:00:00" in all .pot and .po files? No. If such a tarball gets in the hands of a translator, her workflow is negatively impacted.
Should the POT-Creation-Date be replaced by something else that is not a date? Since non-linear versioning control systems are in use (e.g. git with branches), people are sometimes using a 'git describe' output for the version number. But for a translator, a version number "3.4-rc1" definitely makes more sense than "3.4-217-ce814b" or - even worse - just "ce814b".
Should the POT-Creation-Date be replaced by a date that is not the current date but instead reflects the latest modification to the version control history? I am open to such a thing. It could be enabled by the package maintainer in the 'Makevars' file.
|
Bruno Haible <haible>![]() |
Sun 13 Dec 2020 01:41:19 AM UTC, comment #5: From comment #2:
IMHO this doesn't address the main issue: the output of xgettext and msginit cannot be easily generated again after one minute (xgettext) or after one year (msginit) in the scenario you suggest---msginit also replaces the YEAR template from the source header. These tags are the symptom, not the actual issue.
As a translator I consider these tags very useful, although from that POV the date of the last update to the VCS is usually more meaningful and reproducible than a timestamp of the make dist/update-po invocation on the maintainer machine. Only walking with that shoes I'd even bring back the old behavior of msgfmt, perhaps as a flag, keeping POT-Creation-Date inside .mo files as far as its contents are reproducible, to retain that traceability in the binary file too; msgfmt-19 could easily be extended to cover that use case too (e.g. --keep-pot-creation-date to keep the default since 0.20 or --remove-pot-creation-date to revert to the old default.)
As a software developer I'd like to be sure that anybody can reproduce the distributed tarball from the sources used to generate it: MO files generated by msgfmt should be as reproducible as POT files generated by xgettext and PO files generated msginit.
The usage of SOURCE_DATE_EPOCH satisfies both points of view (with extra advantages) and doesn't trim any input data as should be expected.
From comment #3:
I'm glad you ask. I though about this while working on it, as any time/localtime call introduce that kind of reproducibility problems. Reading gnulib's manual I see that at least one project has to use it. GCC has some code for it, which I'm using to discover missing points from my implementation: errno, long vs long long...
Nonetheless, I guess that a library interface should allow client code to identify issues with the variable as suggested by the specification, what do you think about this interface?
It allows the client code to ignore any error like this:
Or to handle them:
Also, I've noticed that the common idiom is to define TZ together with SOURCE_DATE_EPOCH, as gettext's copy of help2man does, even though the spec says that it must be treated always as UTC. If that assumption is made, the modifications to po_strftime can be removed and we can forget about any issue regarding the localtime/gmtime because they will return the same value. If not, I suggest keeping the interface modification from the latest patch.
From comment #4:
That's the main issue with wall clock tests, they need an annoying amount of time. It would be worse for the msginit test, as a call to 'sleep 1y' would have been indeed even less acceptable. :-)
Jokes aside, the test can be trimmed to the main functionality as msginit-5, instead of depending on the wall clock to ensure it isn't used when SOURCE_DATE_EPOCH is provided, but I preferred to err on the safe side and provide the test with the full behavior just in case.
I'm preparing the patches based on these ideas, I hope they'll be available soon. |
Miguel Ángel Arruga Vivas <m1gu3l>![]() ![]() |
Sat 12 Dec 2020 02:11:23 PM UTC, comment #4:
A time requirement of 1 or 2 seconds is acceptable. A time requirement of 61 seconds is not; this is excessive.
The gnulib test 'test-lock' often takes a long time. But this is because on platforms with buggy locks, within just 5 or 10 seconds, it would not crash. It really needs a minute or so to distinguish good and buggy implementations of locks. And locks are a fundamental element of the code; SOURCE_DATE_EPOCH is not. |
Bruno Haible <haible>![]() |
Sat 12 Dec 2020 02:05:53 PM UTC, comment #3: Independently of GNU gettext: The SOURCE_DATE_EPOCH handling that you implemented (especially w.r.t. the time zone) seems to be non-trivial. Since https://reproducible-builds.org/specs/source-date-epoch/ applies to many programs used in a build process, a Gnulib module with this functionality would be useful.
Can you submit a Gnulib module for SOURCE_DATE_EPOCH handling? |
Bruno Haible <haible>![]() |
Sat 12 Dec 2020 02:00:55 PM UTC, comment #2:
In https://savannah.gnu.org/bugs/?49654 I wrote:
Similarly, the PO-Revision-Date is important information for a translator, so that she knows whether she updated the file after she got a new POT file.
But in this case (en@quot catalog), there is no translator. Therefore no PO-Revision-Date is needed.
Now, let's look at the steps:
3. -email is unavailable- is created through
Should the PO-Revision-Date be present after step 1? Yes, since the same POT file is used by the translators.
Should the PO-Revision-Date be present after step 3? No, since it is a binary file that will be installed and should therefore not contain such a date.
Should the PO-Revision-Date be present after step 2? No, since this file is present in tarballs, and the date of production of the tarball is irrelevant.
Where in step 2 should the PO-Revision-Date be removed? The msginit step seems to be the right place, since the information that it's not for a translator is present there (command-line option '--no-translator'). No need to add another step in the pipe.
So, instead of the patch that you propose, I would prefer a simpler approach: If msginit is invoked with option '--no-translator', it omits the PO-Revision-Date from the output. |
Bruno Haible <haible>![]() |
Sat 12 Dec 2020 12:18:24 AM UTC, comment #1: The v2 adds changes a bit the interface of po_strftime in order to provide the UTC field, fixes the missing call to time/localtime on msginit and adds two new tests to the test suite: msginit-5 and xgettext-17.
xgettext-17 takes, unless skipped, at least one minute to complete in order to ensure that the output differs when SOURCE_DATE_EPOCH isn't provided but stays the same when the value is the same, even when the wall clock has changed enough. Is this time requirement acceptable? It also checks that the output is controlled by the variable and the program doesn't emit a fixed timestamp; it could be reduced to only that use case if deemed appropriate. |
Miguel Ángel Arruga Vivas <m1gu3l>![]() ![]() |
Fri 11 Dec 2020 01:19:33 PM UTC, original submission:
This is an extension of the problem reported and solved by https://savannah.gnu.org/bugs/?49654
Version 0.20 excluded POT-Creation-Date from generated .mo files, but this still leads to unreproducible builds when LINGUAS contain automatically generated PO files (e.g en@quot) and the POT file is generated/updated at build time (e.g. building from a vcs checkout where the POT file isn't present or up to date), as the contents of that field are used to fill PO-Revision-Date when the PO file is created with the --no-translator option.
The attached patch implements the default use-case of SOURCE_DATE_EPOCH, but it doesn't signal any error from the variable (allowed by the spec) and simply continues with the usual local timestamp process. Its value is used to fill POT-Creation-Date for POT files generated by xgettext, and PO-Revision-Date for PO files generated by msginit.
Perhaps the test suite should be extended and/or modified to use more this feature. What do you think?
|
Miguel Ángel Arruga Vivas <m1gu3l>![]() ![]() |
Depends on the following items: None found
Items that depend on this one: None found
There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 12 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2021-08-07 | egallager | Carbon-Copy | - | ![]() |
Added egallager |
2020-12-22 | m1gu3l | Summary | xgettext and msginit don't honor SOURCE_DATE_EPOCH | ![]() |
xgettext and msginit output isn't reproducible |
2020-12-18 | m1gu3l | Attached File | - | ![]() |
Added 0001-msginit-Do-not-use-POT-Creation-Date.patch, #50513 |
2020-12-15 | m1gu3l | Attached File | - | ![]() |
Added 0001-source-date-epoch-New-module.patch, #50466 |
2020-12-15 | m1gu3l | Attached File | #50465 | ![]() |
Removed |
2020-12-15 | m1gu3l | Attached File | - | ![]() |
Added v3-0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-time.patch, #50464 |
Attached File | - | ![]() |
Added 0001-source-date-epoch-New-module.patch, #50465 | ||
2020-12-12 | haible | Status | Ready For Test | ![]() |
In Progress |
2020-12-12 | m1gu3l | Status | In Progress | ![]() |
Ready For Test |
2020-12-12 | m1gu3l | Attached File | - | ![]() |
Added 0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-timesta.patch, #50444 |
2020-12-11 | haible | Summary | xgettext and msgint don't honor SOURCE_DATE_EPOCH | ![]() |
xgettext and msginit don't honor SOURCE_DATE_EPOCH |
2020-12-11 | m1gu3l | Attached File | - | ![]() |
Added 0001-xgettext-msginit-Honor-SOURCE_DATE_EPOCH-for-timesta.patch, #50438 |
I wonder if this is related to the following Battle for Wesnoth bug? https://github.com/wesnoth/wesnoth/issues/5989
A commenter seems to think some of the lack of reproducibility there is due to msgmerge...