Dropping patch annotation; we explicitly did not go with Debian's approach; see comment #8 and comment #9.

Also adding some keywords to the summary because for build farmers this may be the single biggest change in groff 1.23.

comment #14:

> My Debian-based system is running groff 1.22.4.
>
> On the other hand, groff from Git HEAD and 1.22.4 do handle the time zone differently; this is easily demonstrated by generating a blank PostScript document.
>

> $ echo | groff | grep '%%CreationDate'
> %%CreationDate: Mon Dec 28 03:17:07 2020
> $ echo | ./build/test-groff | grep '%%CreationDate'
> %%CreationDate: Mon Dec 28 14:17:10 2020
> $ echo | SOURCE_DATE_EPOCH=1609125163 groff | grep '%%CreationDate'
> %%CreationDate: Mon Dec 28 03:12:43 2020
> $ echo | SOURCE_DATE_EPOCH=1609125163 ./build/test-groff | grep '%%CreationDate'
> %%CreationDate: Mon Dec 28 14:12:43 2020

>
> So that's the next item for research.

The above was the product of my repeated error of forgetting that Debian's groff is patched to return `gmtime()` instead of `localtime()`.

So this issue now boils down to the one posed by Eli in the first place.

Should groff implicitly incorporate the system's time zone into built artifacts?

Consensus on the mailing list appears to have roughly converged on "yes", because it is easily overcome--anyone who sets SOURCE_DATE_EPOCH can with equal ease also set TZ=UTC, whereas going the other direction (inferring and reversing a time zone offset) is immensely more complicated.

Comments?

With groff's own build now as reproducible as we can get it without support from other GNU projects, the challenge now is to reproduce the problem described in the original report.

I performed two consecutive package rebuilds of man-db 2.8.5 on my Debian-based system and cannot reproduce the problem with groff-generated PostScript files.  The only differences I observe are in temporary file names produced by GNU Autoconf tests run by ./configure.  I am attaching this artifact.

My strategy was simple:

SOURCE_DATE_EPOCH=1609123901 debuild -us -uc

Note that there is no difference in the generated PostScript files.

$ grep -r '%%CreationDate' build.*
build.new/manual/man_db.ps:%%CreationDate: Mon Dec 28 02:51:41 2020
build.old/manual/man_db.ps:%%CreationDate: Mon Dec 28 02:51:41 2020

My Debian-based system is running groff 1.22.4.

On the other hand, groff from Git HEAD and 1.22.4 do handle the time zone differently; this is easily demonstrated by generating a blank PostScript document.

$ echo | groff | grep '%%CreationDate'
%%CreationDate: Mon Dec 28 03:17:07 2020
$ echo | ./build/test-groff | grep '%%CreationDate'
%%CreationDate: Mon Dec 28 14:17:10 2020
$ echo | SOURCE_DATE_EPOCH=1609125163 groff | grep '%%CreationDate'
%%CreationDate: Mon Dec 28 03:12:43 2020
$ echo | SOURCE_DATE_EPOCH=1609125163 ./build/test-groff | grep '%%CreationDate'
%%CreationDate: Mon Dec 28 14:12:43 2020

So that's the next item for research.

comment #11:

> To get reproducible DVI files produces by TeX engines you have to set the following two environment variables:

> export SOURCE_DATE_EPOCH=...
> export FORCE_SOURCE_DATE=1

> However, to get reproducible PDFs produced by ghostscript, a lot of manual work is necessary.  See
>
> https://bugs.ghostscript.com/show_bug.cgi?id=696765
>
> for more information

Thanks, Werner!

SOURCE_DATE_EPOCH + FORCE_SOURCE_DATE fixed the groff.dvi point but not the logs generated by pdfTeX.

I've got the diff down to points A, C, and D.  These are pretty easy to eyeball.

I'm attaching a diff of two back to back builds I did today with:

SOURCE_DATE_EPOCH=1608607664 CORES=3 make-groff

...where `make-groff` is a local script I have that essentially wraps the following:

./bootstrap && \
    mkdir build && \
    cd build && \
    ../configure && \
    make -j $CORES && \
    make check && \
    make doc && \
    make distcheck

The attached diff is 542KiB.  If, however, I delete the diffs resulting from pdfroff's different choices of temporary filenames, it shrinks to 5.3KiB.

We can literally solve 99% of our reproducibility problem by making pdfroff use deterministic temporary file names.

I further find the following.

A. There are nondeterministic temporary file names in config.log.  I assume this is a problem for GNU autoconf to solve.

B. groff.dvi differs.  Of TeX DVI, it has famously been observed that its "wiki page...feels the need to specifically point out that “DVI is not a document encryption format”"[1].  I propose not to undertake research of this item until the next one is resolved.  In any event, this DVI file is produced not by groff -Tdvi but by makeinfo, so if resolving (C) below doesn't fix this as a side effect, it seems likely that this is GNU Texinfo's problem to solve.

C. Log files created by pdfTeX query the system clock and embed a timestamp.  I assume this is a problem for pdfTeX to solve.

D. The generated distribution tar archives (gzipped) differ.  As far as I can tell from a quick inspection, the gunzipped tar files differ only in file metadata, not a surprise given tar's purpose. GNU tar has a feature to support SOURCE_DATE_EPOCH[2], so I assume that this detail of archive generation is a problem for GNU automake to solve.

E. One test case log differs, appearing two places (the individual and combined test logs).  The difference is in the number of the file descriptor GNU Bash opens when using process substitution, a non-POSIX shell feature.  Ingo has encouraged me to get rid of Bashisms in the test cases, and this item of trouble is sufficient to motivate me to do so in the instant case.

Conclusions: (1) Adjusting pdfroff's temporary file naming will pay a big dividend.  (2) Taking a Bashism out of one of our test cases (which I wrote) will pay a small one.  (3) The rest are going to require effort from other projects.

[1] https://yakshav.es/the-patron-saint-of-yakshaves/
[2] https://www.gnu.org/software/tar/manual/html_section/tar_33.html

(file #50541, file #50542)

Commenting as an ordinary user:

Please do not make groff ignore TZ.  That breaks user functionality (for example \*(DT will be wrong around midnight when using -mm).

The build systems should simply run everything with TZ=UTC to make displays be the same regardless of where the build executes. 

• What would people think if the date(1) command were modified to ignore the locale?   The answer is exactly the same for groff.

---

Also, as a user, I agree that timestamps or any other possibly-privacy-sensitive info should not be be embedded in output files unless there is a functional necessity.  I realize there was not consensus to get rid of %%CreationDate in generated pdfs, but I wanted to express my opinion.

---

Thanks for listening

If we're trying to support reproducible builds--and I acknowledge that Ingo thinks we shouldn't, but I disagree, because when I started working on groff, I wanted to be able to diff two build trees before and after a change I made, and there were numerous silly differences that Colin has fixed with patches like the one I'm attaching.

(file #50208)

I think groff should not write %%CreationDate into PostScript files it creates.  Not because of reproducible builds, which i consider a useless and even detrimental feature, but because of privacy concerns.  That would also solve this so-called "problem" as a side effect.

Basically, it amounts to something like deleting these lines from src/devices/grops/ps.cpp:

  {
    fputs("%%CreationDate: ", out.get_file());
  #ifdef LONG_FOR_TIME_T
    long
  #else
    time_t
  #endif
    t = current_time();
    fputs(ctime(&t), out.get_file());
  }

I nominate this as a blocker because the groff 1.23 release should be reproducible.

> This patch should be incorporated upstream if possible. Was it ever proposed here? Is it suitable for inclusion as-is?

No and no, not as far as I know.  Colin Watson is active on the groff mailing list (and has committed to its repository), so I reckon he'd have brought it forward if he thought it was ready.

The tricky part would seem to be this:

> (Note that this changes the semantics of \n[hours] etc., so may need further work.)

If anything the above is understated; it affects the semantics of every date-related register coarser than the second, I reckon. 

There are localtime offsets smaller than one hour; Australia perversely does this, and as the great Christopher Hitchens pointed out:

"The time-zone difference between India and Pakistan, for example, is half an hour.  That's a nicely irrational and arbitrary slice out of daily life.  In Cyprus, the difference between the clocks in the Greek and Turkish sectors is an hour--but it's the only in-country north-south time change that I am aware of, and it operates on two sides of the same capital city."

The closer to New Year's Day you generate your document, the worse the situation becomes.

That includes the traditional AT&T troff registers: \n[dw], \n[dm], \n[mo], and \n[yr], plus the groff extensions \n[hours], \n[minutes], and \n[year].

(\n[yr] had a Y2K bug.)

I guess we could have a register \n[.utc] which is normally zero but is set to 1 if SOURCE_DATE_EPOCH is in the environment.

Thoughts?

Reproducible builds support was initially implemented via https://git.savannah.gnu.org/cgit/groff.git/commit/?id=e4290210f2c1db5292d97678197a986d6822d044

See discussion here:
https://lists.gnu.org/archive/html/groff/2014-08/msg00112.html
https://lists.gnu.org/archive/html/groff/2015-11/msg00013.html

However, the implementation falls short of true reproducibility. I've been attempting to reproduce Arch Linux's "man-db" package, and discovered that /usr/share/doc/man-db/man-db-manual.ps is not reproducible, with this differing content:

│ ├── usr/share/doc/man-db/man-db-manual.ps
│ │ @@ -1,10 +1,10 @@
│ │  %!PS-Adobe-3.0
│ │  %%Creator: groff version 1.22.4
│ │ -%%CreationDate: Wed Oct 23 21:52:35 2019
│ │ +%%CreationDate: Wed Oct 23 15:52:35 2019
│ │  %%DocumentNeededResources: font Times-Bold
│ │  %%+ font Times-Italic
│ │  %%+ font Times-Roman
│ │  %%+ font Symbol
│ │  %%DocumentSuppliedResources: procset grops 1.22 4
│ │  %%Pages: 26
│ │  %%PageOrder: Ascend

Note the CreationDate. I'm building the software using an exported SOURCE_DATE_EPOCH=1571860355

It is successfully built in the past, but on the other hand, it does not include a timezone, it is not in UTC time, and my timezone, -0500, is being embedded here:

$ date -R -d @1571860355
Wed, 23 Oct 2019 15:52:35 -0400

What I'd expect to see embedded:

$ TZ=UTC date -R -d @1571860355
Wed, 23 Oct 2019 19:52:35 +0000

And the person who built the initial upload of the software:

$ TZ=CET date -R -d @1571860355
Wed, 23 Oct 2019 21:52:35 +0200

In order to properly support reproducible builds, I believe the localtime should not be used, and times must be converted to UTC time.