bugtexinfo - GNU documentation system - Bugs: bug #46018, texi2any makes inconsistent EOL...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #46018: texi2any makes inconsistent EOL style

Submitted by:  Vincent Belaïche <vincentb1>
Submitted on:  Tue 22 Sep 2015 04:41:17 AM UTC  
 
Category: NoneRelease: 
Priority: 5 - NormalSeverity: 3 - Normal
Item Group: NonePrivacy: Public
Open/Closed: ClosedAssigned to: None
Status: Wont Fix

(Jump to the original submission Jump to the original submission)

Thu 01 Oct 2015 08:30:11 AM UTC, comment #6:

As Eli said, there could be ways to handle it automatically, but we don't want the inconvenience of maintaining it, and there are other ways of handling it, like always using LF line ending or running conversion programs when this isn't the case.

Gavin D. Smith <gavin>
Project Administrator
Mon 28 Sep 2015 09:42:45 AM UTC, comment #5:

> I just realized that we have already discussed this sort of EOL conversion in relation with [bug #38795]. In [bug #38795] my problem was that the BBDB manual was made of several files, some of them had svn:eol-style set to LF and others to native becoming CRLF for MSW users, so when compiling the source with makeinfo I also had inconsistent EOL's in the output. I try to summarize the discussion that had happened below:


Thanks.

IME, the only sane way of handling this mess is always maintain the Texinfo sources in Unix-style LF-only format. (For the same reason, I always set up all my VCSes, be it SVN, bzr, Git, or CVS, to never convert the EOLs.)

We made a change in texi2any so that it produces Unix-style EOLs in Info files, because otherwise the CR characters that are typically stripped by text-mode I/O used by native Windows programs would otherwise disrupt following references by byte numbers.

Eli Zaretskii <eliz>
Project Member
Mon 28 Sep 2015 09:32:04 AM UTC, comment #4:

> So, in conclusion, be it perl or any other script (AWK...), the script can be written in a way that it is EOL style insensitive.


This goes without saying. But asking for that means putting a burden on the shoulders of package maintainers, a burden they will likely refuse to shoulder.

Eli Zaretskii <eliz>
Project Member
Sun 27 Sep 2015 06:34:25 PM UTC, comment #3:

I just realized that we have already discussed this sort of EOL conversion in relation with [bug #38795]. In [bug #38795] my problem was that the BBDB manual was made of several files, some of them had svn:eol-style set to LF and others to native becoming CRLF for MSW users, so when compiling the source with makeinfo I also had inconsistent EOL's in the output.

I try to summarize the discussion that had happened below:

  • Patrice said there was not any intention to make any EOL conversion, and if it happens, it is by accident (cf 2013-04/msg00016)
  • Karl (cf 2013-04/msg00020 said that Texinfo should not canonicalize the input EOL's, but the user has to use input files and tools with consistent EOL style, so this was a bug in BBDB manual version control, they should have had the same eol-style property
  • Eli said CRLF EOL must be supported on MSW (cf 2013-04/msg00017), my understanding that LF only was to be used was based on a misleading doc which has probably been corrected since then.

So in a nutshell, the consensus that was to use tools with EOL-style consistent with the input files. This was against my opinion that either some input EOL canonicalization has to be done by Texinfo tools, or Texinfo source code should use only LF.

Then, with this consensus which seemingly has not changed, I am wondering how it can work in the following situation:

  • I get some Texinfo source code from project A, and project A people have decided to set svn:eol-style to LF for Texinfo files, so I am supposed to compile thes files with MSYS perl, the output file will be in LF, so their svn:eol-style need to be also set to LF.
  • I get some Texinfo source code from project B, and project B people have decided to set svn:eol-style to native for Texinfo files, so I am supposed to compile thes files with native perl. The output files will be in native (CRLF on MSW), so their svn:eol-style need to be also set to LF.
  • I could also use a native perl to compile files with LF EOL's, the output will have consistent CRLF EOL's, so that will raise some problem if it is version controlled and has, like input svn:eol-style set to LF

In conclusion, if I want to work both on project A and project B, I need two texi2any wrappers, one using MSYS perl for project A, and one using native perl for project B. This is not very practical.

The only practical way, indeed, would be that project B people set the EOL style property to LF in their source control, which anyway will have no impact on Linux users. So, it would be that Texinfo source code is always LF-EOL'ed

Please note that I am not even sure that it is possible to install texi2any on MSW with a native perl, as far as the installation scripts are MSYS based. In fact, I am using a non installed texi2any with interpreting the perl code in its source directory either with a native perl or an MSYS perl. Using a non installed texi2any raises some problem for the locales ; but that is another issue.

Vincent Belaïche <vincentb1>
Sun 27 Sep 2015 03:58:06 PM UTC, comment #2:

It is true that MSYS programs open text files without doing the CRLF conversion to LF, which is not surprising as they just do like programs on systems which natively uses LF as an EOL.

Anyway all the non-MS programs internally (be it AWK, perl, or C runtimes library) use the LF as a line termination for texts, so on system like MSDos where CRLF is used instead of LF on the file system, there is a CRLF->LF conversion at text input, and LF->CRLF at text output.

However it is still possible to write scripts in a way that they are insensitive to the input EOL style.

For instance look at the following AWK experiment carried out with a MSYS gawk on a DOS console:

The same experiment on an MSYS console:

Now, if I pass RS="\r?\n|\r" to gawk in order to be insensitive to EOL style, then I get for the DOS console:

And this does not change anything for the MSYS console:

If I pass a CRLF EOL in the MSYS console I get the same result:

So, in conclusion, by setting RS="\r?\n|\r" in the BEGIN condition, or passed on the command line, I can make my AWK script EOL insensitive.

The same type of case happens to MSYS Perl.

On a DOS console with an MSYS Perl without doing anything:

On a DOS console with a native Perl without doing anything:

Now, if I set binmode(STDIN,':crlf') (cf perldoc/binmode), then I get with MSYS Perl

But this does not harm the native perl:

Conversely, with only an LF EOL, ie if I do the same experiment under an MSYS console, then I also get, whether it be a native or an MSYS perl, the same good result (no spurious trailing CR):

And the same will occur if I use CRLF under an MSYS console

It will be the same if the EOL is CR:

So, in conclusion, be it perl or any other script (AWK...), the script can be written in a way that it is EOL style insensitive. This way of writing the script is useful not only on MSW, but also if you want to process a source file with non Linux EOL style (MSDOS CRLF, or old MacOS pre-Darwin CR) with a Linux texi2any.

Then, I don't know whether this is a bug or not. I fully agree with Eli's comment, on MSW:

  • make sure that all input files are with LF endings (do ``svn propset svn:eol-style LF'' on the input files).
  • use MSYS perl, so that we get LF also at output (and do ``svn propset svn:eol-style LF'' also on the output files if they are archived).

This is my prefered way and what I did for latexrefman, however, it is would be also possible to have the native svn:eol-style, and use native perl. That would also work w/o conflicts. So maybe this is a bug...

Vincent Belaïche <vincentb1>
Tue 22 Sep 2015 06:30:55 AM UTC, comment #1:

Your Texinfo source has DOS-style CRLF EOLs. So any text produced by texi2any has Unix-style EOLs, but text copied verbatim from your source has DOS EOLs. Remove the CR characters from your Texinfo source, and the problem will likely go away.

In general, when using MSYS programs, you need to work with text files that have Unix EOLs, because MSYS programs use binary I/O which reads and writes CR characters without any conversions.

Eli Zaretskii <eliz>
Project Member
Tue 22 Sep 2015 04:41:17 AM UTC, original submission:

My texi2any is a dev version, close to latest. Here is what the --version outputs:

Now here is my svn log -l 3 for texi2any:

Now, my problem is that when making latex2e-fr.html on latexrefman project (on GNA!), I got inconsistent EOL style in the latex2e-fr.html output. Here is the latexrefman revision concerned by the problem:

So, for my latest update of latex2e-fr.html (ie r441) I had to remove manually the CR characters. latex2e-fr.texi is in UTF-8, and there are macros.

The problem appears in two places:

  • In the copying comment at the beginning of the file
  • In the Math symbols +nomarkup(href="#Math-symbols")-nomarkup node

Attached to this report is a tar zipped file with the input latex2e-fr.texi and the output faulty latex2e-fr.html

Vincent Belaïche <vincentb1>

 

Attached Files
file #34947:  latex2e-fr.tgz added by vincentb1 (179KiB - application/x-gzip-compressed)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gavin (Posted a comment)
  • -unavailable- added by eliz (Posted a comment)
  • -unavailable- added by vincentb1 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 01 Oct 2015 08:30:36 AM UTCgavinOpen/ClosedOpen=>Closed
    Thu 01 Oct 2015 08:30:11 AM UTCgavinStatusNone=>Wont Fix
    Tue 22 Sep 2015 04:41:17 AM UTCvincentb1Attached File-=>Added latex2e-fr.tgz, #34947

    Back to the top


    Powered by Savane 3.1-cleanup1