bugGNU gettext - Bugs: bug #22961, xgettext checks encoding where it...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #22961: xgettext checks encoding where it should not

Submitted by:  None
Submitted on:  Thu 17 Apr 2008 08:20:11 AM UTC  
 
Category: Programmer toolsSeverity: 3 - Normal
Item Group: NoneStatus: Wont Fix
Privacy: PublicAssigned to: Bruno Haible <haible>
Open/Closed: Closed

Sun 20 Apr 2008 05:26:42 AM UTC, comment #5:

> the error
> message is rather misleading: It claims that the translatable
> string contains non-ASCII characters. It took me quite some time
> to figure out that the problem was with the comment.
>
> So I would suggest to at least improve the error message.


Good point. I've now improved the error message to say - in this case -:
"Non-ASCII comment at or before FILE:LINE"

Here's the ChangeLog entry:

2008-04-20 Bruno Haible <bruno@clisp.org>

Improve error message.

  • xgettext.h (lexical_context_ty): New type.

(non_ascii_error_message): New declaration.
(from_current_source_encoding): Add lcontext argument.

  • xgettext.c (non_ascii_error_message): New function.

(from_current_source_encoding): Add lcontext argument. Use
non_ascii_error_message.
(CONVERT_STRING): Add lcontext argument.
(remember_a_message, remember_a_message_plural): Update.

  • x-csharp.c (lexical_context): New variable.

(phase2_getc): Use non_ascii_error_message.
(comment_start, comment_line_end, phase6_get): Set lexical_context.
(extract_csharp): Initialize lexical_context.

  • x-java.c (struct string_buffer): Add lcontext field.

(init_string_buffer): Add lcontext argument.
(string_buffer_flush_curr_buffer): Update from_current_source_encoding
call.
(comment_start): Set lcontext.
(phase5_get): Pass lcontext argument.

  • x-perl.c (get_here_document, phase2_getc,

extract_quotelike_pass1_utf8): Pass lcontext argument.

  • x-python.c (lexical_context): New variable.

(phase2_getc): Use non_ascii_error_message.
(comment_start, comment_line_end, phase6_get): Set lexical_context.
(struct mixed_string_buffer): Add lcontext field.
(init_mixed_string_buffer): Add lcontext argument.
(mixed_string_buffer_flush_curr_buffer): Update
from_current_source_encoding call.
(phase5_get): Set lexical_context. Pass lcontext argument.
(extract_python): Initialize lexical_context.
Reported by Mark Weyer.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu 17 Apr 2008 11:49:26 AM UTC, comment #4:

Seeing that you insist on wontfix, let me mention that the error
message is rather misleading: It claims that the translatable
string contains non-ASCII characters. It took me quite some time
to figure out that the problem was with the comment.

So I would suggest to at least improve the error message.

Mark Weyer

Anonymous
Thu 17 Apr 2008 11:30:14 AM UTC, comment #3:

> I want to run xgettext on all source files at once, or at least
> in the same manner.


If the source files are in different encodings and thus require
different xgettext options, you cannot do that. In this case, you
need to use multiple xgettext invocations: an xgettext invocation
for each subset of source files, each producing a .pot file, and
a final xgettext invocation at the end, with these intermediate
.pot files as arguments and producing the final .pot file.

> When encoding differences concern only the comments,
> they will not be noticed during normal developement, so there
> is no way to make sure, that indeed all files are ISO-8859-1.


When some files are in ISO-8859-1 and some are in UTF-8, there
is a danger that some editor will garble the comments when
saving the file. (Except if all your developers only use 'vim'
:-)). Therefore I would recommend to have all files in the
same encoding.

How to guarantee that all files are in UTF-8?
for f in $files; do iconv -f UTF-8 -t UCS-4 < $f > /dev/null; done

How to guarantee that all files are not in UTF-8?
for f in $files; do
if iconv -f ASCII -t UCS-4 < $f > /dev/null 2>/dev/null; then
:
else
if iconv -f UTF-8 -t UCS-4 < $f > /dev/null 2>/dev/null; then
echo "$f is in UTF-8!" 1>&2
fi
fi
done

> As the compiler does not care, I think xgettext also should not.


In an ideal world, yes. But as I said, doing it like this would
make the xgettext code more complicated.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu 17 Apr 2008 11:07:02 AM UTC, comment #2:

I disagree.

First, the solution you propose is not feasible: I want to run
xgettext on all source files at once, or at least in the same
manner. When encoding differences concern only the comments,
they will not be noticed during normal developement, so there
is no way to make sure, that indeed all files are ISO-8859-1.
In particular, it is quite probable, that some will start to be
unicode in the near future. As the compiler does not care, I
think xgettext also should not.

Second, the explanation you give does not match my observations.
The error message goes away by simply deleting the translatable
string. In fact, other comments do not trigger the message,
simply because there is no translatable string in the next line.
This behaviour is not very robust, and it does not match your
explanation.
I understand, that you have to handle comments right before
translatable strings differently, because they might contain
the TRANSLATORS keyword. I also agree, that xgettext should
complain about unknown encodings in case they do. But this
should not affect other comments.

Best regards,

Mark Weyer

Anonymous
Thu 17 Apr 2008 09:25:44 AM UTC, comment #1:

You are supposed to use option --from-code=ISO-8859-1 in this case.

While it can indeed be argued that xgettext should not require this
option, because the particular comment is not extracted in this
case, xgettext's implementation requires requires to know the
encoding of the entire file. The reason is that ISO C 99 specifies
that the contents of source files is first interpreted as
sequence of multibyte characters and then only subject to
syntactic analysis (like recognition of comments and identifiers),
and xgettext uses the same approach for simplicity. xgettext's
code would be more complicated if it has to mix the multibyte
handling with the syntactic analysis in a single pass.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Thu 17 Apr 2008 08:20:11 AM UTC, original submission:

The situation is as follows:
A string is preceded by a comment.
The string is ASCII-only, the comment contains non-ASCII
characters, in this case iso-latin1. The comment is not marked
to be included in the .pot file.

xgettext complains, that the string's encoding is unspecified.

This happens with the version from Debian Etch, they call it
version 0.16.1-1, so I assume that your name for this version
is 0.16.1

An example:

/* Den Eintrag RestartLastLevel abspeichern, damit er "Events" bekommt,
wenn sein Stromstatus sich �ndert */
gRestartLastLevel = new MenuEintrag(this,_("~Restart last level"),
doRestartLastLevel);

If you need the full file, it is src/blatt.cpp from the cuyo
project at savannah.nongnu.org. (CVS revision 1.50, part of
release 2.-1.1)

Now I agree that it is debatable, whether one should use
comments in ones own language, or whether all comments should
be C. But this is not a decision for xgettext to make.
(And I do not expect this to be xgettext's intention, because
it only happens for comments preceding translatable strings.)

Best regards,

Mark Weyer

Anonymous

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by haible (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 26 Nov 2016 11:54:52 PM UTChaibleCategoryNone=>Programmer tools
    Thu 17 Apr 2008 09:25:44 AM UTChaibleStatusNone=>Wont Fix
      Assigned toNone=>haible
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1