helpSavannah Administration - Support: sr #105528, Deal with encoding issues in...

 
 

sr #105528: Deal with encoding issues in log_accum

Submitted by:  Yavor Doganov <yavor>
Submitted on:  Fri 25 Aug 2006 02:54:11 PM UTC  
 
Category: Source code repositories - developer accessPriority: 5 - Normal
Severity: 1 - WishStatus: Done
Assigned to: Sylvain Beucler <Beuc>Operating System: GNU/Linux
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 17 Jan 2008 06:46:36 PM UTC, comment #13:

I took a look at the log_accum and svnmailer, and I agree that it is not trivial at all to implement auto-guessing of the encoding; also this approach is almost certainly doomed to fail (for example, the log message in ISO-8859-15, one file in UTF-8 and another one in CP1251).

AFAIU svnmailer tries some fairly complex ways to determine the encoding, including checking svn:mime-type, the files themselves, the log message etc. It has a fallback mechanism and it's not 100% sure that the mail header will be correct. It just happens to work in most cases.

I suggest to do the following: If there is an easy way to check the body (perhaps in `build_message_body'), try if it is only US-ASCII, then UTF-8 and finally (if it is not valid), ISO-8859-1. Alternatively, just add this header (in the `send_mail' subroutine?) with UTF-8 hardcoded. That's easy and won't hurt much, I guess.

Yavor Doganov <yavor>
Site Administrator
Mon 14 Jan 2008 08:59:06 PM UTC, comment #12:

<<<It has
Content-Type: text/plain; charset="utf-8"
and all non-ASCII characters are readable. I can review the diff of the whole translation without any trouble.>>>

I'll have a look - but I think problems arise when content was not in UTF-8 in the first place.

Though, declaring every messages as UTF-8 could be dirty work-around that actually works better in the common case. Maybe we'll do that.

Sylvain Beucler <Beuc>In charge of this item.
Mon 14 Jan 2008 08:43:42 PM UTC, comment #11:

"I doubt it, because I don't think there's a magic solution to this issue..."

Maybe -- I don't argue at all, but I read all those mails with Mutt (takes about ~15 minutes to load on my mailbox). Basically, I only experienced this issue when I started the Bulgarian http://www.gnu.org team (and later joined webmasters) -- I receieve all non-US-ASCII messages to -unavailable- and -unavailable- as garbage, and it doesn't happen with other projects. There's a workouround, of course (Mutt and Gnus), so it is just a matter of inconvenience.

"What makes you think svnmailer and other less-sucking VCSs handle encoding correctly?"

Well, I just received this a few mins after your message:
https://www.lagaule.org/pipermail/translators/2008-January/000201.html

It has
Content-Type: text/plain; charset="utf-8"
and all non-ASCII characters are readable. I can review the diff of the whole translation without any trouble.

Yavor Doganov <yavor>
Site Administrator
Mon 14 Jan 2008 08:11:46 PM UTC, comment #10:

"It seems that svnmailer (and other modern VCS tools) have logic to autodetect the encoding for post-commit notifications."

I doubt it, because I don't think there's a magic solution to this issue, and because MUAs usually do not auto-detect encoding (Thunderbird does not, at least).

What makes you think svnmailer and other less-sucking VCSs handle encoding correctly? I should have a look.

Sylvain Beucler <Beuc>In charge of this item.
Mon 14 Jan 2008 07:39:51 PM UTC, comment #9:

Sylvain, I think that you can safely ignore this issue. I checked two other projects (non-Savannah) and it looks like none of them is adding a `Content-Type' header.

It seems that svnmailer (and other modern VCS tools) have logic to autodetect the encoding for post-commit notifications.

I guess it's not worth diving into is, given the fact that only old farts like me use CVS/RCS and medieval MUAs that can't autodetect encoding in the absence of this header.

Sorry for disturbing.

Yavor Doganov <yavor>
Site Administrator
Mon 14 Jan 2008 06:15:00 PM UTC, comment #8:

> Yavor asked for progress on this one.


As far as I'm concerned I didn't progress. We would welcome more information about how CVS deals with commit messages:
- does it add any encoding metadata? (e.g. the locale in use at commit time, if cvs is used from the command line)
- is the encoding considered ASCII?
- how well do the encoding "auto-guess" libraries can perform on such small messages?
- how practical would it be to add a charset marker at the beginning of each commit message?

As for the content of the message, this is probably more difficult - first the diff needs to be independant, for example an attachment. Then the encoding is almost certainly unknown to CVS, because any kind of text file can be versioned independently of its charset. Probably only an encoding "auto-guess" method would work.

If anyone can research about this, we use log_accum and commit_prep from the contrib/ directory in project CVS.
http://cvs.sv.gnu.org/viewvc/ccvs/contrib/?root=cvs

Sylvain Beucler <Beuc>In charge of this item.
Thu 18 Jan 2007 10:54:38 PM UTC, comment #7:

This is not an easy problem to tackle with, so we can't fix it very soon.

I had it in the tasks list:
https://savannah.gnu.org/maintenance/CvS

Sylvain Beucler <Beuc>In charge of this item.
Tue 03 Oct 2006 06:40:00 AM UTC, comment #6:

Thank you very much. Here's a link: http://zver.fsa-bg.org/pipermail/dict-notifications/2006-October/000338.html

Yavor Doganov <yavor>
Site Administrator
Tue 03 Oct 2006 06:35:35 AM UTC, comment #5:

I'll have a deeper look at the issue.
Can you give me a direct link to a list archive where this issue happens?

Sylvain Beucler <Beuc>In charge of this item.
Tue 03 Oct 2006 06:31:17 AM UTC, comment #4:

If this is a general setting that affects all projects, I completely agree.

But if it's possible to make this change only for "www-bg", I kindly ask you to do so. Our diffs and commit messages are always in UTF-8, so people that are using mailers that are not UTF-8-capable are doomed anyway. My request was for those who use MUAs that cannot change the encoding of the body, if it has declared (obviously wrong) encoding in the Content-Type header.

Yavor Doganov <yavor>
Site Administrator
Mon 02 Oct 2006 07:31:08 PM UTC, comment #3:

Some MUAs don't support UTF-8 either so I have as much chances to break things than to fix them. Therefore I'm not willing to work on this :)

Sylvain Beucler <Beuc>In charge of this item.
Mon 02 Oct 2006 03:49:01 PM UTC, comment #2:

Unfortunately it seems that the messages are coming with "Content-Type: text/plain; charset=iso-8859-1" header which make them unreadable. Some prominent MUAs like Mutt and Gnus can handle this and show them with a proper encoding, but others don't.

Could you please change the encoding to "utf-8", if it's not so hard?

Yavor Doganov <yavor>
Site Administrator
Sat 26 Aug 2006 08:58:42 AM UTC, comment #1:

Done. :)

Steven Robson <stevenr>
Fri 25 Aug 2006 02:54:11 PM UTC, original submission:

Dear Savannah hackers,

We need the commits to the Sources repository of the Bulgarian Translation Team for gnu.org to be sent as notifications by e-mail.

Project name: www-bg
Commit logs to be sent to: -unavailable-
Diffs should be included
Directories to monitor: everything

Please make the above for the Sources repository only.
Thanks in advance and happy hacking.

Yavor Doganov <yavor>
Site Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by yavor (Posted a comment)
  • -unavailable- added by Beuc (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 18 Jan 2007 10:54:38 PM UTCBeucOpen/ClosedOpen=>Closed
    Tue 03 Oct 2006 06:35:35 AM UTCBeucAssigned tostevenr=>Beuc
      Open/ClosedClosed=>Open
      SummaryPlease enable commit logs for the `www-bg' project=>Deal with encoding issues in log_accum
    Sat 26 Aug 2006 08:58:42 AM UTCstevenrStatusNone=>Done
      Assigned toNone=>stevenr
      Open/ClosedOpen=>Closed
    Fri 25 Aug 2006 02:54:11 PM UTCyavorCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup