mainSavannah Administration - Support: sr #105528, Deal with encoding issues in...

 
 

sr #105528: Deal with encoding issues in log_accum

Submitter:  Yavor Doganov <yavor>
Submitted:  Fri 25 Aug 2006 02:54:11 PM UTC
   
 
Category:  Source code repositories - developer access Priority:  5 - Normal
Severity:  1 - Wish Status:  Done
Privacy:  Public Assigned to:  Beuc
Operating System:  GNU/Linux Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 17 Jan 2008 06:46:36 PM UTC, comment #13: 

I took a look at the log_accum and svnmailer, and I agree that it is not trivial at all to implement auto-guessing of the encoding; also this approach is almost certainly doomed to fail (for example, the log message in ISO-8859-15, one file in UTF-8 and another one in CP1251).

AFAIU svnmailer tries some fairly complex ways to determine the encoding, including checking svn:mime-type, the files themselves, the log message etc.  It has a fallback mechanism and it's not 100% sure that the mail header will be correct.  It just happens to work in most cases.

I suggest to do the following: If there is an easy way to check the body (perhaps in `build_message_body'), try if it is only US-ASCII, then UTF-8 and finally (if it is not valid), ISO-8859-1.  Alternatively, just add this header (in the `send_mail' subroutine?) with UTF-8 hardcoded.  That's easy and won't hurt much, I guess.

Yavor Doganov <yavor>
Mon 14 Jan 2008 08:59:06 PM UTC, comment #12: 

<<<It has
Content-Type: text/plain; charset="utf-8"
and all non-ASCII characters are readable. I can review the diff of the whole translation without any trouble.>>>

I'll have a look - but I think problems arise when content was not in UTF-8 in the first place.


Though, declaring every messages as UTF-8 could be dirty work-around that actually works better in the common case. Maybe we'll do that.

Sylvain Beucler <Beuc>
Mon 14 Jan 2008 08:43:42 PM UTC, comment #11: 

"I doubt it, because I don't think there's a magic solution to this issue..."

Maybe -- I don't argue at all, but I read all those mails with Mutt (takes about ~15 minutes to load on my mailbox).  Basically, I only experienced this issue when I started the Bulgarian www.gnu.org team (and later joined webmasters) -- I receieve all non-US-ASCII messages to -email is unavailable- and -email is unavailable- as garbage, and it doesn't happen with other projects.  There's a workouround, of course (Mutt and Gnus), so it is just a matter of inconvenience.

"What makes you think svnmailer and other less-sucking VCSs handle encoding correctly?"

Well, I just received this a few mins after your message:
https://www.lagaule.org/pipermail/translators/2008-January/000201.html

It has
Content-Type: text/plain; charset="utf-8"
and all non-ASCII characters are readable.  I can review the diff of the whole translation without any trouble.

Yavor Doganov <yavor>
Mon 14 Jan 2008 08:11:46 PM UTC, comment #10: 

"It seems that svnmailer (and other modern VCS tools) have logic to autodetect the encoding for post-commit notifications."

I doubt it, because I don't think there's a magic solution to this issue, and because MUAs usually do not auto-detect encoding (Thunderbird does not, at least).

What makes you think svnmailer and other less-sucking VCSs handle encoding correctly? I should have a look.

Sylvain Beucler <Beuc>
Mon 14 Jan 2008 07:39:51 PM UTC, comment #9: 

Sylvain, I think that you can safely ignore this issue.  I checked two other projects (non-Savannah) and it looks like none of them is adding a `Content-Type' header.

It seems that svnmailer (and other modern VCS tools) have logic to autodetect the encoding for post-commit notifications.

I guess it's not worth diving into is, given the fact that only old farts like me use CVS/RCS and medieval MUAs that can't autodetect encoding in the absence of this header.

Sorry for disturbing.

Yavor Doganov <yavor>
Mon 14 Jan 2008 06:15:00 PM UTC, comment #8: 


> Yavor asked for progress on this one.


As far as I'm concerned I didn't progress. We would welcome more information about how CVS deals with commit messages:
- does it add any encoding metadata? (e.g. the locale in use at commit time, if cvs is used from the command line)
- is the encoding considered ASCII?
- how well do the encoding "auto-guess" libraries can perform on such small messages?
- how practical would it be to add a charset marker at the beginning of each commit message?

As for the content of the message, this is probably more difficult - first the diff needs to be independant, for example an attachment. Then the encoding is almost certainly unknown to CVS, because any kind of text file can be versioned independently of its charset. Probably only an encoding "auto-guess" method would work.

If anyone can research about this, we use log_accum and commit_prep from the contrib/ directory in project CVS.
http://cvs.sv.gnu.org/viewvc/ccvs/contrib/?root=cvs

Sylvain Beucler <Beuc>
Thu 18 Jan 2007 10:54:38 PM UTC, comment #7: 

This is not an easy problem to tackle with, so we can't fix it very soon.

I had it in the tasks list:
https://savannah.gnu.org/maintenance/CvS

Sylvain Beucler <Beuc>
Tue 03 Oct 2006 06:40:00 AM UTC, comment #6: 
Yavor Doganov <yavor>
Tue 03 Oct 2006 06:35:35 AM UTC, comment #5: 

I'll have a deeper look at the issue.
Can you give me a direct link to a list archive where this issue happens?

Sylvain Beucler <Beuc>
Tue 03 Oct 2006 06:31:17 AM UTC, comment #4: 

If this is a general setting that affects all projects, I completely agree.

But if it's possible to make this change only for "www-bg", I kindly ask you to do so.  Our diffs and commit messages are always in UTF-8, so people that are using mailers that are not UTF-8-capable are doomed anyway.  My request was for those who use MUAs that cannot change the encoding of the body, if it has declared (obviously wrong) encoding in the Content-Type header.

Yavor Doganov <yavor>
Mon 02 Oct 2006 07:31:08 PM UTC, comment #3: 

Some MUAs don't support UTF-8 either so I have as much chances to break things than to fix them. Therefore I'm not willing to work on this :)

Sylvain Beucler <Beuc>
Mon 02 Oct 2006 03:49:01 PM UTC, comment #2: 

Unfortunately it seems that the messages are coming with "Content-Type: text/plain; charset=iso-8859-1" header which make them unreadable.  Some prominent MUAs like Mutt and Gnus can handle this and show them with a proper encoding, but others don't.

Could you please change the encoding to "utf-8", if it's not so hard?

Yavor Doganov <yavor>
Sat 26 Aug 2006 08:58:42 AM UTC, comment #1: 

Done. :)

Steven Robson <stevenr>
Fri 25 Aug 2006 02:54:11 PM UTC, original submission:  

Dear Savannah hackers,

We need the commits to the Sources repository of the Bulgarian Translation Team for gnu.org to be sent as notifications by e-mail.

Project name: www-bg
Commit logs to be sent to: -email is unavailable-
Diffs should be included
Directories to monitor: everything

Please make the above for the Sources repository only.
Thanks in advance and happy hacking.

Yavor Doganov <yavor>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by yavor (Posted a comment)
  • -email is unavailable- added by Beuc (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-01-18 Beuc Open/ClosedOpen Closed
    2006-10-03 Beuc Assigned tostevenr Beuc
        Open/ClosedClosed Open
        SummaryPlease enable commit logs for the `www-bg' project Deal with encoding issues in log_accum
    2006-08-26 stevenr StatusNone Done
        Assigned toNone stevenr
        Open/ClosedOpen Closed
    2006-08-25 yavor Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code