bugGNU nano - Bugs: bug #39895, [Request] show a message when file...


bug #39895: [Request] show a message when file contains bytes invalid in current encoding

Submitted by:  None
Submitted on:  Thu 29 Aug 2013 09:41:21 PM UTC  
Severity:  3 - Normal Status:  Wont Fix
Assigned to:  None Open/Closed:  Closed

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 24 Sep 2013 11:31:34 AM UTC, comment #4:

Hmm, I think very few non-technical people will be using nano. And if they do, they won't even know what utf-8 is, or iso-8859. So they will most probably think that the file contains some "crap", was corrupted, damaged, and will likely refrain from touching it.

The "workaround" that I proposed does indeed work. By default my system had only utf8 locales installed, so I could not test it. But I've created some iso-8859-1 locales now, and nano still shows those questioning diamonds. Only when also changing the encoding of the terminal and then starting nano in the 8859 locale do the proper characters show up. So... for those who work completely in an ISO-8859-* environment, nano will function fine. Those who work with mixed encodings should know what they are doing.

Sure, it would be nice if nano would show a message when the file contains invalid UTF-8 (when the locale is UTF-8). But... are you willing to provide a patch? :)

But it is a dying problem -- soon no one will be using anything else any more but UTF-8. Even the page you quoted suggests so: "The best way to deal with encoding issues [...] is to serve all your pages in UTF-8."

Benno Schulenberg <bens>
Project Administrator
Mon 23 Sep 2013 06:30:41 PM UTC, comment #3:

Additional information.

As an example, in Perl, a regular expression testing for UTF-8 may look as follows:

$field =~
[\x09\x0A\x0D\x20-\x7E] # ASCII
| [\xC2-\xDF][\x80-\xBF] # non-overlong 2-byte
| \xE0[\xA0-\xBF][\x80-\xBF] # excluding overlongs
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
| \xED[\x80-\x9F][\x80-\xBF] # excluding surrogates
| \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3
| [\xF1-\xF3][\x80-\xBF]{3} # planes 4-15
| \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16

This expression can be adapted to other programming languages. It takes care of various issues, such as illegal overlong encodings and illegal use of surrogates. It will return true if $field is UTF-8, and false otherwise.

Mon 23 Sep 2013 06:11:17 PM UTC, comment #2:

Thank you for posting a proposed fix. I suppose that will work, but I imagine a lot of non-technical people are going to run into this and say, "nano doesn't support utf8 correctly!!!!" -- when in fact, it does.
The file command can correctly identify what the text files are, but most people don't even know to run it.

file --mime old_text_file.html
old_text_file_6.html: text/x-c; charset=iso-8859-1

Whereas, when you open it with nano, it treats it as utf-8 (if your locale is set to such).

I was just thinking maybe nano could put up a small message that something is amiss. Maybe it could just be as simple as saying "check file charset, malformed utf-8" when it spits out (U+FFFD) characters.

Fri 13 Sep 2013 11:34:59 AM UTC, comment #1:

Have you tried starting nano with an ISO-8859-1 locale (instead of your apparent default UTF-8 locale)?

Nano cannot know what encoding your file uses from looking at it. The DOS-format detection is something very specific, and probably not foolproof, but good enough in practice. But to guess from the simple bytes what code was intended when they are not valid UTF-8, this is in general not possible.

Benno Schulenberg <bens>
Project Administrator
Thu 29 Aug 2013 09:41:21 PM UTC, original submission:

I have a file called stuff.js

$ file stuff.js
stuff.js: HTML document text

I have a line in stuff.js that is
var vowel = "aeiouyáëíúâ";

Opening the file with nano, the line looks like
var vowel = "aeiouy�����";

Any plans to work with iso 8859?

I can run
iconv -f iso-8859-1 -t UTF-8 stuff.js > stuff8.js

And then nano will recognize the characters correctly when I open up stuff8.js
I think a lot of people do not realize what nano is doing with files and how to handle this problem. Perhaps a message should be displayed along with the "Read lines (DOS format)" that shows the end user what type of file nano thinks it is dealing with.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by bens (Posted a comment)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-02-24 bens StatusNone => Wont Fix
        Open/ClosedOpen => Closed
    2014-06-14 bens Summary[request] show a message when file contains bytes invalid in current encoding => [Request] show a message when file contains bytes invalid in current encoding
    2014-06-13 bens Summaryiso 8859 => [request] show a message when file contains bytes invalid in current encoding

    Back to the top

    Powered by Savane 3.3