bugGNU Wget - Bugs: bug #47689, Support parsing of UTF-16 HTML...

 
 

bug #47689: Support parsing of UTF-16 HTML encoding

Submitter:  kenorb <kenorb>
Submitted:  Wed 13 Apr 2016 06:42:52 PM UTC
   
 
Category:  Localization Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Open/Closed:  Closed
Release:  1.17.1 Operating System:  None
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  2.0 Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 08 Jul 2016 07:07:26 PM UTC, comment #5: 

See "Planned release" - it is 2.0.

You have to build/install from git right now, since 2.0 hasn't been released yet. But your issue is already fixed.

First steps are
git clone git://git.savannah.gnu.org/wget/wget2.git
[Now read 'Requirements' in README.md, install that stuff]
./bootstrap
./configure
make check

src/wget2 -r http://www.free-energy-info.co.uk/

And be prepared that your HTML files are UTF-16... looks like garbage in an editor that does not support the BOM.

Tim Ruehsen <rockdaboot>
Group administrator
Fri 08 Jul 2016 06:46:06 AM UTC, comment #4: 


>The following site has UTF-16 encoding:
>http://www.free-energy-info.co.uk/
>W3C claim it's UTF-16LE, but it's not relevant.


I have just spent the last two hours trying to wget this same site. When I finally figured out that it wasn't working because of UTF-16 I googled how to get wget to support UTF-16 and found this bug.

I want to mirror this site but wget finds no urls to follow in the index.html

I was able to convert the index.html to UTF-8 but no way that I know of to easily feed that back to wget for mirroring.

Pls advise. Or fix. Thanks

HB <bluestix>
Sun 24 Apr 2016 01:30:58 PM UTC, comment #3: 

I just made a commit to Wget2 to support parsing of UTF-16 encoded HTML pages. Since the very general nature of the summary, I changed it.

Comments:
-k/--convert-links does not work so far, but that's another issue.

Are there any real life UTF-16 CSS files served (also another issue) ?

Tim Ruehsen <rockdaboot>
Group administrator
Thu 14 Apr 2016 07:43:41 AM UTC, comment #2: 

Downloading works, but this issue is about content parsing (recursive downloads).

The server does not state a content-encoding, but the document (index.html) contains a BOM (Byte Order Mark) that says it is UTF-16LE encoded.

What has to be done is to convert the "input-stream" into UTF-8 (that's what wget is able to work with).

Currently we assume input data usable with traditional C string functions. UTF-16 (Unicode) can't be used with traditional C string functions.

See https://html.spec.whatwg.org/multipage/syntax.html#the-input-byte-stream

@Eli After downloading, try
$ wget -d -r --local-encoding=UTF-16LE --input-file index.html --force-html --base http://www.free-energy-info.co.uk

Tim Ruehsen <rockdaboot>
Group administrator
Wed 13 Apr 2016 07:22:14 PM UTC, comment #1: 

This site does work for me with Wget 1.16.1 on MS-Windows, with the exact command you have shown.  The file index.html is downloaded and it is encoded in UTF-16LE on my disk.

So I'm unsure why it doesn't work for you.

Eli Zaretskii <eliz>
Wed 13 Apr 2016 06:42:52 PM UTC, original submission:  

The following site has UTF-16 encoding:
http://www.free-energy-info.co.uk/
W3C claim it's UTF-16LE, but it's not relevant.

By default wget doesn't recognise the source of it, because it's not following any links when using with -m or -r.

When specifying remote-encoding, it doesn't work either:

$ wget --remote-encoding=UTF-16 http://www.free-energy-info.co.uk/
This version does not have support for IRIs

The same for any format, including when specifying `--no-iri`.

What should be the fix in order that encoding of that site can be parsed by wget?

Related: http://stackoverflow.com/q/36605946/55075

kenorb <kenorb>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bluestix (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by kenorb (Submitted the item)
  •  

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-24 rockdaboot Assigned toNone rockdaboot
        Open/ClosedOpen Closed
        Release1.16.3 1.17.1
        Operating SystemMac OS None
        Planned ReleaseNone 2.0
        SummarySupport for UTF-16 encoding. Support parsing of UTF-16 HTML encoding
    2016-04-14 rockdaboot StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code