bugGNU Parallel - Bugs: bug #53885, Output from --verbose ignores...

 
 

bug #53885: Output from --verbose ignores current locale, quoting every non-ASCII character

Submitter:  Paulo Marcel Coelho Aragão <marcelpaulo>
Submitted:  Sat 12 May 2018 05:33:31 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 22 Jul 2018 10:09:00 AM UTC, comment #15: 

Yes, 20180722 fixes this issue ! Thank you so much ! I've already discarded my sloppy workaround, which should rather be called breakaround because it breaks more things than works around them !

I've looked around and can't find how to close this issue.

Paulo Marcel Coelho Aragão <marcelpaulo>
Sun 22 Jul 2018 09:31:57 AM UTC, comment #14: 

parallel --verbose echo ::: ã

is fixed in 20180722 (which is, however, alpha quality).

If it solves the issue, please close it.


Ole Tange <tange>
Group administrator
Sat 30 Jun 2018 10:11:55 PM UTC, comment #13: 

Commit 4e30138 hasn't fixed this bug. The individual bytes are still quoted, so that:

parallel --verbose echo ::: ã

will still output an invalid UTF-8 string:


paulo:~/src/parallel (master)$ parallel --verbose echo ::: ã 2>&1 | hd
00000000  65 63 68 6f 20 5c c3 5c  a3 0a c3 a3 0a           |echo \.\.....|
0000000d


Paulo Marcel Coelho Aragão <marcelpaulo>
Tue 12 Jun 2018 03:11:15 AM UTC, comment #12: 

I just had an idea: we're both right and we're both wrong ;-) We've been discussing 2 very different problems:

1. Quoting the command line to output it [myself]
2. Quoting the command line to run it [you]

(1) and (2) should be handled separately, since they have different requirements. (1) actually doesn't need any quoting at all, it will produce a string in the current locale. (2) has to take care of the particularities of each shell, hence those edge cases you presented. Currently, (1) and (2) are being done by the same set of functions, but shouldn't.

I was oversimplifying the problem, because actually (1) is a very straightforward problem. And I though you were complicating it unnecessarily, but actually (2) is a knotty and thorny problem, and has to be treated with caution.

So, actually, my workaround is not even needed. It's just a question of not using those quoting functions on the output string, and let them work on the command line that will be run.

Don't you think ?

Paulo Marcel Coelho Aragão <marcelpaulo>
Mon 11 Jun 2018 06:19:21 PM UTC, comment #11: 

You are welcome.

If you find a way of generating a list of all valid input characters for the different locales, please post it, so we can make a quoting function that deals correctly with each locale.

Ole Tange <tange>
Group administrator
Mon 11 Jun 2018 01:23:54 PM UTC, comment #10: 


> your workaround does not fix the two examples given


I stand corrected. If the workaround breaks anything, then it's not valid and should not be used. I apologize. I hadn't envisaged such edge cases.

> This problem is more serious than -v giving a command that looks wrong


Perhaps I should have explained how I hit this issue. I use parallel in almost all my geeqie plugins, to handle batches of images in parallel. geeqie aborts plugins that output text which is not UTF-8. As I converted all plugins to use parallel with -v, eventually they all broke when the image file names had UTF-8 characters. So for my use case, it wasn't just a question of the commands looking right. Initially, as a workaround, I just removed option -v, but then I felt instigated to fix it, hence my workaround.

Well, that's the beauty of open source: you can discuss issues directly with developers and eventually implement a workaround that doesn't work for all use cases but works for one's own particular use case.

Thanks for the discussion.

Paulo Marcel Coelho Aragão <marcelpaulo>
Mon 11 Jun 2018 06:22:59 AM UTC, comment #9: 

While I agree that we should be quoting by character, not by byte, your workaround does not fix the two examples given.

The zh_HK.big5hkscs tries to run a command in /tmp. This can be mitigated by forcing LC_ALL=C. This problem is more serious than -v giving a command that looks wrong, but which does the correct thing.

The csh goes into an infinitely loop. This can be mitigated by quoting \240.


Ole Tange <tange>
Group administrator
Mon 11 Jun 2018 12:33:11 AM UTC, comment #8: 

I'm not sure I understand what you mean with these 2 examples.

But I get the impression you're approaching this issue from an invalid angle: you seem to be looking at bytes, when you should be looking at characters. That's valid only for encodings that encode a character in a single byte, such as the C or ISO-8859* family locales. For the UTF* family of locales, a character may be encoded in >=1 byte.

That's the root cause of this issue: the quoting code traverses the string byte by byte, when it should do it character by character. That's the only way of being sure it will work for ALL locales.

My patch is not actually a solution but just a workaround. The quoting code should be corrected to work with characters and not with individual bytes.

Paulo Marcel Coelho Aragão <marcelpaulo>
Sun 10 Jun 2018 10:33:09 PM UTC, comment #7: 

The solution should also work for:

printf '\243`/tmp/test\243`\n' | LC_ALL=zh_HK.big5hkscs parallel -v echo

parallel --tag -q -k {} -c perl\ -e\ \'print\ pack\(\"c\*\",1..255\)\'\ \|\ parallel\ -0\ --shellquote ::: csh

This is the reason, I really would prefer generating a list of all valid input characters.

Ole Tange <tange>
Group administrator
Sun 10 Jun 2018 08:35:30 PM UTC, comment #6: 

But the fact is: right now ALL locales (with the exception of C/ASCII) are already broken, since they all use characters in the range \202-\377, which parallel quotes.

So, I don't think it's necessary to check all locales: with the exception of C/ASCII, all characters can be used for all locales, therefore, they should not be quoted.

The only characters that should be quoted are the shell metacharacters, all the other characters should be output as they are. So, coming to think of it, not even the characters in the range \002-\011 and \013-\032 should be quoted. Not only it's not necessary, but it's wrong.

Paulo Marcel Coelho Aragão <marcelpaulo>
Sun 10 Jun 2018 08:16:08 PM UTC, comment #5: 

I would love to make this change, if it does not break anything in any shell for any locale.

But how can we check that?

How can we generate all legal characters for each locale?

Ole Tange <tange>
Group administrator
Sun 13 May 2018 02:05:58 AM UTC, comment #4: 

In case you deem that my solution won't break anything elsewhere, I've attached a patch.

(file #44144)

Paulo Marcel Coelho Aragão <marcelpaulo>
Sun 13 May 2018 01:28:32 AM UTC, comment #3: 

I can see, I tested it, that removing \202-\377 from the list of characters to be quoted solves the problem:


:paulo:~/src/parallel/src$ ./parallel --verbose echo ::: ã
echo ã
ã


But then again, I don't know if this doesn't break something elsewhere. And of course, it would have to be removed from all shell_quote_* functions.

Paulo Marcel Coelho Aragão <marcelpaulo>
Sun 13 May 2018 01:20:31 AM UTC, comment #2: 

I think I managed to pinpoint where the issue is in the code (even though I don't know how to fix it :-( ):


sub shell_quote_scalar_default {
    # Quote for other shells
    my $a = $_[0];
    if(defined $a) {
        # zsh wants '=' quoted
        # Solaris sh wants ^ quoted.
        # $a =~ s/([\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\>\<\~\|\; \"\!\$\&\'\202-\377])/\\$1/g;
        # This is 1% faster than the above
        if(($a =~ s/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\'\202-\377]/\\$&/go)
           +
           # quote newline as '\n'
           ($a =~ s/[\n]/'\n'/go)) {
            # A string was replaced
            # No need to test for "" or \0
        } elsif($a eq "") {
            $a = "''";
        } elsif($a eq "\0") {
            $a = "";
        }
    }
    return $a;
}


This seems to be quoting the string bite by byte, irrespective of the current locale. It should be fixed to work character by character, according to the locale.

Paulo Marcel Coelho Aragão <marcelpaulo>
Sat 12 May 2018 05:44:08 PM UTC, comment #1: 

Somehow my original post got clipped. Here's the continuation:


paralell --verbose echo -n ::: ã 2>&1 | hd


3. You'll see that the output from echo is correct, but the command output from parallel --verbose quotes each non-ASCII byte:


+paulo:~$ parallel --verbose echo -n ::: ã 2>&1 | hd
00000000  65 63 68 6f 20 2d 6e 20  5c c3 5c a3 0a c3 a3     |echo -n \.\....|


Paulo Marcel Coelho Aragão <marcelpaulo>
Sat 12 May 2018 05:33:31 PM UTC, original submission:  

parallel --verbose ignores the current locale when it outputs the command to be executed, quoting every non-ASCII character.

I built parallel from the git repo source (latest commit 3fdd3a6) on Xubuntu 18.04. Here's its version:


+paulo:~$ parallel --version
GNU parallel 20180422


Here's my locale:


LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_BR.UTF-8
LC_NUMERIC=pt_BR.UTF-8
LC_TIME=pt_BR.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_BR.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_BR.UTF-8
LC_NAME=pt_BR.UTF-8
LC_ADDRESS=pt_BR.UTF-8
LC_TELEPHONE=pt_BR.UTF-8
LC_MEASUREMENT=pt_BR.UTF-8
LC_IDENTIFICATION=pt_BR.UTF-8
LC_ALL=


and here's the output from parallel --verbose with a UTF-8 input source:


+paulo:~$ parallel --verbose echo -n ::: ã 2>&1 | hd
00000000  65 63 68 6f 20 2d 6e 20  5c c3 5c a3 0a c3 a3     |echo -n \.\....|


As you can see, the output from echo shows that the character ã is represented in UTF-8 as C3 A3, but the command line parallel --verbose output has each of those bytes quoted: 5C C3 5C A3. parallel seems to have ignored that the current locale is UTF-8 and treated each byte individually, quoting them because they're not ASCII.

To reproduce this error:

1. Set the current locale to en_US.UTF-8:


export LC_ALL=en_US.UTF-8


2. Run parallel --verbose echo -n with an UTF-8 input source, dumping the output in hexa:


paralell --verbose echo -n ::: ã 2>&1 | hd

3. You'll see that the output from echo is correct, but the command output from parallel --verbose quotes each non-ASCII byte:

+verbatim+
+paulo:~$ parallel --verbose echo -n ::: ã 2>&1 | hd
00000000  65 63 68 6f 20 2d 6e 20  5c c3 5c a3 0a c3 a3     |echo -n \.\....|


Paulo Marcel Coelho Aragão <marcelpaulo>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44144:  parallel-01-don't-quote-utf-8-bytes.patch added by marcelpaulo (2KiB - text/x-patch - Patch: don't quote bytes that might belong to UTF-8 characters)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  • -email is unavailable- added by marcelpaulo (Submitted the item)
  • -email is unavailable- added by marcelpaulo
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-03 tange Open/ClosedOpen Closed
    2018-07-22 tange StatusNone Fixed
    2018-06-11 tange Assigned toNone tange
    2018-05-13 marcelpaulo Attached File- Added parallel-01-don't-quote-utf-8-bytes.patch, #44144
    2018-05-12 marcelpaulo Carbon-Copy- Added marcelpaulo

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code