bugGNU Parallel - Bugs: bug #54321, parallel sets locale to POSIX...

 
 

bug #54321: parallel sets locale to POSIX which breaks programs that need a UTF-8 locale

Submitter:  Paulo Marcel Coelho Aragão <marcelpaulo>
Submitted:  Tue 17 Jul 2018 03:41:40 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  herrbiber
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 25 Aug 2018 10:21:43 PM UTC, comment #5: 

Indeed, parallel is not changing the locale anymore!

I'm running the latest master:


paulo:~/tmp$ parallel --version
GNU parallel 20180822


I created foo:

#!/bin/bash
locale

First checked the current locale:


paulo:~/tmp$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_BR.UTF-8
LC_NUMERIC=pt_BR.UTF-8
LC_TIME=pt_BR.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_BR.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_BR.UTF-8
LC_NAME=pt_BR.UTF-8
LC_ADDRESS=pt_BR.UTF-8
LC_TELEPHONE=pt_BR.UTF-8
LC_MEASUREMENT=pt_BR.UTF-8
LC_IDENTIFICATION=pt_BR.UTF-8
LC_ALL=


Then ran foo under parallel:


paulo:~/tmp$ parallel ./foo ::: 1
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_BR.UTF-8
LC_NUMERIC=pt_BR.UTF-8
LC_TIME=pt_BR.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_BR.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_BR.UTF-8
LC_NAME=pt_BR.UTF-8
LC_ADDRESS=pt_BR.UTF-8
LC_TELEPHONE=pt_BR.UTF-8
LC_MEASUREMENT=pt_BR.UTF-8
LC_IDENTIFICATION=pt_BR.UTF-8
LC_ALL=


I can see there was a lot of work on 75007fc to make this happen. parallel is now locale-foolproof. This is really great, thanks a lot for that!

Paulo Marcel Coelho Aragão <marcelpaulo>
Sat 25 Aug 2018 01:32:45 PM UTC, comment #4: 

Please test version 20180822.

Ole Tange <tange>
Group administrator
Tue 17 Jul 2018 10:13:49 AM UTC, comment #3: 

I can confirm that this commit introduced the bug. I reverted it and the issue is gone.

Paulo Marcel Coelho Aragão <marcelpaulo>
Tue 17 Jul 2018 04:08:23 AM UTC, comment #2: 

This issue seems to be caused by this commit.

Paulo Marcel Coelho Aragão <marcelpaulo>
Tue 17 Jul 2018 03:44:36 AM UTC, comment #1: 

This bug report complains of the same issue.

Paulo Marcel Coelho Aragão <marcelpaulo>
Tue 17 Jul 2018 03:41:40 AM UTC, original submission:  

I use cuetag (from cuetools) to set ID3 tags in mp3 files. cuetag reads a CUE file and calls mid3v2 (from mutagen) to set the ID3 tags in the mp3 files. I call cuetag from a bash script, in a function that splits an audio file, sets the ID3 tags of the splitted files and then rename them. This script calls parallel to run this function over all audio files that need splitting.

I started to notice that cuetag failed to set the ID3 tags whenever the CUE file had UTF-8-encoded characters, complaining that its ASCII decoder couldn't recognize such characters. However, when I ran cuetag with the same CUE file, but from a bash interactive session, it managed to set all tags.

I took a really long time to discover what was happening. parallel sets the locale to C, and this is propagated to all programs it runs. So my parallel-run bash function is working under locale C, and also cuetag, and more importantly, mid3v2. As mid3v2 thinks it's working under locale C, it tries to decode all strings it receives from cuetag as ASCII strings, and can't recognize UTF-8-encoded characters.

I read parallel's man page all over, many times, and couldn't find any mention of it setting the locale to C. IMHO, parallel shouldn't change the environment in any way, in particular the locale, lest it may break any programs it runs, as it happened with cuetag/mid3v2. It will break ANY program that depends on the current locale to work properly. Or at least there should be a command-line option to dictate which locale parallel will use for the programs.

Paulo Marcel Coelho Aragão <marcelpaulo>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  • -email is unavailable- added by marcelpaulo (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-08-27 tange StatusNone Fixed
        Assigned toNone herrbiber
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code