bugGNU Core Utilities - Bugs: bug #11004, Results of "sort" fail...

 
 

You are not allowed to post comments on this tracker with your current authentification level.

bug #11004: Results of "sort" fail "sort -c" check if LANG is set and memory is low

Submitted by:  James Youngman <jay>
Submitted on:  Mon Nov 15 02:27:54 2004  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Wont Fix
Privacy: PublicAssigned to: None
Open/Closed: Open

(Jump to the original submission Jump to the original submission)

Tue May 15 16:39:51 2007, comment #11:

I'm afraid the glibc fix doesn't mean that we have an obvious fix for coreutils. I guess the idea is that we should go through the coreutils source and replace all instances of "setlocale (LC_ALL, "");" with "if (! setlocale (LC_ALL, "") && errno == ENOMEM) error (exit_failure, ENOMEM, "memory exhausted");". But we have no way of telling whether setlocale reliably sets errno. The Debian glibc bug tracker entry doesn't confirm that setlocale sets errno reliably in glibc 2.5. And even if it did, I don't offhand see any way at run-time, or even at configure-time, to tell whether the errno value is reliable. So I'm afraid coreutils is still stuck.

Paul Eggert <eggert>
Project Administrator
Tue May 15 14:35:59 2007, comment #10:

glibc Bugzilla entry: http://sourceware.org/bugzilla/show_bug.cgi?id=4416

Debian glibc bug tracker entry: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=281313%3A

James Youngman <jay>
Tue May 15 13:03:59 2007, comment #9:

How was it fixed in glibc? Do you have a pointer to the fix? Are any changes needed to GNU "sort"?

Paul Eggert <eggert>
Project Administrator
Tue May 15 08:41:47 2007, comment #8:

This has now been fixed (as a bug in glibc).

James Youngman <jay>
Thu Oct 5 13:26:55 2006, comment #7:

Please note the the two invocations of "sort" are always with the same value of $LANG. The only aspect of the environment which is different between the initial and the "-c" invocation of sort is the virtual memory limit (ulimit -v).

If the two invocations of sort were with different values of LC_ALL or LANG, I would strongly agree with you.

As it is, I would still assert that if LANG or LC_ALL is set, and sort produces output which is not correctly sorted according to the rules for the selected environment, it should return a nonzero status in order to indicate to the caller that it has failed.

I think then that your suggestion regarding the value returned by setlocale() and errno is probably the right answer.

James Youngman <jay>
Wed Oct 4 17:28:28 2006, comment #6:

I looked at the bug again, and now I think I understand it
better. You're calling "sort" twice, in two different
environments. In one environment "sort' has an artificially
restricted amount of memory, so that the call to "setlocale"
fails, and "sort" behaves as if LC_ALL is equal to C. In
the other, there is no such restriction and "sort" behaves
as if LC_ALL is equal to en_GB.ISO-8859-15. The two locales
are not consistent, so "sort" sorts differently under them.

You suggested 'if sort is unable to produce an output that
would pass a "sort -c" check, I think it should exit with a
nonzero status (probably 2).' But if you invoke "sort -c"
in the same environment as "sort", it will pass the "sort
-c" check. So I don't think this is a serious bug.

One possible fix would be for "sort" to report an error
message if setlocale fails. However, this would have
negative consequences. I've been on systems that behave
that way, and whenever you set the locale to something
"wrong", you start getting diagnostics from nearly every
program that you run. The normal Unix utility tradition is
therefore to ignore setlocale failures, and silently fall
back on the POSIX locale.

A better fix might be to modify glibc setlocale to set errno
reliably (currently it doesn't), and to report an error if
we are using glibc and if the errno value indicates a memory
problem. This will require a change to glibc. Perhaps you
could tackle that task?

Paul Eggert <eggert>
Project Administrator
Wed Oct 4 09:47:25 2006, comment #5:

The bug still exists with glibc 2.3.6 and coreutils 5.97.

James Youngman <jay>
Tue Sep 19 22:21:08 2006, comment #4:

No activity for a while on this one, so I assume the bug got fixed in glibc or something.

Paul Eggert <eggert>
Project Administrator
Tue Dec 21 21:01:42 2004, comment #3:

As suggested, I have raised this witht he Debian project as a possible glibc problem. You can visit the bug report at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=286717

James Youngman <jay>
Tue Dec 21 07:15:54 2004, comment #2:

I suspect this is a problem with your GNU C library setup, not with "sort" per se. glibc attempts to mmap the locale-archive file as a performance speedup, and falls back on the locale.alias file if the locale-archive can't be mmapped (which in this case it can't, because you're low on memory). Perhaps these two sources of locale information are inconsistent on your host. Or possibly you have found a glibc bug. Most likely you can work around the bug by setting LC_ALL="C" in your environment.

Paul Eggert <eggert>
Project Administrator
Mon Nov 15 02:30:23 2004, comment #1:

I forgot to mention that this is with coreutils 5.2.1.

James Youngman <jay>
Mon Nov 15 02:27:54 2004, original submission:

I normally run with $LANG set. If I artificially restrict the amount of VM that sort can use, it will exit with return status 0, and yet the output file will fail the check performed by sort -c with the same value for $LANG.

If I unset $LANG or I don't restrict the amount of VM available, everything works fine. If sort is unable to produce an output that would pass a "sort -c" check, I think it should exit with a nonzero status (probably 2).

The problem can be reproduced with a three line test file, as shown below. In the session below, I demonstrate the problem, unset LANG to show that the problem goes away, and then set it again and show that the problem comes back.

Specifying "-S8000" seems to alter the set of system calls made (no reference to /proc/meminfo) but doesn't change the behaviour. I attach an "strace" output file of the initial sort, both a working example (without the VM limit) and a non-woring example. Comparison of the two files shows that the significant difference in behaviour occurs where the program attempts to mmap() the file /usr/share/locale/locale.alias.

james@orbital:~$ cat -vet testfile
/var/cache/apt/archives/libpopt-dev_1.7-5_i386.deb$
/var/cache/apt/archives/libpopt0_1.7-5_i386.deb$
/var/cache/apt/archives/libwrap0_7.6.dbs-6_i386.deb$
james@orbital:~$ ( ulimit -v 1800 >/dev/null ; sort < testfile ) | sort -c
sort: -:2: disorder: /var/cache/apt/archives/libpopt0_1.7-5_i386.deb
james@orbital:~$ echo $LANG
en_GB.ISO-8859-15
james@orbital:~$ uname -a
Linux orbital 2.6.8-1-k7-smp #1 SMP Thu Oct 7 03:43:05 EDT 2004 i686 GNU/Linux
james@orbital:~$ unset LANG
james@orbital:~$ ( ulimit -v 1800 >/dev/null ; sort < testfile ) | sort -c
james@orbital:~$ LANG=en_GB.ISO-8859-15; export LANG
james@orbital:~$ ( ulimit -v 1800 >/dev/null ; sort < testfile ) | sort -c
sort: -:2: disorder: /var/cache/apt/archives/libpopt0_1.7-5_i386.deb
james@orbital:~$ ( ulimit -v 1800 >/dev/null ; sort < testfile ) | cat -vet
/var/cache/apt/archives/libpopt-dev_1.7-5_i386.deb$
/var/cache/apt/archives/libpopt0_1.7-5_i386.deb$
/var/cache/apt/archives/libwrap0_7.6.dbs-6_i386.deb$

James Youngman <jay>

 

Attached Files
file #2004:  TRACE.working.nopid.gz added by jay (2kB - application/x-tar - Working example - LANG set but memory not limited)
file #2003:  TRACE.failing.nopid.gz added by jay (3kB - application/x-tar - strace log of the memory-limited sort with $LANG set)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by eggert (Posted a comment)
  • -unavailable- added by jay (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed Oct 4 17:28:28 2006eggertOpen/ClosedClosed=>Open
    Tue Sep 19 22:21:08 2006eggertStatusNone=>Wont Fix
      Open/ClosedOpen=>Closed
    Mon Nov 15 02:28:53 2004jayAttached File-=>Added TRACE.working.nopid.gz, #1871
    Mon Nov 15 02:27:54 2004jayAttached File-=>Added TRACE.failing.nopid.gz, #1870

    Back to the top


    Powered by Savane 3.1-cleanup