bugGNU Core Utilities - Bugs: bug #11004, Results of "sort" fail...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #11004: Results of "sort" fail "sort -c" check if LANG is set and memory is low

Submitter:  James Youngman <jay>
Submitted:  Mon 15 Nov 2004 02:27:54 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Wont Fix
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Tue 15 May 2007 04:39:51 PM UTC, comment #11: 

I'm afraid the glibc fix doesn't mean that we have an obvious fix for coreutils.  I guess the idea is that we should go through the coreutils source and replace all instances of "setlocale (LC_ALL, "");" with "if (! setlocale (LC_ALL, "") && errno == ENOMEM) error (exit_failure, ENOMEM, "memory exhausted");".  But we have no way of telling whether setlocale reliably sets errno.  The Debian glibc bug tracker entry doesn't confirm that setlocale sets errno reliably in glibc 2.5.  And even if it did, I don't offhand see any way at run-time, or even at configure-time, to tell whether the errno value is reliable.  So I'm afraid coreutils is still stuck.

Paul Eggert <eggert>
Group administrator
Tue 15 May 2007 02:35:59 PM UTC, comment #10: 
James Youngman <jay>
Tue 15 May 2007 01:03:59 PM UTC, comment #9: 

How was it fixed in glibc?  Do you have a pointer to the fix?  Are any changes needed to GNU "sort"?

Paul Eggert <eggert>
Group administrator
Tue 15 May 2007 08:41:47 AM UTC, comment #8: 

This has now been fixed (as a bug in glibc).

James Youngman <jay>
Thu 05 Oct 2006 01:26:55 PM UTC, comment #7: 

Please note the the two invocations of "sort" are always with the same value of $LANG.   The only aspect of the environment which is different between the initial and the "-c" invocation of sort is the virtual memory limit (ulimit -v).  

If the two invocations of sort were with different values of LC_ALL or LANG, I would strongly agree with you. 

As it is, I would still assert that if LANG or LC_ALL is set, and sort produces output which is not correctly sorted according to the rules for the selected environment, it should return a nonzero status in order to indicate to the caller that it has failed.

I think then that your suggestion regarding the value returned by setlocale() and errno is probably the right answer.  

James Youngman <jay>
Wed 04 Oct 2006 05:28:28 PM UTC, comment #6: 

I looked at the bug again, and now I think I understand it
better.  You're calling "sort" twice, in two different
environments.  In one environment "sort' has an artificially
restricted amount of memory, so that the call to "setlocale"
fails, and "sort" behaves as if LC_ALL is equal to C.  In
the other, there is no such restriction and "sort" behaves
as if LC_ALL is equal to en_GB.ISO-8859-15.  The two locales
are not consistent, so "sort" sorts differently under them.

You suggested 'if sort is unable to produce an output that
would pass a "sort -c" check, I think it should exit with a
nonzero status (probably 2).'  But if you invoke "sort -c"
in the same environment as "sort", it will pass the "sort
-c" check.  So I don't think this is a serious bug.

One possible fix would be for "sort" to report an error
message if setlocale fails.  However, this would have
negative consequences.  I've been on systems that behave
that way, and whenever you set the locale to something
"wrong", you start getting diagnostics from nearly every
program that you run.  The normal Unix utility tradition is
therefore to ignore setlocale failures, and silently fall
back on the POSIX locale.

A better fix might be to modify glibc setlocale to set errno
reliably (currently it doesn't), and to report an error if
we are using glibc and if the errno value indicates a memory
problem.  This will require a change to glibc.  Perhaps you
could tackle that task?

Paul Eggert <eggert>
Group administrator
Wed 04 Oct 2006 09:47:25 AM UTC, comment #5: 

The bug still exists with glibc 2.3.6 and coreutils 5.97.

James Youngman <jay>
Tue 19 Sep 2006 10:21:08 PM UTC, comment #4: 

No activity for a while on this one, so I assume the bug got fixed in glibc or something.

Paul Eggert <eggert>
Group administrator
Tue 21 Dec 2004 09:01:42 PM UTC, comment #3: 

As suggested, I have raised this witht he Debian project as a possible glibc problem.  You can visit the bug report at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=286717

James Youngman <jay>
Tue 21 Dec 2004 07:15:54 AM UTC, comment #2: 

I suspect this is a problem with your GNU C library setup, not with "sort" per se.  glibc attempts to mmap the locale-archive file as a performance speedup, and falls back on the locale.alias file if the locale-archive can't be mmapped (which in this case it can't, because you're low on memory).  Perhaps these two sources of locale information are inconsistent on your host.  Or possibly you have found a glibc bug. Most likely you can work around the bug by setting LC_ALL="C" in your environment.

Paul Eggert <eggert>
Group administrator
Mon 15 Nov 2004 02:30:23 AM UTC, comment #1: 

I forgot to mention that this is with coreutils 5.2.1.

James Youngman <jay>
Mon 15 Nov 2004 02:27:54 AM UTC, original submission:  

I normally run with $LANG set.   If I artificially restrict the amount of VM that sort can use, it will exit with return status 0, and yet the output file will fail the check performed by sort -c with the same value for $LANG. 

If I unset $LANG or I don't restrict the amount of VM available, everything works fine.  If sort is unable to produce an output that would pass a "sort -c" check, I think it should exit with a nonzero status (probably 2).

The problem can be reproduced with a three line test file, as shown below.    In the session below, I demonstrate the problem, unset LANG to show that the problem goes away, and then set it again and show that the problem comes back.

Specifying "-S8000" seems to alter the set of system calls made (no reference to /proc/meminfo) but doesn't change the behaviour.  I attach an "strace" output file of the initial sort, both a working example (without the VM limit) and a non-woring example.  Comparison of the two files shows that the significant difference in behaviour occurs where the program attempts to mmap() the file /usr/share/locale/locale.alias.



james@orbital:~$ cat -vet  testfile
/var/cache/apt/archives/libpopt-dev_1.7-5_i386.deb$
/var/cache/apt/archives/libpopt0_1.7-5_i386.deb$
/var/cache/apt/archives/libwrap0_7.6.dbs-6_i386.deb$
james@orbital:~$ ( ulimit  -v 1800 >/dev/null ; sort < testfile ) | sort -c
sort: -:2: disorder: /var/cache/apt/archives/libpopt0_1.7-5_i386.deb
james@orbital:~$ echo $LANG
en_GB.ISO-8859-15
james@orbital:~$ uname -a
Linux orbital 2.6.8-1-k7-smp #1 SMP Thu Oct 7 03:43:05 EDT 2004 i686 GNU/Linux
james@orbital:~$ unset LANG
james@orbital:~$ ( ulimit  -v 1800 >/dev/null ; sort < testfile ) | sort -c
james@orbital:~$ LANG=en_GB.ISO-8859-15; export LANG
james@orbital:~$ ( ulimit  -v 1800 >/dev/null ; sort < testfile ) | sort -c
sort: -:2: disorder: /var/cache/apt/archives/libpopt0_1.7-5_i386.deb
james@orbital:~$ ( ulimit  -v 1800 >/dev/null ; sort < testfile ) | cat -vet
/var/cache/apt/archives/libpopt-dev_1.7-5_i386.deb$
/var/cache/apt/archives/libpopt0_1.7-5_i386.deb$
/var/cache/apt/archives/libwrap0_7.6.dbs-6_i386.deb$

James Youngman <jay>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #2004:  TRACE.working.nopid.gz added by jay (2KiB - application/x-tar - Working example - LANG set but memory not limited)
file #2003:  TRACE.failing.nopid.gz added by jay (3KiB - application/x-tar - strace log of the memory-limited sort with $LANG set)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by eggert (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2006-10-04 eggert Open/ClosedClosed Open
    2006-09-19 eggert StatusNone Wont Fix
        Open/ClosedOpen Closed
    2004-11-15 jay Attached File- Added TRACE.working.nopid.gz, #1871
    2004-11-15 jay Attached File- Added TRACE.failing.nopid.gz, #1870

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code