bugGNU libunistring - Bugs: bug #54453, abort in u8_possible_linebreaks in...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #54453: abort in u8_possible_linebreaks in libunistring 0.9.10

Submitter:  Natanael Copa <ncopa>
Submitted:  Mon 06 Aug 2018 10:48:26 AM UTC
   
 
Category:  unilbrk Severity:  3 - Normal
Item Group:  None Status:  Fix Released
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Jump to the original submission

Mon 03 Jan 2022 06:04:51 PM UTC, comment #8: 

I think the root cause of the problem is that Alpine packages get built with deficient versions of 'sed', 'join' and other POSIX utilities.

Here's my complete analysis:

1) I built, on a glibc system, the packages that you mention: gettext 0.19.8.1, against libunistring 0.9.10.

$ ./configure --prefix=/tmp/inst --with-libunistring-prefix=/gnu-inst-libunistring/0.9.10
$ make
$ make install


2) The symbols in the libunistring.so.2.1.0 that I built and the one in https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/libunistring-0.9.10-r1.apk are different:


$ nm --dynamic /tmp/inst/lib/libgettextlib.so | grep lbrk
0000000000035280 T unilbrk_is_all_ascii
0000000000035230 T unilbrk_is_utf8_encoding
0000000000057b80 R unilbrkprop
00000000000578a0 R unilbrk_table
$ nm --dynamic /gnu-inst-libunistring/0.9.10/lib/libunistring.so.2 | grep lbrk
0000000000022b60 T libunistring_unilbrk_is_all_ascii
0000000000022b20 T libunistring_unilbrk_is_utf8_encoding
00000000000bff60 R libunistring_unilbrkprop
00000000000bfbc0 R libunistring_unilbrk_table


The Ubuntu binary packages have these symbols:

$ nm --dynamic /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0 | grep lbrk
0000000000021f50 T libunistring_unilbrk_is_all_ascii
0000000000021f00 T libunistring_unilbrk_is_utf8_encoding
00000000000c0260 R libunistring_unilbrkprop
00000000000bfec0 R libunistring_unilbrk_table
$ nm --dynamic /usr/lib/i386-linux-gnu/libunistring.so.2.1.0 | grep lbrk
0001ea10 T libunistring_unilbrk_is_all_ascii
0001e9c0 T libunistring_unilbrk_is_utf8_encoding
000bc0a0 R libunistring_unilbrkprop
000bbd00 R libunistring_unilbrk_table


With the Alpine apk, however, the symbols are different:

$ nm --dynamic usr/lib/libunistring.so.2.1.0 | grep lbrk
000000000001ecde T unilbrk_is_all_ascii
000000000001eca6 T unilbrk_is_utf8_encoding
00000000000b4180 R unilbrkprop
00000000000b3de0 R unilbrk_table


3) A program like msgmerge is linked against these libraries, in order:

msgmerge - libgettextsrc.so - libgettextlib.so - libunistring.so

This means, by the usual ELF rules, symbols in libgettextsrc or libgettextlib will override symbols in libunistring.

The invocation chain, with binary location, of your example is:
On Alpine Linux:

main                                                 - msgmerge
write-po.c                                           - libgettextsrc.so
ulc_width_linebreaks                                 - libunistring.so
u8_width_linebreaks                                  - libunistring.so
u8_possible_linebreaks                               - libunistring.so
unilbrkprop, unilbrk_table                           - libgettextlib.so


Whereas on Ubuntu and other distros it is:

main                                                 - msgmerge
write-po.c                                           - libgettextsrc.so
ulc_width_linebreaks                                 - libunistring.so
u8_width_linebreaks                                  - libunistring.so
u8_possible_linebreaks                               - libunistring.so
libunistring_unilbrkprop, libunistring_unilbrk_table - libunistring.so


4) The libunistring/lib/Makefile.am contains logic to make the libunistring library namespace-clean, that is, to avoid collisions with symbols that may possibly occur in executables and other libraries. This is done by prefixing all internal symbols of the library with 'libunistring_'. unilbrkprop, unilbrk_table are such symbols. This is what, e.g. on Ubuntu, avoids a collision between 'unilbrk_table' (in libgettextlib) and 'libunistring_unilbrk_table' (in libunistring).

This explains why the crash is seen on Alpine Linux but not on other OSes.

5) From the lack of 'libunistring_' prefix in the Alpine binaries one can infer that the logic in libunistring/lib/Makefile.am did not work right. That is, the libunistring.sym file that it composed was incorrect.

In the mean time, there have been two improvements to this logic, to work around deficient versions of 'join' and 'sed' on Alpine Linux.

For 'join', the problem is described in https://lists.gnu.org/archive/html/bug-gnulib/2021-04/msg00041.html , and the workaround is in https://git.savannah.gnu.org/gitweb/?p=libunistring.git;a=commitdiff;h=ca45f9feb81f5c3eb8aeecb4902e3a828c216767 .

For 'sed', one of the problems is described in http://lists.busybox.net/pipermail/busybox/2022-January/089400.html , and the workaround is in https://git.savannah.gnu.org/gitweb/?p=libunistring.git;a=commitdiff;h=facdea659083095a6ec8bcc7aa322f683898739b .

This explains why the symbol list was wrong on Alpine Linux and correct everywhere else.

6) The build recipe in Alpine looks reasonable. https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/libunistring/APKBUILD . There's no apparent bug here.

In summary, it costs me time (two workarounds already, and this bug report here), to deal with the deficient utilities in Alpine Linux. Things would be simpler if Alpine Linux packages would be built with GNU coreutils and GNU sed in $PATH.

Bruno Haible <haible>
Group administrator
Thu 09 Aug 2018 07:21:58 PM UTC, comment #7: 

Turns out to be a bug in gettext which bundles an old copy of libunistring. The shared, system, libunistring code is used but ends up use the data table from the bundled, older libunistring and things goes wrong.

(Thanks to Timo Teräs for helping me find that out).


Natanael Copa <ncopa>
Thu 09 Aug 2018 03:10:06 PM UTC, comment #6: 

The binaries comes from official Alpine Linux x86_64 builder:
http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/

I build them locally with debugging symbols to be able to provide backtraces.

Alpine Linux edge(rolling release) x86_64.

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/6.4.0/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-6.4.0/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl --target=x86_64-alpine-linux-musl --with-pkgversion='Alpine 6.4.0' --enable-checking=release --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers --enable-__cxa_atexit --enable-default-pie --enable-cloog-backend --enable-languages=c,c++,objc,java,fortran,ada --disable-libssp --disable-libmpx --disable-libmudflap --disable-libsanitizer --enable-shared --enable-threads --enable-tls --with-system-zlib --with-linker-hash-style=gnu
Thread model: posix
gcc version 6.4.0 (Alpine 6.4.0)


musl libc 1.1.19 + some backported patches.

Same toolchain for both 0.9.9 and 0.9.10. It only happens with 0.9.10.

What is somewhat weird is that a couple of times when I have built libunistring with -g, the problem no longer manifests itself. When it does, the gettext have exactly 21 failing test in testsuite.

It is quite possible that this is a memory corruption or undefined behavior somewhere else, in either gettext, libunistring or gnulib, but I have no clue where the problem may be.

Natanael Copa <ncopa>
Thu 09 Aug 2018 02:45:42 PM UTC, comment #5: 

The gettext testsuite failures that you posted in https://lists.gnu.org/archive/html/bug-gettext/2018-08/msg00008.html also point to a broken u8_possible_linebreaks function.

However, the source code of u8_possible_linebreaks and related functions has not changed between libunistring 0.9.9 and 0.9.10. Therefore, compiling the two libunistring version with the same GCC and the same compiler options should produce the same result.

May I ask: Where did you get these two binary packages from? And with which GCC versions were they compiled, respectively?

Bruno Haible <haible>
Group administrator
Thu 09 Aug 2018 09:01:36 AM UTC, comment #4: 

all tests in gettext-0.19.8.1 testsuite also passes with libunistring-0.9.9.

Natanael Copa <ncopa>
Thu 09 Aug 2018 08:52:28 AM UTC, comment #3: 

Interestingly, gettext-0.19.8.1's testsuite has 21 failing tests when its built with libunistring-0.9.10, which all passed when built with libunistring-0.9.7.

Natanael Copa <ncopa>
Thu 09 Aug 2018 08:39:18 AM UTC, comment #2: 

Another backtrace, with lt_TO.po:

Core was generated by `/usr/bin/msgmerge lt_LT.po.new filezilla.pot -o lt_LT.po.new~'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007ff7b9738f1b in __syscall4 (a4=<optimized out>, a3=<optimized out>,
    a2=<optimized out>, a1=<optimized out>, n=<optimized out>)
    at ./arch/x86_64/syscall_arch.h:38
38      ./arch/x86_64/syscall_arch.h: No such file or directory.
[Current thread is 1 (LWP 30519)]
(gdb) bt
#0  0x00007ff7b9738f1b in __syscall4 (a4=<optimized out>, a3=<optimized out>,
    a2=<optimized out>, a1=<optimized out>, n=<optimized out>)
    at ./arch/x86_64/syscall_arch.h:38
#1  __restore_sigs (set=set@entry=0x7ffcda4e3530) at src/signal/block.c:43
#2  0x00007ff7b9739035 in raise (sig=sig@entry=6) at src/signal/raise.c:13
#3  0x00007ff7b9710fb4 in abort () at src/exit/abort.c:9
#4  0x00007ff7b879b294 in u8_possible_linebreaks (
    s=0x55ff98668083 "ymel_s\271\367\177",
    s@entry=0x55ff98668080 "_ymel_s\271\367\177", n=n@entry=11,
    encoding=encoding@entry=0x7ff7b94e3c17 "UTF-8",
    p=0x55ff986231a3 "\001\001\001\001\001\001\001\001\271\367\177",
    p@entry=0x55ff986231a0 '\001' <repeats 11 times>, "\271\367\177")
    at unilbrk/u8-possible-linebreaks.c:161
#5  0x00007ff7b879b320 in u8_width_linebreaks (
    s=s@entry=0x55ff98668080 "_ymel_s\271\367\177", n=n@entry=11,
    width=width@entry=77, start_column=start_column@entry=7,
    at_end_columns=at_end_columns@entry=0, o=o@entry=0x55ff98643640 "",
    encoding=0x7ff7b94e3c17 "UTF-8",
    p=0x55ff986231a0 '\001' <repeats 11 times>, "\271\367\177")
    at unilbrk/u8-width-linebreaks.c:46
#6  0x00007ff7b879b83e in ulc_width_linebreaks (
    s=0x55ff98668080 "_ymel_s\271\367\177", n=11, width=77,

Natanael Copa <ncopa>
Mon 06 Aug 2018 12:15:45 PM UTC, comment #1: 

Can you please attach the files he_IL.po.new and filezilla.pot that are mentioned in the command-line? Without these files, I have little chance of reproducing the issue.

Bruno Haible <haible>
Group administrator
Mon 06 Aug 2018 10:48:26 AM UTC, original submission:  

Hi,

After upgrading libunistring 0.9.10 on Alpine Linux, build of filezilla ends with an abort() when calling /usr/bin/msgmerge (from gettext).

coredump indicate that it happens in libunistring. Downgrading libunistring solves it.

Backtrace from core dump:


````
Core was generated by `/usr/bin/msgmerge he_IL.po.new filezilla.pot -o he_IL.po.new~'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f13cefb4f1b in __syscall4 (a4=<optimized out>, a3=<optimized out>, a2=<optimized out>,
    a1=<optimized out>, n=<optimized out>) at ./arch/x86_64/syscall_arch.h:38
38      ./arch/x86_64/syscall_arch.h: No such file or directory.
[Current thread is 1 (LWP 12586)]
(gdb) bt
#0  0x00007f13cefb4f1b in __syscall4 (a4=<optimized out>, a3=<optimized out>, a2=<optimized out>,
    a1=<optimized out>, n=<optimized out>) at ./arch/x86_64/syscall_arch.h:38
#1  __restore_sigs (set=set@entry=0x7ffecef34020) at src/signal/block.c:43
#2  0x00007f13cefb5035 in raise (sig=sig@entry=6) at src/signal/raise.c:13
#3  0x00007f13cef8cfb4 in abort () at src/exit/abort.c:9
#4  0x00007f13ce017294 in u8_possible_linebreaks (s=0x55bf9c4f8f2b "___\001",
    s@entry=0x55bf9c4f8f20 "%H:%M:%S __\001", n=n@entry=17, encoding=encoding@entry=0x7f13ced5fc17 "UTF-8",
    p=0x55bf9c5313eb "\001\001\001\001\001\001src/interface/QueueView.cpp",
    p@entry=0x55bf9c5313e0 "\001\002\001\001\002\001\001\002\003\001\001\001\001\001\001\001\001src/interface/QueueView.cpp") at unilbrk/u8-possible-linebreaks.c:161
#5  0x00007f13ce017320 in u8_width_linebreaks (s=s@entry=0x55bf9c4f8f20 "%H:%M:%S __\001", n=n@entry=17,
    width=width@entry=77, start_column=start_column@entry=7, at_end_columns=at_end_columns@entry=0,
    o=o@entry=0x55bf9c539f20 "", encoding=0x7f13ced5fc17 "UTF-8",
    p=0x55bf9c5313e0 "\001\002\001\001\002\001\001\002\003\001\001\001\001\001\001\001\001src/interface/QueueView.cpp") at unilbrk/u8-width-linebreaks.c:46
#6  0x00007f13ce01783e in ulc_width_linebreaks (s=0x55bf9c4f8f20 "%H:%M:%S __\001", n=17, width=77,
    start_column=7, at_end_columns=0, o=0x55bf9c539f20 "", encoding=0x7f13ced5fc17 "UTF-8",
    p=0x55bf9c5313e0 "\001\002\001\001\002\001\001\002\003\001\001\001\001\001\001\001\001src/interface/QueueView.cpp") at unilbrk/ulc-width-linebreaks.c:120
#7  0x00007f13ced42cdb in ?? () from /usr/lib/libgettextsrc-0.19.8.1.so
#8  0x00007f13ced44330 in ?? () from /usr/lib/libgettextsrc-0.19.8.1.so
#9  0x00007f13ced41584 in msgdomain_list_print () from /usr/lib/libgettextsrc-0.19.8.1.so
#10 0x000055bf9b265b7e in ?? ()
#11 0x00007f13cef8cad6 in __libc_start_main (main=0x55bf9b265288, argc=5, argv=0x7ffecef34668)
    at src/env/__libc_start_main.c:74
#12 0x000055bf9b265bd3 in ?? ()
#13 0x0000000000000000 in ?? ()
````

Natanael Copa <ncopa>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #44755:  test-suite.log added by ncopa (11KiB - text/x-log - gettext-tools 0.19.8.1: tests/test-suite.log)
file #44754:  libunistring-0.9.10-r0.log added by ncopa (500KiB - text/x-log - build log of libunistring)
file #44753:  lt_LT.po added by ncopa (326KiB - text/x-gettext-translation)
file #44751:  he_IL.po added by ncopa (272KiB - text/x-gettext-translation)
file #44752:  filezilla.pot added by ncopa (221KiB - application/vnd.ms-powerpoint)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by ncopa (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-04 haible StatusFixed Fix Released
    2022-01-03 haible StatusNeed Info Fixed
        Assigned toNone haible
        Open/ClosedOpen Closed
    2021-12-16 haible CategoryNone unilbrk
    2018-08-09 haible StatusNone Need Info
    2018-08-09 ncopa Attached File- Added test-suite.log, #44755
    2018-08-09 ncopa Attached File- Added libunistring-0.9.10-r0.log, #44754
    2018-08-09 ncopa Attached File- Added lt_LT.po, #44753
    2018-08-09 ncopa Attached File- Added he_IL.po, #44751
        Attached File- Added filezilla.pot, #44752

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code