bugGNU Octave - Bugs: bug #51948, GUI fails to start when Address...

 
 

bug #51948: GUI fails to start when Address Sanitizer is compiled in

Submitter:  Rik <rik5>
Submitted:  Tue 05 Sep 2017 10:53:14 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 14 Sep 2017 04:04:07 PM UTC, comment #3: 

I fixed the problem in this cset (http://hg.savannah.gnu.org/hgweb/octave/rev/f50ef29d874e).  This was a classic off-by-one error where the amount of memory allocated for the terminal was


lines * columns


but when the associated memory was used the calculation was


(lines + 1) * columns



Rik <rik5>
Group administrator
Tue 12 Sep 2017 05:03:14 AM UTC, comment #2: 

Thanks for the pointer.  I commented out three lines in QUnixTerminalImpl.cpp beginning at 264 and it now produces a stack trace.


    /*
    dup2 (fds, STDIN_FILENO);
    dup2 (fds, STDOUT_FILENO);
    dup2 (fds, STDERR_FILENO);
    */


The stack trace is



    #0 0x7f149575b934 in __asan_memcpy (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x8c934)
    #1 0x7f14950aaf24 in TerminalView::updateImage() libgui/qterminal/libqterminal/unix/TerminalView.cpp:973
    #2 0x7f149509407a in QUnixTerminalImpl::resizeEvent(QResizeEvent*) libgui/qterminal/libqterminal/unix/QUnixTerminalImpl.cpp:228
    #3 0x7f14902092ed in QWidget::event(QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x1a22ed)
    #4 0x7f14901c3afb in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x15cafb)
    #5 0x7f14901c9035 in QApplication::notify(QObject*, QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x162035)
    #6 0x7f148f88e2a7 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Core.so.5+0x2822a7)
    #7 0x7f1490201ca1 in QWidgetPrivate::sendPendingMoveAndResizeEvents(bool, bool) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19aca1)
    #8 0x7f1490205a42 in QWidgetPrivate::show_helper() (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19ea42)
    #9 0x7f14902059c6 in QWidgetPrivate::showChildren(bool) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19e9c6)
    #10 0x7f1490205a5e in QWidgetPrivate::show_helper() (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19ea5e)
    #11 0x7f14902059c6 in QWidgetPrivate::showChildren(bool) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19e9c6)
    #12 0x7f1490205a5e in QWidgetPrivate::show_helper() (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x19ea5e)
    #13 0x7f149020887c in QWidget::setVisible(bool) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x1a187c)
    #14 0x7f1495131936 in main_window::set_window_layout(QSettings*) libgui/src/main-window.cc:1317
    #15 0x7f1495130eb8 in main_window::read_settings() libgui/src/main-window.cc:1247
    #16 0x7f149515125d in octave::gui_application::execute() libgui/src/octave-gui.cc:232
    #17 0x401e8d in main src/main-gui.cc:104
    #18 0x7f1490f0282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #19 0x401828 in _start (/home/rik/wip/Projects_Mine/octave-dbg/src/.libs/lt-octave-gui+0x401828)

0x61900018be40 is located 0 bytes to the right of 960-byte region [0x61900018ba80,0x61900018be40)
allocated by thread T0 here:
    #0 0x7f14957686b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x7f149509c19b in ScreenWindow::getImage() libgui/qterminal/libqterminal/unix/ScreenWindow.cpp:67
    #2 0x7f14950aa53a in TerminalView::updateImage() libgui/qterminal/libqterminal/unix/TerminalView.cpp:835
    #3 0x7f149509407a in QUnixTerminalImpl::resizeEvent(QResizeEvent*) libgui/qterminal/libqterminal/unix/QUnixTerminalImpl.cpp:228
    #4 0x7f14902092ed in QWidget::event(QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x1a22ed)

SUMMARY: AddressSanitizer: heap-buffer-overflow ??:0 __asan_memcpy
Shadow bytes around the buggy address:
  0x0c3280029770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3280029780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3280029790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c32800297a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c32800297b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c32800297c0: 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa fa
  0x0c32800297d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c32800297e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c32800297f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3280029800: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c3280029810: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==23669==ABORTING





Rik <rik5>
Group administrator
Wed 06 Sep 2017 02:10:39 AM UTC, comment #1: 

I think the STDERR redirection is done in QUnixTerminalImpl.cpp (connectToPty()).

Torsten Lilge <ttl>
Group Member
Tue 05 Sep 2017 10:53:14 PM UTC, original submission:  

When Octave is compiled with --enable-address-sanitizer-flags the GUI refuses to start.  It was unclear why this was, but now I have a strong indication that it is due to a heap-buffer-overflow.

I configured the build environment like so


setenv CFLAGS "-ggdb3 -O0 -pipe"
setenv CXXFLAGS "${CFLAGS}"
setenv FFLAGS "${CFLAGS}"
./configure --prefix=/home/rik/local --enable-address-sanitizer-flags --disable-docs --disable-java --disable-hg-id -C |& tee myconfig.log


Using run-octave fails, so I cd'ed to the src directory and used


strace octave-gui.exe


results were


write(2, "    #0 0x7f5a2e3ec934 in __asan_"..., 88) = 88
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59eda51000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ee2b0000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59edb09000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59eda4e000
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59eda47000
mmap(NULL, 61440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59eda38000
mmap(NULL, 126976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59eda19000
write(2, "    #1 0x7f5a2dd3bbc0 in Termina"..., 109) = 109
write(2, "    #2 0x7f5a2dd24d16 in QUnixTe"..., 132) = 132
write(2, "    #3 0x7f5a28e9d2ed in QWidget"..., 105) = 105
write(2, "    #4 0x7f5a28e57afb in QApplic"..., 135) = 135
write(2, "    #5 0x7f5a28e5d035 in QApplic"..., 121) = 121
write(2, "    #6 0x7f5a285222a7 in QCoreAp"..., 131) = 131
write(2, "    #7 0x7f5a28e95ca1 in QWidget"..., 140) = 140
write(2, "    #8 0x7f5a28e99a42 in QWidget"..., 111) = 111
write(2, "    #9 0x7f5a28e999c6 in QWidget"..., 116) = 116
write(2, "    #10 0x7f5a28e99a5e in QWidge"..., 112) = 112
write(2, "    #11 0x7f5a28e999c6 in QWidge"..., 117) = 117
write(2, "    #12 0x7f5a28e99a5e in QWidge"..., 112) = 112
write(2, "    #13 0x7f5a28e9c87c in QWidge"..., 108) = 108
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed9e2000
mmap(NULL, 61440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed9d3000
mmap(NULL, 126976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed9b4000
mmap(NULL, 258048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed975000
munmap(0x7f59ed9b4000, 126976)          = 0
write(2, "    #14 0x7f5a2ddc15ac in main_w"..., 100) = 100
write(2, "    #15 0x7f5a2ddc0b2e in main_w"..., 86) = 86
write(2, "    #16 0x7f5a2dde2179 in octave"..., 90) = 90
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed6f5000
munmap(0x7f59ed6f5000, 45056)           = 0
munmap(0x7f59ed800000, 1003520)         = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59edaec000
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed500000
munmap(0x7f59ed600000, 1048576)         = 0
write(2, "    #17 0x401e8d in main src/mai"..., 45) = 45
write(2, "    #18 0x7f5a29b9682f in __libc"..., 86) = 86
write(2, "    #19 0x401828 in _start (/hom"..., 101) = 101
write(2, "\n", 1)                       = 1
munmap(0x7f5a021c4000, 8192)            = 0
munmap(0x7f5a068af000, 3584)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5a068af000
write(2, "\33[1m\33[32m0x61900018b940 is locat"..., 115) = 115
munmap(0x7f5a068af000, 4096)            = 0
write(2, "\33[1m\33[35mallocated by thread T0 "..., 46) = 46
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5a068af000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5a021c4000
write(2, "    #0 0x7f5a2e3f96b2 in operato"..., 104) = 104
write(2, "    #1 0x7f5a2dd2ce37 in ScreenW"..., 105) = 105
write(2, "    #2 0x7f5a2dd3b1d6 in Termina"..., 109) = 109
write(2, "    #3 0x7f5a2dd24d16 in QUnixTe"..., 132) = 132
write(2, "    #4 0x7f5a28e9d2ed in QWidget"..., 105) = 105
write(2, "\n", 1)                       = 1
munmap(0x7f5a021c4000, 8192)            = 0
munmap(0x7f5a068af000, 3584)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5a068af000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5a021c5000
write(2, "SUMMARY: AddressSanitizer: heap-"..., 67) = 67
munmap(0x7f5a021c5000, 1024)            = 0
munmap(0x7f5a068af000, 1024)            = 0
mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed9bb000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59ed9b7000
write(2, "Shadow bytes around the buggy ad"..., 4693) = 4693
munmap(0x7f59ed9b7000, 16384)           = 0
munmap(0x7f59ed9bb000, 32768)           = 0
getpid()                                = 4165
write(2, "==4165==ABORTING\n", 17)      = 17
exit_group(1)                           = ?
+++ exited with 1 +++


You can see that ASAN starts to write out on file descriptor 2 (stderr).  Later there is


write(2, "SUMMARY: AddressSanitizer: heap-"..., 67) = 67


which sure looks like it was writing "heap-buffer-overflow".  Obviously stderr must have been redirected somewhere because I don't get these on the screen.

@jwe: Is there a way to temporarily halt the redirection of stderr while debugging this?


Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by rik5
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-09-14 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2017-09-12 rik5 StatusNone Confirmed
    2017-09-05 rik5 Carbon-Copy- Added jwe
        Carbon-Copy- Added dasergatskov

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code