bugGNU Octave - Bugs: bug #58257, urlwrite 403 Forbidden crashes...

 
 

bug #58257: urlwrite 403 Forbidden crashes octave

Submitter:  Ben Stanley <benstanley>
Submitted:  Mon 27 Apr 2020 01:42:56 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Duplicate Assigned to:  None
Originator Name:  Ben Stanley Open/Closed:  * Closed
Release:  * 6.0.90 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 27 Apr 2020 07:54:59 PM UTC, comment #8: 

Thanks. This has already been reported and discussed at bug #56828. Closing this bug report as a duplicate.

Mike Miller <mtmiller>
Group Member
Mon 27 Apr 2020 07:37:52 PM UTC, comment #7: 

It looks like urlwrite can't be interrupted.
strace keeps on spitting


poll([{fd=7, events=POLLIN}, {fd=10, events=POLLIN}, {fd=14, events=POLLIN}, {fd=16, events=POLLIN}, {fd=19, events=POLLIN}], 5, -1) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\1\0\0\0\0\0\0\0", 16)         = 8
recvmsg(10, {msg_namelen=0}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(10, {msg_namelen=0}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=7, events=POLLIN}, {fd=10, events=POLLIN}, {fd=14, events=POLLIN}, {fd=16, events=POLLIN}, {fd=19, events=POLLIN}], 5, -1) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\1\0\0\0\0\0\0\0", 16)         = 8
recvmsg(10, {msg_namelen=0}, 0)         = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(10, {msg_namelen=0}, 0)         = -1 EAGAIN (Resource temporarily unavailable)


until I kill the octave process.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 27 Apr 2020 07:29:28 PM UTC, comment #6: 

I run it through strace
(strace octave 2> urlerr.log)

The last 1000 lines of this log is attached.

Dmitri.




(file #48959)

Dmitri A. Sergatskov <dasergatskov>
Mon 27 Apr 2020 07:17:12 PM UTC, comment #5: 

Oh, I missed that this is reporting a hang, not a crash. If that's the case, then maybe the server is simply never returning anything, is that possible?

So to clear things up a bit, is the bug here that Octave does something wrong in response to a 403 error, or is the bug that urlread can't be interrupted while it waits for a response? Useful example might be http://httpbin.org/delay/10

Mike Miller <mtmiller>
Group Member
Mon 27 Apr 2020 07:12:04 PM UTC, comment #4: 

FWIW httpbin works for me, but nasdaq hangs as in OP report
(with dev aee0f20c8029 tip @ version):


octave:1> [f, suc, msg] = urlwrite ('http://httpbin.org/status/403', '/tmp/out')
f =
suc = 0
msg = HTTP response code said error
octave:2> [f, suc, msg] = urlwrite ('https://www.nasdaq.com/api/v1/historical/amzn/stocks/2010-04-27/2020-04-27', '/tmp/HistoricalQuotes.csv')


(this is on Centos 8)

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Mon 27 Apr 2020 06:35:22 PM UTC, comment #3: 

As a side note, the httpbin service is a more reliable provider of http responses, status codes, etc. This also works correctly for me, no crash:


>> [f, suc, msg] = urlwrite ('http://httpbin.org/status/403', '/tmp/out')
f =
suc = 0
msg = HTTP response code said error


Mike Miller <mtmiller>
Group Member
Mon 27 Apr 2020 06:32:22 PM UTC, comment #2: 

Hmm, I can't confirm this on my system. I get the following from urlwrite


>> [f, suc, msg] = urlwrite ('https://www.nasdaq.com/api/v1/historical/amzn/stocks/2010-04-27/2020-04-27', '/tmp/HistoricalQuotes.csv')
f =
suc = 0
msg = HTTP response code said error


No crash for me. Is this reproducible?

As to workaround, the supported way to set the User-Agent header in Octave version 6 is to use webread with weboptions. The equivalent to urlwrite is websave, but that function isn't yet implemented in Octave.

Mike Miller <mtmiller>
Group Member
Mon 27 Apr 2020 03:31:39 PM UTC, comment #1: 

Confirmed.

For the moment, I would work around this as you suggest.

Rik <rik5>
Group administrator
Mon 27 Apr 2020 01:42:56 PM UTC, original submission:  

The following command crashes octave 6.0.1:

[f, success, message ] = urlwrite( 'https://www.nasdaq.com/api/v1/historical/amzn/stocks/2010-04-27/2020-04-27', '/tmp/HistoricalQuotes.csv' )

Expected result:
/tmp/HistoricalQuotes.csv contains data downloaded from nasdaq.com.

Actual result:
Octave creates an empty file /tmp/HistoricalQuotes.csv
Octave never returns to the command prompt >>
CTRL-C at the command prompt has no effect
Attempting to close the octave gui window has no effect.
It is necessary to kill and re-start octave to regain control.

Please note that I am able to download many other URLs successfully with urlwrite. This particular nasdaq URL is special. It returns error 403 Forbidden unless you set the UserAgent to empty. The following commands successfully download the content:

wget --output-document=HistoricalQuotes.csv --user-agent="" https://www.nasdaq.com/api/v1/historical/amzn/stocks/2010-04-27/2020-04-27

curl --output HistoricalQuotes.csv --user-agent "" https://www.nasdaq.com/api/v1/historical/amzn/stocks/2010-04-27/2020-04-27

Removing the --user-agent option from these commands causes them to fail as well, but at least they don't lock up.

For the moment I plan to work around this problem by using the system command to run curl or wget as shown above. I suspect that allowing urlwrite to set the UserAgent would also solve the problem, but I did not find a way of doing that.

I tested octave built from source, downloaded with mercurial.
I built the latest commit on the stable branch.

Octave successfully built.  Now choose from the following:

   ./run-octave    - to run in place to test before installing
   make check      - to run the tests
   make install    - to install (PREFIX=/usr/local)

   HG ID for this build is "50d83252f867"

make[2]: Leaving directory '/home/ben/octave-src/octave-7/.build'

After installing octave, I started it with the command
octave --gui


Ben Stanley <benstanley>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48959:  urlerr.log.tail.gz added by dasergatskov (4KiB - application/gzip)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by benstanley (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-27 mtmiller Severity4 - Important 3 - Normal
        StatusConfirmed Duplicate
        Open/ClosedOpen Closed
        Dependencies- Depends on bugs #56828
    2020-04-27 dasergatskov Attached File- Added urlerr.log.tail.gz, #48959
    2020-04-27 mtmiller Severity3 - Normal 4 - Important
    2020-04-27 rik5 Carbon-CopyRemoved 72865 -
    2020-04-27 rik5 StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code