bugGNU Octave - Bugs: bug #50575, Integer overflow in whos when...

 
 

bug #50575: Integer overflow in whos when calculating total elements in sparse matrices

Submitter:  None
Submitted:  Sat 18 Mar 2017 12:22:04 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  3 - Low Item Group:  Unexpected Error or Warning
Status:  Confirmed Assigned to:  None
Originator Name:  Mark Yagnatinsky Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 23 Aug 2018 05:38:43 PM UTC, comment #6: 

Matlab 2014a:


>> [a, maxint] = computer()
a =
GLNXA64
maxint =
   2.8147e+14

>> A = sprand(1e13, 1e7, 1e-15);

>> nnz(A)
ans =
      100000

>> size(A)
ans =
        1e+13        1e+07

>> numel(A)
Warning: Number of elements exceeds maximum flint 2^53-1.
The result may be inaccurate.
ans =
        1e+20

>> whos
  Name                   Size                     Bytes  Class     Attributes

  A         10000000000000x10000000            81600008  double    sparse


(Notice they warn in numel when > flintmax.  This seems like a good idea, especially with 64-bit indexing: I will file different issue.)

Note, they can linear index into such a large matrix, the condition is just that that linear index itself cannot be larger than maxint:


>> A(10,10)
ans =
   All zero sparse: 1-by-1

>> A(100)
ans =
   All zero sparse: 1-by-1

>> A(1234567)
ans =
   All zero sparse: 1-by-1

>> i = 282*int64(1000)^4 + 123
i =
      282000000000123

>> A(i)
Maximum variable size allowed by the program is exceeded.


>> # note i is just slightly larger than maxint
>> double(i)
ans =
     2.82e+14
>> maxint
maxint =
   2.8147e+14


Colin Macdonald <cbm>
Thu 23 Aug 2018 09:02:04 AM UTC, comment #5: 

Confirmed with Octave 4.4.1 on Win-10.
Reopen and retitle (for sparse matrices)

Avinoam Kalma <avinoam>
Group Member
Wed 22 Aug 2018 06:10:37 PM UTC, comment #4: 

A reasonable argument can be made for why such a sparse matrix cannot be linearly indexed, e.g., A(10) as opposed to A(1,1) b/c A(end) is longer than maxint).

But the point is, if I can create such a matrix, I should be able to call "whos" and see that I made one.

Also, note `numel(A)` returns a double from the users point of view so from her point of view, why should `prod(size(A))` work but `numel(A)` doesn't.

Colin Macdonald <cbm>
Wed 22 Aug 2018 05:59:05 PM UTC, comment #3: 

I don't think this is fixed.  Or maybe it has been fixed for dense but not sparse?

Here is the current master branch:


octave:18> [a, maxint] = computer()
a = x86_64-pc-linux-gnu
maxint = 2.1475e+09

octave:19> A = sprand(1e5, 1e5, 1e-9);

octave:20> prod(size(A))
ans = 1e+10

octave:21> A
A =
Compressed Column Sparse (rows = 100000, cols = 100000, nnz = 10 [1e-07%])

  (27568, 526) -> 0.90261
  (2571, 9637) -> 0.95277
  (74535, 10460) -> 0.72937
  (31147, 18753) -> 0.73106
  (50591, 21541) -> 0.62384
  (22220, 35691) -> 0.23058
  (87851, 39654) -> 0.0946
  (51929, 81427) -> 0.27443
  (41244, 88864) -> 0.29328
  (4317, 91034) -> 0.68548

octave:22> A(1,1)
ans = Compressed Column Sparse (rows = 1, cols = 1, nnz = 0 [0%])

octave:23> A(2571, 9637)
ans = Compressed Column Sparse (rows = 1, cols = 1, nnz = 1 [100%])

  (1, 1) -> 0.95277



octave:24> numel(A)
error: out of memory or dimension too large for Octave's index type


octave:24> whos
Variables in the current scope:

error: out of memory or dimension too large for Octave's index type
octave:24>


(That is compiled with --disable-64).

Here is a 4.4.1 from flatpak which has --enable-64:


octave:22> [a, maxint] = computer()
a = x86_64-pc-linux-gnu
maxint = 9.2234e+18
octave:23> A = sprand(1e12, 1e7, 1e-15);
octave:24> prod(size(A))
ans = 1e+19
octave:25> nnz(A)
ans = 10000
octave:26> A(1000,11234)
ans = Compressed Column Sparse (rows = 1, cols = 1, nnz = 0 [0%])



octave:27> numel(A)
error: out of memory or dimension too large for Octave's index type
octave:27> whos
Variables in the current scope:

error: out of memory or dimension too large for Octave's index type
octave:27>


Colin Macdonald <cbm>
Sat 18 Mar 2017 05:28:35 PM UTC, comment #2: 

This has been a problem in the past, but has already been fixed on the development branch (which will eventually become release 4.4).

To access the fix you will need to build from the development sources available in a Mercurial repository, or wait for the 4.4 release.  The development version is actually very stable and I use it every day.

Rik <rik5>
Group administrator
Sat 18 Mar 2017 12:28:59 PM UTC, comment #1: 

Also, numel seems to have a similar issue.

Anonymous
Sat 18 Mar 2017 12:22:04 PM UTC, original submission:  

On Ubuntu 14.04, the following occurs; worth noting that
46340^2 < 2^31 < 46341^2.  Might also be worth checking if the bytes calculation is correct; I can't seem to allocate a matrix big enough to test this: magic(3e4) fails for me.

octave:1> eye(46340);whos;eye(46341);whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        ans     46340x46340                 370720  double

Total is 2147395600 elements using 370720 bytes

Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        ans     46341x46341                 370728  double

Total is 18446744071562072601 elements using 370728 bytes

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by avinoam (Posted a comment)
  • -email is unavailable- added by cbm (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-24 mtmiller Priority5 - Normal 3 - Low
        Item GroupNone Unexpected Error or Warning
        Release3.8.1 dev
    2018-08-23 avinoam StatusFixed Confirmed
        Open/ClosedClosed Open
        SummaryInteger overflow in whos when calculating total elements in matrix Integer overflow in whos when calculating total elements in sparse matrices
    2017-03-18 rik5 CategoryNone Interpreter
        StatusNone Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code