bugPSPP - Bugs: bug #47090, Running a procudure on a dataset...

 
 

bug #47090: Running a procudure on a dataset with a large number of variables takes an unreasonably long time.

Submitted by:  John Darrington <jmd>
Submitted on:  Mon 08 Feb 2016 11:38:43 AM UTC  
 
Category: SheetSeverity: 9 - Blocker
Status: FixedAssigned to: None
Open/Closed: ClosedRelease: Before 0.10.0
Effort: 0.00

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 01 Mar 2016 08:39:22 AM UTC, comment #11:

These waiting times are not ideal. But they are much better than the 5 minutes which we had before, and no worse than what I remember from version 0.8.5

Try the "sheet" branch if you want to see it faster!

I'm therefore closing this item.

John Darrington <jmd>
Project Administrator
Tue 01 Mar 2016 08:06:27 AM UTC, comment #10:

Hi John,

I tried the master git 81f50c24cae1f7b007c3 version which includes your patch. The test below starts in about 16 seconds. I clicked around a bit and could not see immediate problems.

When i try to resize a column in the data window, then this is possible, but it takes about 1-2 seconds after the release of the button, before the change is rendered. This is only the case with 2500 variables. When i delete 2400 variables and have just 100 left, then this is gone. So I guess, the patch works.

Friedrich

Friedrich Beckmann <beckmanf>
Project Member
Tue 01 Mar 2016 05:29:33 AM UTC, comment #9:

I pushed this patch.

Friedrich, can you please check to see if it causes any ill effects for you?

John Darrington <jmd>
Project Administrator
Sun 28 Feb 2016 06:03:33 PM UTC, comment #8:

> Here is an alternative patch based on Ben's suggestion.

If that also fixes the problem, then it looks great to me. Thank you!

Ben Pfaff <blp>
Project Administrator
Sun 28 Feb 2016 07:49:34 AM UTC, comment #7:

Here is an alternative patch based on Ben's suggestion.

(file #36498)

John Darrington <jmd>
Project Administrator
Thu 25 Feb 2016 09:59:55 PM UTC, comment #6:

With Johns proposed faster rendering patch I can run the example in approx. 15 seconds. Without the patch it takes one minute for 1500 variables. I stopped the 2500 variables test after 4 minutes.

I attached the patch that John mentioned in an email.

I changed the severity to blocking with respect to the discussion in bug #47266

(file #36466)

Friedrich Beckmann <beckmanf>
Project Member
Sat 20 Feb 2016 05:15:09 PM UTC, comment #5:

This slow-down should not be a surprise, since commit 8298a4c6e326a39e204ca58d686497758333a102 deletes the optimization that allowed large numbers of columns to be efficient, which I introduced in commit 45861388123559f "pspp-sheet-view: Optimize by making column header widgets lazy."

Ben Pfaff <blp>
Project Administrator
Sun 14 Feb 2016 01:28:57 PM UTC, comment #4:

Hi John,

the mentioned commit is in the pspp-sheet area. As you already tested, it should be gone, once the efficient sheet is alive...

Friedrich

Friedrich Beckmann <beckmanf>
Project Member
Sat 13 Feb 2016 11:03:32 AM UTC, comment #3:

Using git-bisect I found that this bug appears to have crept in at
commit 8298a4c6e326a39e204ca58d686497758333a102

John Darrington <jmd>
Project Administrator
Fri 12 Feb 2016 07:25:47 PM UTC, comment #2:

Running the following syntax:

input program.
vector vec(2500).
loop #v = 1 to 2500.
compute vec(#v) = #v.
end loop.
end case.
end file.
end input program.

descriptives all.

takes:

without a gui: 0.3 seconds
with 0.8.5 gui: 11 seconds
with master HEAD gui: 3 minutes
with a special hacked up gui using the efficient-sheet I'm working on: < 0.5 seconds

I wonder what we've done since 0.8.5 to make it so slow?

John Darrington <jmd>
Project Administrator
Wed 10 Feb 2016 05:29:15 PM UTC, comment #1:

Investigation shows that executing a procedure in the GUI is an operation which is O(n^2) in the number of variables.

I didn't think this was the case :(

John Darrington <jmd>
Project Administrator
Mon 08 Feb 2016 11:38:43 AM UTC, original submission:

In the GUI, running a procedure on a dataset with 2500 variables and 200 cases takes well over 5 minutes.

John Darrington <jmd>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by blp (Posted a comment)
  • -unavailable- added by beckmanf (Posted a comment)
  • -unavailable- added by jmd (Reporter)
  • -unavailable- added by jmd (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 01 Mar 2016 08:39:22 AM UTCjmdStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Sun 28 Feb 2016 07:49:34 AM UTCjmdAttached File-=>Added 0001-Avoid-multiple-resize-callbacks-when-adding-or-remov.patch, #36498
    Fri 26 Feb 2016 05:50:16 AM UTCjmdReleaseNone=>Before 0.10.0
    Thu 25 Feb 2016 09:59:55 PM UTCbeckmanfAttached File-=>Added 0001-Johns-faster-rendering-patch.patch, #36466
      Severity5 - Average=>9 - Blocker
    Wed 10 Feb 2016 05:29:15 PM UTCjmdStatusNone=>Confirmed
      Carbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1