Date: Mon, 17 Mar 2003 21:37:19 +0200 From: Tuomas Lukka To: gzz developers list Subject: [Gzz] Benchmark experiments: what to do and what not Ok, some things to point out to Matti and others about how to design benchmark experiments to find out bottlenecks, focused on OpenGL rendering. 1) Create a *SINGLE VOBSCENE* that shows the problem 2) Render it thousands of times and measure the time, in timeRender(). This is the only way to get really meaningful results. The reasons are: - There's a LOT of noise in computers, due to other processes, other threads, cache, ... -> need to repeat a lot of times to average out - We don't want to measure vobscene generation + rendering Measuring some composite like that is ridiculous. You must find out *SEPARATELY* the time to generate it and the time to render it. NEVER measure both and assume that delays occur in one of them. -> generate the vobscene and then time the thousands of renders -> measure the time to generate 1000 times the vobscene (without any rendering) - We want a meaningful measure -> don't measure an animation from one place to another The scene changes inbetween and this means that the noise mentioned above will play a greater role. Also, any change to the rendering code will change the actual frames rendered and invalidate comparisons to the earlier times. Define *ONE* *SINGLE* *SCENE* that is slower to either generate or render than it should be. After this, vary parameters in the one scene and little by little, isolate what the bottleneck in it is. Then, make a simple benchmark that has just the bottleneck in one scene. Again, using timeRender(). Once we have this, we can really start working on speeding up the code. Anything less and we'll be shooting around blindly. Tuomas Small example by Matti(not tested, though) :: from org.nongnu.libvob import AbstractUpdateManager # Draw n letter vobs and benchmark it by pressing Ctrl-b class SmallScene: def __init__(self): self.count = 0 def scene(self, vs): bgcolor = (0.6, 0.7, 0.8) putnoc(vs, background(bgcolor)) alph = 'abcdefghijklmnopqrstuvwxyz' r = java.util.Random() size = vs.getSize() for i in range(0, self.count): cs = vs.translateCS(0, 'TEXT'+str(count), r.nextFloat()*size.width, r.nextFloat()*size.height) putText(vs, cs, alph[(i%len(alph)], color=(0,0,0), h=14, y=10) def benchmark(self): for self.count in range(0,801,100): vs = w.createVobScene() self.scene(vs) # if need to see what happens AbstractUpdateManager.chg() print self.count, ' letter vobs, ',\ 'time: ', w.timeRender(vs, 1000) def key(self, k): if k == "Ctrl-B": self.benchmark() currentScene = SmallScene()