newsLibreDWG - News: Major speedup for big DWG's

Latest News
Major speedup for big DWG's posted by reini_urban, Mon 11 Jun 2018 06:15:49 PM UTC - 0 replies
Enabled r2018 support posted by reini_urban, Mon 04 Jun 2018 08:40:47 AM UTC - 0 replies
Smokers and mirrors posted by reini_urban, Tue 08 May 2018 04:13:26 PM UTC - 0 replies
New hackers' wiki posted by pitanga, Wed 13 Jan 2010 01:41:11 AM UTC - 0 replies
[Submit News]
[4 news in archive]

Major speedup for big DWG's

Item posted by Reini Urban <reini_urban> on Mon 11 Jun 2018 06:15:49 PM UTC.

Thanks to David Bender and James Michael DuPont for convincing me that we need a hash table for really big DWGs. I got a DWG example with 42MB, which needed 2m to process and then 3m to free the dwg struct. I also had to fix a couple of internal problems.

We couldn't use David Bender's hashmap which he took from Android (Apache 2 licensed), and I didn't like it too much neither. So today I sat down and wrote a good int hashmap from scratch, with several performance adjustments, because we never get a key 0 and we won't need to delete keys.
So it's extremely small and simple, using cache-friendly open addressing, and I got it right at the second attempt.

Performance with this hash table now got down to 7 seconds.
Then I also removed the unneeded dwg_free calls from some cmdline apps, because the kernel does it much better then libc malloc/free. 3 minutes for free() is longer than the slowest garbage collector I've ever seen.
So now processing this 42MB dwg needs 7s.

While I was there I also had to adjust several hard coded limits used for testing our small test files. With realistic big DWG's they failed all over. There are still some minor problems, but the majority of the DWG's can be read.
And I pass through now all new error codes, with a bitmask of all occured uncritical and critical errors. On any critical error the library aborts, on some errors it just aborts for this particular object and some uncritical errors are just skipped over. See dwg.h or the docs for the error levels.

What's left is writing proper DXF's, which is apparently not that easy. With full precision mode as in newer acad's with subclass markers and all kind of internal handles, it's very strict, and I wasn't yet able to import any such DXF into acad. Should be possible though to import these into any other app.
So now I'm thinking of adding a third level of DXF: minimal, low and full (default). minimal are entities only, and are documented by acad. low would be new, the level of DXF other applications produce, such as libdxfrw or pythoncad. This is a basic DXF. Full is what teigha and acad produce.
In the end I want full, because I want to copy over parametric constraints from and to better modelers, full 3d data (3dsolid, surface) and full rendering and BIM data (Material, Sun, Lights, ...).

Reading DXF will have to wait for the next release, as I'm stuck with writing DXF's first. This needs to be elegant and maintainable, and not such a mess as with other libraries. I want to use the same code for reading and writing DXF, JSON (e.g. GeoJSON), and XML (e.g. OpenstreetMap, FreeCAD).

No messages in Major speedup for big DWG's


Back to the top

Powered by Savane 3.3