bugGNU Octave - Bugs: bug #58701, [octave forge] (mapping) gpxread...

 
 

bug #58701: [octave forge] (mapping) gpxread time parsing error with decimal seconds

Submitter:  None
Submitted:  Sat 04 Jul 2020 11:40:32 AM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Geoff Warne Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * other
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 04 Aug 2020 01:46:22 PM UTC, comment #14: 

My pleasure.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 04 Aug 2020 12:33:54 PM UTC, comment #13: 

Yes am happy to close the bug report. Thanks for all your work.

Geoff Warne <gwarne64>
Tue 04 Aug 2020 11:43:52 AM UTC, comment #12: 

Yep you're right.
Do you think that with that last mod we can close this bug report? The original issue has been fixed.

What I couldn't test was a "real" file w/o time and ele subnodes. I just created one by regexprep'ing out the time nodes from "COURSE_19341403.gpx" (easily done in notepad++) but then you get an artificial file, not one made by GPS devices or googleearth or so. It worked fine though.

There will come a moment when I look into time subnodes for waypoints (it's on the ToDo list), but that is another issue than the subject of this bug report (which has already drifted somewhat).

Philip Nienhuis <philipnienhuis>
Group Member
Tue 04 Aug 2020 09:49:50 AM UTC, comment #11: 

Problem with "COURSE_19341403.gpx"
error on line 266 because dcol is 0. It does not get assigned correctly.

This if statement doesn't work for case of having time but not elevation.
I think the has_ele should be removed from first if statement.

            if (it && has_ele)
              if (! has_ele)
                ## No "ele" subnodes
                ptrnt = ptrnt1C;
                dcol = 3;
                ncols = 3;
              elseif (has_ele < it)
                ## ele nodes come before time nodes
                ptrnt = ptrnt1A;
                dcol = 4;
                ncols = 4;
              else
                ## time nodes come before ele nodes
                ptrnt = ptrnt1B;
                dcol = 3;
                ncols = 4;
              endif
            elseif (has_ele)
              ptrnt = ptrnt2;
              ncols = 3;
            else
              ## Just Lat and Lon nodes
              ptrnt = ptrnt3;
              ncols = 2;
            endif


After changing this the file was read correctly

Geoff Warne <gwarne64>
Mon 03 Aug 2020 07:30:20 PM UTC, comment #10: 

OK, next series of mods:
http://hg.code.sf.net/p/octave/mapping/rev/a7de743fd1b8
Along the way I fixed & adapted a few more issues, some of them just looming until now.

Please again try the newest attached gpxread.m.

(file #49621)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 03 Aug 2020 10:45:19 AM UTC, comment #9: 

Right, so dropping a regexprep makes for a little better performance. Good to know, thanks.

The warning seems to be issued because the 'trkseg' nodes lack elevation subnodes; so there's another regexp format string required plus some fixes in the subsequent array processing logic. Tonight I'll have a look at that.
My next fear is that there may be .gpx files Out There that lack both time and elevation subnodes. Just to be sure I'll try to tackle that as well.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 03 Aug 2020 08:41:14 AM UTC, comment #8: 

The new file for testing is

COURSE_19341403.gpx

Geoff Warne <gwarne64>
Mon 03 Aug 2020 08:38:22 AM UTC, comment #7: 

Have tested successfully with about 100 files and found a couple with an issue.
Here is the Warning.

warning: your pattern caused PCRE to hit its MATCH_LIMIT; trying harder now, but this will be slow
warning: called from
    gpxread at line 219 column 19

I will attach an example file.

Also looking at your fix I think the second regexprep is not needed as datenum will match with 1, 2 or 3 decimal places filled.

>> datenum('2020-08-03T18:05:29.001Z','yyyy-mm-ddTHH:MM:SS.FFFZ')

ans =  738006.75381

>> datenum('2020-08-03T18:05:29.01Z','yyyy-mm-ddTHH:MM:SS.FFFZ')

ans =  738006.75381

>> datenum('2020-08-03T18:05:29.1Z','yyyy-mm-ddTHH:MM:SS.FFFZ')

ans =  738006.75381

>> datenum('2020-08-03T18:05:29.Z','yyyy-mm-ddTHH:MM:SS.FFFZ')

ans =  738006.75381

>> datenum('2020-08-03T18:05:29Z','yyyy-mm-ddTHH:MM:SS.FFFZ')

error: ds(0): subscripts must be either integers 1 to (2^63)-1 or logicals
error: called from
    datevec>__date_str2vec__ at line 268 column 12
    datevec at line 144 column 54
    datenum at line 113 column 46


Geoff Warne <gwarne64>
Sun 02 Aug 2020 08:33:36 PM UTC, comment #6: 

Well, I had another idea, a bit simpler but at the cost of some performance. Maybe a kludge but it seems to work reliably for me:
http://hg.code.sf.net/p/octave/mapping/rev/de81ba0397f6

Please try attached new gpxread.m (with this fix) and report back.

I also had a go at reading time nodes for waypoints but that is a headache as such nodes may be missing for some waypoints.

(file #49613)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Aug 2020 03:47:54 PM UTC, comment #5: 

Yeah, mixed time format fields are going to be a big PITA.
That is because Octave's datenum function (like Matlab's) isn't smart enough to parse a date string vector with varying formats.
I have an idea on how to tackle this but I'm afraid it's going to be a kludge at first; an IMO satisfactory solution may need a little bit more time. Hopefully you'll stay with this bug report until a/the final fix is available.
Good that you uploaded a test file, thanks.

BTW I have plans to make a new mapping release soon, hopefully before Octave-6.1.0 is released (I guess within 4-6 weeks). As to gpxread.m, the current fixed version as it stands now is the bottom line so to say. Improved versions might come after a new release.


Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Aug 2020 01:56:34 AM UTC, comment #4: 

Have tested gpxread with original file I submitted and it works well. However tried with another file and it fails. The new file "Mick's Loop2.gpx" has a mix of the integer seconds and fractional seconds in the time fields.

Geoff Warne <gwarne64>
Sat 01 Aug 2020 11:11:47 AM UTC, comment #3: 

I've pushed a fix here:
http://hg.code.sf.net/p/octave/mapping/rev/cd4ece3dd87e
and attached a new gpxread.m here.
Please try it and report if it fixes your problem.

[Slightly OT]
I also had a look at uncovering Time values from Waypoint nodes (these are currently not read). ATM that isn't possible w/o incurring a severe performance penalty.
In case someone wants that functionality please open a new bug report / feature request for it.

(file #49596)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Jul 2020 05:47:09 PM UTC, comment #2: 

Thanks, I'll have a look.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jul 2020 07:47:24 PM UTC, comment #1: 

Could you please upload an example gpx file that shows the issue?
The ones I have (and that I used for development) are all accepted by gpxread.m.
If confirmed kmlread and kmzread might also be affected.

Note that I'll have very limited time for Octave the next weeks so don't expect further progress until early August. (Unless someone beats me to it.)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 04 Jul 2020 11:40:32 AM UTC, original submission:  

when gpx file contains the date with a resolution of less than a second the error occurs
eg <time>2020-07-04T09:05:51.05Z</time>

it appears the gpxread function expects the time to have integer seconds for the datenum conversion.

error: datevec: DATE not parsed correctly with given format
error: called from
    datevec at line 147 column 11
    datenum at line 113 column 46
    gpxread at line 229 column 21

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49621:  gpxread.m added by philipnienhuis (10KiB - application/octet-stream)
file #49617:  COURSE_19341403.gpx added by gwarne64 (227KiB - application/octet-stream)
file #49613:  gpxread.m added by philipnienhuis (9KiB - application/octet-stream)
file #49601:  Mick@s@Loop2.gpx added by gwarne64 (55KiB - application/octet-stream)
file #49596:  gpxread.m added by philipnienhuis (9KiB - application/octet-stream)
file #49577:  activity_5152872389.gpx added by gwarne64 (80KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by gwarne64 (Updated the item)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-04-28 siko1056 Summary[octave-forge] (mapping) gpxread time parsing error with decimal seconds [octave forge] (mapping) gpxread time parsing error with decimal seconds
    2020-08-04 philipnienhuis StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-08-03 philipnienhuis Attached File- Added gpxread.m, #49621
    2020-08-03 gwarne64 Attached File- Added COURSE_19341403.gpx, #49617
    2020-08-02 philipnienhuis Attached File- Added gpxread.m, #49613
    2020-08-02 gwarne64 Attached File- Added Mick@s@Loop2.gpx, #49601
    2020-08-01 philipnienhuis Attached File- Added gpxread.m, #49596
        StatusNeed Info Ready For Test
    2020-07-29 gwarne64 Attached File- Added activity_5152872389.gpx, #49577
    2020-07-04 philipnienhuis Carbon-Copy- Added philipnienhuis
    2020-07-04 philipnienhuis StatusNone Need Info
        Assigned toNone philipnienhuis
        Release5.2.0 other
        Operating SystemMicrosoft Windows Any
    2020-07-04 mmuetzel Carbon-CopyRemoved 102357 -
    2020-07-04 mmuetzel Summarygpsread time parsing error with decimal seconds [octave-forge] (mapping) gpxread time parsing error with decimal seconds

    Back to the top

    Powered by Savane 3.13-54b4.
    Corresponding source code