bugGNU Octave - Bugs: bug #58854, [octave forge] (mapping) kmlread...

 
 

bug #58854: [octave forge] (mapping) kmlread does not handle id tags when trying to parse placemark text

Submitter:  Geoff Warne <gwarne64>
Submitted:  Wed 29 Jul 2020 11:51:19 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Geoff Warne Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 06 Aug 2020 02:15:07 AM UTC, comment #16: 

your welcome

Geoff Warne <gwarne64>
Wed 05 Aug 2020 09:05:37 PM UTC, comment #15: 

OK, I pushed that fix to the repo crediting you (hope you don't mind):
http://hg.code.sf.net/p/octave/mapping/rev/2ecda99293bd

BTW, about the time formats, I've entered a patch for datevec.m to incude the datetime formats in kmlread. I've seen those formats elsewhere too (a.o., .ods spreadheet files); see patch #9964.
If accepted I think the datetime string polishing trick of comment #13 can be dropped within say 4-5 years, when Octave 7 (to be released hopefully this year or early next year) gets outdated.

As you write that all your files are read fine I'll close this bug report. Many thanks for your cooperation.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 05 Aug 2020 11:55:30 AM UTC, comment #14: 

Found another issue reading linestrings.
The coordinates seem to be produced in 2 ways.
1 separated by spaces
2 separated by linefeeds

have changed line 97 to
lines = strsplit (strtrim (coords), {" ","\n"});

an example file with linefeeds is "greaterhobarttrails-hobart-hobart-to-the-pinnacle.kml"

After this fix all my kml's passed.

Geoff Warne <gwarne64>
Tue 04 Aug 2020 06:15:09 PM UTC, comment #13: 

Couldn't resist having a look (turns out my daughter prepares dinner tonight rather than me so I have a brief spell)

As I wrote over in bug #58701 (gpxread), comment #5:

> mixed time format fields are going to be a big PITA. That is because Octave's datenum function (like Matlab's) isn't smart enough to parse a date string vector with varying formats.

... no matter how valid Google declares it to be for kml files :-(

Of course, in theory preparing for different date/time formats in a vector should be possible but in practice it's going to be a complicated affair having fragile results at best.
Luckily Google standardized the formats a bit. So checking the lengths using

cellfun ("numel", times);


...followed by filling in missing month/day/hour/minute/second strings with zeros for each string < 20 chars could be a way out.  However time zone corrections (the last time format in comment #9) would screw up royally.

So inserting:

      ltime = cellfun ("numel", times);
      if (any (ltime < 20))
        ## Year resolution
        times(ltime == 4) = ...
         cellfun (@(x) [x "-00-00T00:00:00Z"], times(ltime == 4), "uni", 0);
        ## Month resolution
        times(ltime == 7) = ...
         cellfun (@(x) [x "-00T00:00:00Z"], times(ltime == 7), "uni", 0);
        ## Day resolution
        times(ltime == 10) = ...
         cellfun (@(x) [x "T00:00:00Z"], times(ltime == 10), "uni", 0);
      endif


after L.194 solves reading 2020-50km-Course.kml time fields (I just verified).

As to the missing elevation in COURSE_20667693.kml, that fix was just needed for Points and involved an easy check on nr. of coordinates.

OK, fixes pushed here:
http://hg.code.sf.net/p/octave/mapping/rev/5325fa8506f7
(I took the liberty of crediting you for your assistance)
and kmlread.m attached, please try, hopefully it works well enough.


(file #49628)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 04 Aug 2020 01:43:02 PM UTC, comment #12: 

That Nan thing is a very old bug :-) thx forspotting that. I'll fix it.

<altitudeMode>clampToGround</altitudeMode> doesn't quite rule out elevation data. The elevations are behind the Lat/Lon coordinates, see e.g., first Placemark in ParraWirra.kml:

:
          <Placemark>
            <TimeSpan>
              <begin>2010-01-01T00:00:00Z</begin>
              <end>2010-01-01T00:00:00Z</end>
            </TimeSpan>
            <Style>
              <IconStyle>
              <scale>0.9</scale>
              <Icon>
                <href>http://connect.garmin.com/image/main/icons/kml/grn-play.png</href>
              </Icon>
            </IconStyle>
            </Style>
            <Point>
              <altitudeMode>clampToGround</altitudeMode>
              <coordinates>138.83341203443706,-34.710941025987267,346</coordinates>
            </Point>
          </Placemark>
:


The last ",346" in the coordinates subnode is the elevation. I think the "clamptoground" option usually does need an elevation as the processing application undoubtedly needs to know the ground level elevation to clamp to in the first place, doesn't it.
But yes in COURSE_20667693.kml they're lacking. I guess to cater for that requires testing for nr. of commas (field separators) in the coordinates subnode and based on that perform the reshape and Z assignment.
Needs to be done elsewhere in kmlread.m as well, I'm afraid. Could be a bit of work.

As I said in comment #6 I'll put this aside for the next period; if you can suggest fixes or upload patches I'll happily review them.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 04 Aug 2020 10:35:15 AM UTC, comment #11: 

line 109 'Nan' undefined.
obvious typo should be NaN

Geoff Warne <gwarne64>
Tue 04 Aug 2020 10:31:41 AM UTC, comment #10: 

Another file I have has points without elevation data. It uses the <altitudeMode>clampToGround</altitudeMode> method.

This fails on line 72 as xyz doesn't have a 3rd entry for elevation.

attached file is "COURSE_20667693.kml"

Geoff Warne <gwarne64>
Tue 04 Aug 2020 10:20:23 AM UTC, comment #9: 

Found a problem with a kml file that had a mixed time format in the <when> </when> field.

It happened that when the time ticked over midnight there was one entry with only the date, no time.

I will attach the file its called "2020-50km-Course.kml"

This grab from a KML Reference indicates that it is valid.



Elements Specific to TimeStamp
<when>
Specifies a single moment in time. The value is a dateTime, which can be one of the following:
dateTime gives second resolution
date gives day resolution
gYearMonth gives month resolution
gYear gives year resolution
The following examples show different resolutions for the <when> value:

gYear (YYYY)
<TimeStamp>
  <when>1997</when>
</TimeStamp>

gYearMonth (YYYY-MM)
<TimeStamp>
  <when>1997-07</when>
</TimeStamp>

date (YYYY-MM-DD)
<TimeStamp>
  <when>1997-07-16</when>
</TimeStamp>

dateTime (YYYY-MM-DDThh:mm:ssZ)
Here, T is the separator between the calendar and the hourly notation of time, and Z indicates UTC. (Seconds are required.)
<TimeStamp>
  <when>1997-07-16T07:30:15Z</when>
</TimeStamp>

dateTime (YYYY-MM-DDThh:mm:sszzzzzz)
This example gives the local time and then the ± conversion to UTC.
<TimeStamp>
  <when>1997-07-16T10:30:15+03:00</when>
</TimeStamp>

Geoff Warne <gwarne64>
Mon 03 Aug 2020 08:30:33 PM UTC, comment #8: 

Better take this kmlread.m (forgot to move some cell2mat code into if blocks).

(file #49623)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 03 Aug 2020 08:26:24 PM UTC, comment #7: 

kmlread.m attached.

(file #49622)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 03 Aug 2020 08:25:46 PM UTC, comment #6: 

I've pushed some fixes here:
http://hg.code.sf.net/p/octave/mapping/rev/35aca628cc6b
and attached the resulting kmlread.m.
Please report if this fixes reading other kml files you have lying around.

From what I can see in ParraWirra.kml the XYZ coordinates of the individual Placemarks and those in the LineString section are identical.

Anyway, based on your findings in comment #4 I'll probably postpone further fixes. Which doesn't mean I won't work at it, or someoe else might, it's just that I have no time & priority for it in the coming months.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Aug 2020 04:07:34 PM UTC, comment #5: 

Thanks for Parrawirra.kmz, let's see what can be done.
The upload limit here is small (16 MB) and I never know if that is before or after 7-bit encoding which tends to expand files by sometimes 30 % :-)
But zipped and esp. 7-zipped or bzipped files may make it within the limits.

Looking TL;DR tru the file I see a few more things that "need attention". TimeSpan rather than Time nodes, MultiGeometry (although that just duplicates the Lon/Lat coordinates), to name a few.
.kml files (actually .xml) are read using regular expressions (XML gurus probably frown here). Now, regexps, while usually much faster than XML parsers, expect a fairly rigid file layout. For complicated and much varying file formats like .kml it is very hard to be able to catch all file layouts.
On top of that Octave has no options for XML I/O (see a.o., bug #58004).

And disclaimer: kmlread.m has been marked "experimental" for a reason.

But I'll have a look. As with gpxread.m a fix may not be in time for a next mapping package release though.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Aug 2020 03:38:21 AM UTC, comment #4: 

Have tested kmlread on a few files with id tags and name subnodes and it works well

Geoff Warne <gwarne64>
Sun 02 Aug 2020 12:40:52 AM UTC, comment #3: 

Have added file Parrawirra.kml in  replace of Hubert_100_social_training_run.kml as for some reason it didn't want to upload. It was large but I thought it was smaller that the upload limit.

Geoff Warne <gwarne64>
Sat 01 Aug 2020 12:31:02 PM UTC, comment #2: 

Thanks for this bug report and your analysis.

Rather than "Hubert_100_social_training_run.kml" you supplied "KoziClassic.kml" but that also gives some insight :-)

That KoziClassic.kml file shows that many more nodes can have an id tag; e.g., the Point, LineString and Polygon subnodes. So the regexp format strings will have to be adapted a bit more than you suggest.
I provisionally fixed the latter issue by omitting the last ">" from the regexp format strings because the id tags can have arbitrary length and consequently, that last ">" could fall of the substring selected in the regexp call anyway.

A first draft of a kmlread.m fixed for the id tags is attached, please try.

Can you still upload that Hubert_100_social_training_run.kml? than I can look into the missing "name" subnodes as well.



(file #49597)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Jul 2020 09:05:08 AM UTC, comment #1: 

Have also notices that if there isn't a <name> </name> entry the parsing will also fail. an example file is attached. Hubert_100_social_training_run.kml

Geoff Warne <gwarne64>
Wed 29 Jul 2020 11:51:19 PM UTC, original submission:  

the part of kmlread that tries to parse the placemark text does not handle the case when the placemark has an id or the following point/linestring/polygon has an id.

eg the kml syntax for placemark and point are

<Placemark>
Syntax
<Placemark id="ID">
  <!-- inherited from Feature element -->
  <name>...</name>                      <!-- string -->
  <visibility>1</visibility>            <!-- boolean -->
  <open>0</open>                        <!-- boolean -->
  <atom:author>...<atom:author>         <!-- xmlns:atom -->
  <atom:link href=" "/>                <!-- xmlns:atom -->
  <address>...</address>                <!-- string -->
  <xal:AddressDetails>...</xal:AddressDetails>  <!-- xmlns:xal -->
  <phoneNumber>...</phoneNumber>        <!-- string -->
  <Snippet maxLines="2">...</Snippet>   <!-- string -->
  <description>...</description>        <!-- string -->
  <AbstractView>...</AbstractView>      <!-- Camera or LookAt -->
  <TimePrimitive>...</TimePrimitive>
  <styleUrl>...</styleUrl>              <!-- anyURI -->
  <StyleSelector>...</StyleSelector>
  <Region>...</Region>
  <Metadata>...</Metadata>              <!-- deprecated in KML 2.2 -->
  <ExtendedData>...</ExtendedData>      <!-- new in KML 2.2 -->

  <!-- specific to Placemark element -->
  <Geometry>...</Geometry>
</Placemark>


<Point>
Syntax
<Point id="ID">
  <!-- specific to Point -->
  <extrude>0</extrude>                        <!-- boolean -->
  <altitudeMode>clampToGround</altitudeMode>
        <!-- kml:altitudeModeEnum: clampToGround, relativeToGround, or absolute -->
        <!-- or, substitute gx:altitudeMode: clampToSeaFloor, relativeToSeaFloor -->
  <coordinates>...</coordinates>              <!-- lon,lat[,alt] -->
</Point>



I have it working by changing line 58 to
 pnam = cell2mat (regexp (xml(max (1, il - 1000):il+7), ...
                               '<Placemark.*?<name>(.+?)</name>.*?<Point', "tokens")){end};

lines 81 and 116 could be changed similarly.

I have not tested this extensively

Geoff Warne <gwarne64>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49635:  greaterhobarttrails-hobart-hobart-to-the-pinnacle.kml added by gwarne64 (12KiB - application/vnd.google-earth.kml+xml)
file #49628:  kmlread.m added by philipnienhuis (8KiB - application/octet-stream)
file #49627:  COURSE_20667693.kml added by gwarne64 (98KiB - application/vnd.google-earth.kml+xml)
file #49626:  2020-50km-Course.kml added by gwarne64 (712KiB - application/vnd.google-earth.kml+xml)
file #49623:  kmlread.m added by philipnienhuis (8KiB - application/octet-stream)
file #49622:  kmlread.m added by philipnienhuis (8KiB - application/octet-stream)
file #49600:  ParraWirra.kml added by gwarne64 (145KiB - application/vnd.google-earth.kml+xml)
file #49597:  kmlread.m added by philipnienhuis (8KiB - application/octet-stream)
file #49578:  KoziClassic.kml added by gwarne64 (13KiB - application/vnd.google-earth.kml+xml)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by gwarne64 (Submitted the item)
  • -email is unavailable- added by gwarne64
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 18 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-08-05 philipnienhuis StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-08-05 gwarne64 Attached File- Added greaterhobarttrails-hobart-hobart-to-the-pinnacle.kml, #49635
    2020-08-04 philipnienhuis Attached File- Added kmlread.m, #49628
    2020-08-04 gwarne64 Attached File- Added COURSE_20667693.kml, #49627
    2020-08-04 gwarne64 Attached File- Added 2020-50km-Course.kml, #49626
    2020-08-03 philipnienhuis Attached File- Added kmlread.m, #49623
    2020-08-03 philipnienhuis Attached File- Added kmlread.m, #49622
    2020-08-03 philipnienhuis StatusIn Progress Ready For Test
    2020-08-02 philipnienhuis StatusNeed Info In Progress
    2020-08-02 gwarne64 Attached File- Added ParraWirra.kml, #49600
    2020-08-01 philipnienhuis Attached File- Added kmlread.m, #49597
        StatusNone Need Info
        Assigned toNone philipnienhuis
        Release5.2.0 other
    2020-07-30 siko1056 Summarykmlread does not handle id tags when trying to parse placemark text [octave forge] (mapping) kmlread does not handle id tags when trying to parse placemark text
    2020-07-29 gwarne64 Attached File- Added KoziClassic.kml, #49578
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code