Ir-xml
ir-xml Message Formats
This is the first public release of our innovative real-time XML message formats.
We have chosen to use extensible XML message formats and the Open Source Apacheactivemq as the main transport.
We use Attributed XML because it is compact and it beats JSON by a few characters (we tested it). The total character count is significant in real-time messaging systems when you are handing the amount of data per second that our IR-GIS ® product suite handles. The product applicaions use a specially-crafted high-speed XML parser that is an order of magnitude faster than the Xalan parser, and we have also developed some very fast JSON parsers too along the same lines, for the same reasons, just incase.
The product suite also employs special purpose concurrent collections that down-size and give memory back to the JVM, and we claim that our products have no memory leaks.
Observations
An observation represents a position report at a specified time - they map to a PlotFeaturePositonReport class.
The associated identifying attributes <at> and the amplifying details <dt> are associated with the observation point and stored in an AttributeContext class.
<ci v="1" sacc="%s" csn="%s" fd="%s"> // version, class, designator, feed
<ob [id="%s"] ts="%d" sc="%s"> // optional assigned-id, timestamp milliseconds, source
<p lo="%.4f" la="%.4f" al="%d"/> // longitude, latitude, altitude
<rb r="%.4f" b="%.1f"/> // range, bearing from sensor
<dt n="%s" v="%s"/> // detail: name, value pair e.g. callsign, vh-ums
<at n="%s" v="%s"/> // attribute: name, value e.g. iff, 1200
</ob>
</ci>
Observations <ob/> can contain repeating <dt/> and <at/> elements. <at/> elements are used to identify and participate in tracking the platform that is responsible for the position reports. <dt/> elements are used to describe the position report but are not used in machine identification or tracking.
This is processed by dropping any nested <ob> data and stored as:
<ob>
<p ts="%d" lo="%.4f" la="%.4f" [al="%.d"] r="%.1f" b="%.2f" s="%.1f" h="%.1f" e="%.3f" q="%.3f"/>
<at id="ob%d.%d" ci.csn="%s" ci.sc="%s" %s="%s" ... />
<dt ci.sacc="%s" ci.fd="%s" ci.v="1" %s="%s" ... />
</ob>
Estimates or Tracks
Estimates are forwarded from a tracker and represent the estimated position of a moving platform. A tracker typically uses a Kalman filter to follow and smooth position reports from multiple sensors to arrive at estimated positions, though other simplistic policies can be used, particularly when data is derived from GPS positions and raw messages contain CRC for error detection so erroneous data can be rejected.
<tk id="%s" q="%.5f" sacc="%s"> // id, quality, class
<es ts="%d" e="%.5f" m="2"> // time, error, model
<p lo="%.4f" la="%.4f" al="%d"/> // estimated longitude, latitude, altitude
<cr s="%.1f" h="%.1f"/> // speed,heading
<ob id="%s" ts="%d" sc="%s"> // assigned-id, timestamp milliseconds, source
<p lo="%.4f" la="%.4f" al="%d"/> // longitude, latitude, altitude
<rb r="%.4f" b="%.1f"/> // range, bearing from sensor
<at n="%s" v="%s"/> // attribute: name, value e.g. iff, 1200
<dt n="%s" v="%s"/> // detail: name, value pair e.g. callsign, vh-ums
<dt n="%s" v="%s"/> // detail: name, value pair e.g. callsign, vh-ums
...
</ob>
<ob id="%s" ts="%d" sc="%s"> // assigned-id, timestamp milliseconds, source
<p lo="%.4f" la="%.4f" al="%d"/> // longitude, latitude, altitude
<rb r="%.4f" b="%.1f"/> // range, bearing from sensor
<at n="%s" v="%s"/> // attribute: name, value
<dt n="%s" v="%s"/> // detail: name, value
<dt n="%s" v="%s"/> // detail: name, value
...
</ob>
</es>
...
<at n="%s" v="%s"/> // attribute: name, value
<at n="%s" v="%s"/> // attribute: name, value
...
<dt n="%s" v="%s"/> // detail: name, value
<dt n="%s" v="%s"/>
<pf ... /> // platform element
...
</tk>
Track <tk/> elements contain an estimated position for one or more <ob/> observations. Both <tk/> and <ob/> may have attributes <at/> and details <dt/> associated with them, the nested observations are ignored so we can preserve performance.
These are processed as semantically equivalent forms for storage and subsequent transmission from the historical pullback engine as follows:
<tk> <p ts="%d" lo="%.4f" la="%.4f" al="%d" e="%.1f" q="%.1f" s="%.1f" h="%.1f"/> <at fid="%s" .../> <dt tk.sacc="%s" tk.m="2" tk.v="1" tk.pf="..." .../> </tk>
Nested Annotations
Additional elements may be nested inside, any data, in particular for track data we have this example of an 'n' nesting and how it is parsed:
<tk id="12345">
<n
fir="%s" // FIR
t="%s" // tail-id
c="%s" // callsign
dp="%s" // departure point
ds="%s" // destination
ad="%s" // alternate
dt="%s" // departure time
eta="%s" // estimate time of arrival
ac="%s" // type
ft="%s" // flight type
tn="%s" // track number
/>
<tk/>
This, by default without any additional schema, will be replaced as though it were a <dt> element with attributes using nested names. Since it is nested inside a <tk> element it will have its attribute name prefixed by tk.. The same goes with any other nesting, the attributes are prefixed by the element tag when the structures are flattened out, with each nesting resulting in a new prefix separated by a dot (.). Any number of elements may be nested inside other elements and undergo the same processing.
For example the attribute in an <n dt="0020" nesting underneath a <tk element will be ultimately named tk.n.dt as though it where in <tk n.dt="0020" ... />. These "flattened" names are how attributes are passed by the parsers inside the IR-GIS ® application's memory structures, including IR-GIS ® MapView, IR-GIS ® HPS, and IR-GIS ® MapWorld.
ir-xml attribute values
As mentioned previously, attributes that are included in a PositionReport element <p> are called Dynamic or Kinematic attributes, and are stored with PlotFeaturePositionReport, while Identifying <at> attributes and Detail <dt> attributes are stored in the AttributeContext. Any time a message occurs where either an Identifing (very unlikely) or Detail attribute value changes (more likely) a new AttributeContext must be contructed to hold the new values and associated with the PlotFeaturePositionReport. Whereas Dynamic Attributes are directly associated in each PlotFeaturePositionReport, and expected to change at each report. Fundamental Dynamic Attributes are embedded, such as time, longitude, latitude, altitude and may be extended by placing such attributes in the <p element.
An associative array, indexed by the schema-driven identifier is used to find the corresponding Attribute values. Dynamic values are assigned negative indexes and their values are associated with the PositionReport, whereas positive indexes are into the AttributeContext values. When new attributes are discovered, if they are dynamic they cause the PositionReport storage to extend in the negative direction, and if they are identifing or detail attributes they cause an AttributeContext to extend in the positive direction, Schema is automatically made to match the runtime type if schema has not been supplied by the applications users. Schema can be employed to override message attribute type classification to optimise storage e.g. any attribute associated witth a feature that changes at each report should be classified as dynamic and stored with the report to minimise storage overhead, otherwise a new AttributeContext will be constructed. Usually AttributeContexts are shared between multiple position reports because they are comprised from values that do not change often, if at all. Only the Dynamic attributes are expected to change, but changes will be handled in the other associated attributes should they change - at the expense of increased memory usage.
Events
An event may be sent using the <ev> element. Currently we support events that contain a position report, and this can be extended in the future to support events that do not contain a position report but just attributes and details where the id is used to obtain associated information from a database.
<ev ts="123445">
<p la="-35.5" lo="123.5"/>
<at id="sensor1"/>
<dt n1="v1..." .../>
</ev>
In the future when an <ev> is received without a position report <p> then a request will go out on the wire requesting that the server return the <p> details to the client for the event location. This might be done by the central historical pullback server, or by another server.
This data will be concatenated onto the <at>, <dt> or <p> element data depending on the types stored in the database, and will be correlated via the mandatory id.
The wire communication details are yet to be designed but will include the id= and the ts= to assist with the identification.
The recommendation is that the <ev> should also contain a type or discriminant so style rules can be written to render different graphics for each event type.
Detecting broken JMS connections
There are several techniques to keep JMS connections alive:
- use the JMS keep-alive mechanism (if it were to work properly).
- use periodic expected traffic to reset a timeout variable and reconnect the connection when the timeout is exceeded.
- use a <ka.../> message to keep a CommandServer connection open
- use a <ping.../> message to keep a CommandClient and CommandServer bi-directional connection open.
real-time JMS inputs
The IR-GIS client monitors the JMS feed topics and it will reconnect when traffic has not been received for a period (of 30 seconds). This is a self-monitoring feature to ensure reliability on systems where traffic is expected 24/7.
Command Server
Currently the command-channels are connected to queues and the server would not necessarily know if the queue connection is functioning unless traffic is received on the queue (due to the failure of JMS to keep them alive), and by their nature traffic is expected to be low on these command queues.
Since a client is not expected to send frequent commands to these queues the CommandServer framework periodically sends a keep-alive message to the queue it is listening too. The keep-alive is only sent after the dead-period expires and then at every ping interval thereafter until the maximum pings pending any message being received is exceeded - in which case the connection is reset. The dead-period is 10 seconds, the ping interval is 1 second thereafter, and the maximum pings pending is 5. So the maximum time the CommandServer must tollerate a broken command connection is 15 seconds.
The keep-alive format is as follows:
<ka ts="...time of send..." name="...task name sending.."/>
Because the CommandServer is responding to a private command destination when sending data back to the client it may send a ping to determine whether the destination is valid (the historical server does).
The ping: <ping-client ts="..." name="...'/> The client is required to respond: <client-response ts="..." name="..."/>
Command Clients
The CommandClient suffers from a similar problem, it doesn't know if it is connected to the CommandServer and whether the CommandServer is listening. Thus the ComandClient can periodically send a <ping> command to ellicit a response. If after sending, one or more pings, and no response is seen after the pending ping count exceeds the maximum pings pending the channel is reconnected.
The client sends: <ping-server ts="..." name="..."/> The server is meant to respond with: <server-response ts="..." name="..."/>
Of course the client is not required to send a <ping if a message has been received within the time-out period, or when it has other messages to send.
NOTE: All control commands are marked with the boolean property "ir-gis-control-command" = true and if they contain a correlationID the receiver must send that correlationID back to the sender in the response.
Historical Pullback Command and Response Message Formats using ir-xml
Realtime Layers are named by names familiar to the customer, e.g. ATC and TRACKS. These layers have FeatureCollections that accept their real-time traffic from JMS topics following the publish and subscribe paradigm where the topic name is specified by the broker and source specified in e.g the jms-spec.src:
<?xml version="1.0" encoding="UTF-8"?>
<root>
<source broker="tcp://localhost:61616" src="ATC" layer="ATC" ageing="360" archive="true" sld="ATC" schema="atc-schema.xml"/>
<source broker="tcp://localhost:61616" src="TRACKS" layer="TRACKS" ageing="360" archive="true" sld="TRACKS" schema="tracks-schema.xml"/>
<source broker="tcp://localhost:61616" src="EVENTS" layer="EVENTS" ageing="360" archive="true" sld="EVENTS" schema="events-schema.xml"/>
</root>
Historical map layers receive their data from one-to-one temporary destinations (which may be queues or topics depending on the implementation returned by the CommsFactory) these destintaions are established for the duration of the historical pullback, so only the originating client (historicTool), and the responding server (historicServer) are involved in the communication.
Source commands
For each real-time layer supporting communications on a topic, where that topic is being recorded because it is marked as archive="true", there can be a historical layer that supports communications with the historical pullback server listening to a command queue. These command-queue destinations are named by example as: queue:ATC in the historical sense, and topic:ATC in the real-time sense. The archive historical-pullback server source will subscribe to the corresponding topic and record the topic messages and these messages can be retrieved from that server by sending command's to the server source's corresponding command queue for that source. These commands are referred to as source commands.
search command
The first request destined to the historical server from the client is composed as a query with optional components and sent to the command queue for the layer FeatureSource's destination. For berevity the optional components are enclosed with [ ], these brackets are not part of the message traffic.
<command com=”search”>
[ <mbr ula=”lat” ulo=”lon” rla=”lat” rlo=”lon”/> ]
[ <date [sd=”timestamp”] [ed=”timestamp”]/> ]
[ <index attribute=tk.pf.ai conj="or">RED BLUE</index> ]
[ <index attribute=tk.pf.ai conj="and">FAST SLOW</index> ]
[ <words conj="and">abc def ghi jkl mno</words> ]
[ <words conj="or">RED YELLOW BLUE</words> ]
[ <word v=”abc”/> ]
</command>
and the server's response on the client's replyTo destination is a formatted message corresponding to the first feature report that was matched to the selection criteria. This response contains an entity id which is used to obtain all the reports corresponding to the selection criteria and constraints. So we pick off the eid attribute, <at> and <dt> elements for display in the user selection table.
historical index values
The addition of specialised indexes is planned, they will be specified in the schema file. There will be four index modes required:
When these new modes are activated, the search command is enhanced with index elements, where the attribute is specified and the conjunction or and and or are specified. The search command is treated as an out and, and the conjunctions are applied underneath. I expect that for any particular search will would permit them to include the index attributes in the search more than once, I am not sure about the word search - but it would make sense to let them send more than one word search with a different conjunction as well. Later on we can expand the expression handling of the search to support nested expression - the above proposal is simpler to get going in the short time-frame.
search response
The response to this search is easy it is composed from the reconstruction of the recorded traffic, which does not record nor send the nested observations, all we need is the: <tk> <p> and <at> <dt> elements, but we can also accept full vang-xml for backwards compatibility to the current Terra Text interface. Though it is not our desireable (new) format.
<tk hp="20101230-0-2400" hid="201009101234567">
<p ts="3333333" lo="145.1" la="-35.5" al="100" e="1.0" s="100" h="075"/>
<at fid="1234567.123" AAA="BBB" CCC="DDD" EEE="FFF" GGG="HHH"/>
<dt tk.sacc="XXX" EEE="FFF" GGG="HHH" pf.ai="RED"/>
</tk>
<ob hp="20101230-0-2400" hid="201009101234568">
<p ts="3333333" lo="145.1" la="-35.5" al="100" e="1.0" s="100" h="075"/>
<at fid="1234567.123" AAA="BBB" CCC="DDD" EEE="FFF" GGG="HHH"/>
<dt tk.sacc="XXX" EEE="FFF" GGG="HHH" pf.ai="RED"/>
</ob>
<ev hp="20101230-0-2400" hid="2010091012345679">
...
</ev>
There will be one <tk>, <ob> or <ev> item message sent asynchronously back to the client for every matching feature for the selection criteria because the historical-pullback server processes every selection asynchronously to permit it to retrieve data from multiple partitions spanning multiple days. This requires the client to maintain its temporary destination connection open for the duration. (The recommendation is that the client only close the connection on error, and that the abort command be used to terminate a session.)
The returned candidate <tk>, <ob> or <ev> messages need to be parsed and appended to the Result display in the user's Result selection table so the user can select which features they want to pull-back. These selection responses may take some time to trickle in. (The recommendation is to issue the abort command and wait a while before issuing the "retrieve" command to cancel any pending messages.)
Multiple candidates can be sent, one per message response, and will be captured over time for display in the selection table. It could take quite some time before the server finishes trickling in results because results might span a long way over time and involve complex indexing. The client should keep the connection open for the duration otherwise the client's temporary queue will dissappear. The temporary replyTo destination is maintained for the duration of the extent of the client's connection.
Once the user chooses the track(s) and submits the selection to the server in a retrieve command, the server is expected to send all the reports for the corresponding &<key hp="..." hid="..."> elements for all the selected features so we can display the complete feature history that corresponds to the original selection parameters. This may take a longer time than the initial response to the select command because it involves a lot more data.
The <tk hp and <tk hid attributes are included by the historical archive server and are used to partition and index features. The hp specifies the partition root name, and hid specifies the feature entity index. These attributes will be parsed as tk.hp and tk.hid respectively and will appear in the info tool as attributes on the retrieved features.
empty response
An empty query response is signified by sending an empty MapMessage back to the client, which intercepts it and displays "No results" on the status bar.
retrieve command
The command specifies what the client wants retrieved, and the original mbr and date constraints, so the server does not need to maintain that state, and the server will send that response to the clients replyTo destination (be that a queue or topic).
The retrieve message format follows:
<command com=”retrieve”>
[ <mbr ula=”lat” ulo=”lon” rla=”lat” rlo=”lon”/> ]
[ <date [sd=”timestamp”] [ed=”timestamp”]/> ]
<key hp="201009-0-2400" hid="201009101234567"/>
<key hp="..." hid="..."/ ts="..."/>
</command>
A key hp hid tuple must be sent for each feature that the user choses from the Search Results table, and the server shall respond by sending the recorded ir-xml that matches the constraints to the replyTo destination, single reports at a time, and these will be displayed in the Historical layer of the selected map. The RIParser for ir-xml/vang-xml has been adapted to parse the hp and hid attributes correctly and these are turned into feature descriptions. They are just indexing handles that must be used because the historical server partitions features based on time, and retrieves features in the database based on their entity id. The atttribute hp has the local-date of a midnight, partition offset, and partition duration encoded in it, while the hid is the entity index. (The h prefix means historical, p means partition, and id is the identity identifier.)
Each feature message returned to the client will be comprised from only one feature report at a time - similar to the regular traffic, and the ir-xml is very similar to but more optimised than vang-xml.
The server response shall be to the replyTo destination in the usual ir-xml format:
<tk hp="..." hid="...">
... etc in accordance with ir-xml
</tk>
<ob hp="..." hid="...">
...
</ob>
<ev hp="..." hid="...">
...
</ev>
Note the responses are self-contained and they DO NOT wrap any other elements. The <tk> elements are sent to the destinaton in their recorded semantically equivalent formats. (We can have <tk> ... </tk> for tracks, <ob> <p> .. <at> <dt> </ob> elements for observations, and similarly for <ev> events. Single <tk>, <ob> and <ev> messages will be in each JMS Message sent to the client's temporary queue.
This specification permits interaction with the existing Terra-text framework, provided that frame work is altered to send all responses to the temporary replyTo destination, and provided we send the information to the correct command queue that the TeraText interface is subscribed to. (My recommendation would be for them to change to subscribing on the TRACKS queue to be compatible with us, and to modify their code so the responses are all on the replyTo queue - if we ever are forced to use their historical-pullback server. Instead I would recommend conversion from their server to our partitions.)
Because the client-server connection may be closed or aborted by the client, and the client can press submit again at anytime, there is no need for an ACK. There is also no need for any time-outs provided the user can issue requests anytime they want i.e. provided there are no restrictions, and the command channel is kept open while ever the client is waiting for a response, which could be a response to the select or the retrieve. The select response is just a list of partial tracks, and their eids, while the response to a retrieval is a stream of ir-xml messages correlated to the selection constraints.
The client is at liberty to use a UUID for each session however the server will just reflect the UUID back to the client, it is up to the client to consider what it does with the clientId.
Abort command
Any client may send an abort command to the server, this will cause the server to drop any pending messages for this client. As the server is multi-threaded there may be instances of messages pending in the "pipe".
<command com="abort"/>
Server Commands
In addition to the source search, retrieve and abort commands, there is one command queue defined in the historical.server.properties file. This command queue is used to change the state of the server with the following commands:
* <command com="pause"/> - which will pause all sources of the server * <command com="resume"/> - which will resume all sources after a pause * <command com="stop"/> - which will stop the server * <command com="rebuild" sd="" ed=""/> - which is used to rebuild indexes for all sources between the period
The period may be omitted, which will imply the current partition. sd="0" means from the first partition, with ed="0" means the last partition, using the same semantics as the search command.
Customer Command Formats
Simple command formats are used to exchange commands and acknowledgements between the macMap and an engine on the COMMAND topic. We recommend that all commands go to a queue and that all responses are returned on a tempoarary replyTo queue i.e. that the developers migrate over to and adopt the command-response pattern.
<cmd t="engine" f="macMap" id="12345" u="sid"> <ch tk="id" [attributes from pf]/> </cmd> <ack t="macMap" f="engine" id="12345" r="success"/>
Commands will be packaged up as templates with expression references to fill in parameters, similar to OCF and String.format statements, this way our client can customise what editing and buttons place into the messaging stream.
Parsing of the <ack> will be similar to sscanf parsing.
The commands fall into:
Import/Export message formats
The export formats are relatively easy to implement as they can be included as a combination of methods on PlotFeaturePositionReport and the SaveToFile classes.
The import is more complex as it is no longer true that one input record contains a PositionReport, in the case of KML multiple associated file records need to be read to build up multiple position reports for each track because the <gx:Track> format requires the tracks to be sent in their entirety. I.e. GoogleEarth cannot track, whereas IR-GIS can.
Example formats for each type follow:
ir-xml
<tk><p ts="1299977223378" lo="110.04167" la="-63.75000" al="10000.00000" e="1.0" q="1.0" r="0.0" b="0.0" s="8071.9" h="90.0"/><at tk.iff="172" tk.c="red" tk.pf.ai="red" tk.n.c="FL90" tk.n.t="TAIL-90" tk.m="2" fid="1299977215372.90"/><dt tk.n.d="test1" tk.n.e="test2" tk.n.f="test3" tk.n.g="test4" tk.n.h="test5" tk.n.i="test6" tk.n.j="test7"/></tk> <tk><p ts="1299977223378" lo="110.04167" la="-62.87500" al="10000.00000" e="1.0" q="1.0" r="0.0" b="0.0" s="8010.1" h="90.0"/><at tk.iff="201" tk.c="green" tk.n.ai="green" tk.n.c="FL97" tk.n.t="TAIL-97" tk.m="2" fid="1299977215372.97"/><dt tk.n.d="test1" tk.n.e="test2" tk.n.f="test3" tk.n.g="test4" tk.n.h="test5" tk.n.i="test6" tk.n.j="test7"/></tk>
CSV format
The following example shows that the start of the CSV export file contains a heading for each value, values are emitted on each subsequent lines and the attributes correspond to the heading. Null values result in an empty element e.g. ,,; and the values are emitted in order corresponding with the heading names:
ts,la,lo,al,e,q,r,b,s,h,tk.iff,tk.c,tk.n.ai,tk.n.cn,tk.n.c,tk.n.t,tk.n.op,tk.n.dp,tk.n.ds,tk.n.ac,tk.n.qn,tk.n.ft,tk.n.eta,tk.n.dt,tk.eid,tk.n.kl,tk.m,t.n.fdi,tk.pf.tn,tk.n.rl,tk.n.rm,tk.n.ft,fid,tk.n.d,tk.n.e,tk.n.f,tk.n.g,tk.n.h,tk.n.i,tk.n.j 19/05/2010 101231,56.125,115.458333,10000.0,1.0,1.0,0.0,0.0,7472.3,90.0,2071,green,green,,FL1049,TAIL-1049,,,,,,,,,,,2,,,,,,1299971779023.1049,test1,test2,test3,test4,test5,test6,test7,,,,,,,,,
KML format
Substantial changes have occurred in GoogleEarth over a period, so much so that we can now use GoogleEarth 5.0 to do stuff that its predecessors could not do.
To obtain an idea of the history the PocketFMS tesselated track format is captured from my GPS and was used to display in Google 34 and Google 4. This is an inefficient form as it does not cope with time correctly. When looking at this older format the position information is output with one second separation.
The KML OGC standard is available here media:OGC KML.pdf.
GoogleEarth <gx:track> format
A more appropriate format is found in the new GoogleEarth 5.0 <gx:track> format <gx:track> .
<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2">
<Document>
<name>ir-gis 2011-03-21T21:35:52.201+11:00</name>
<Snippet>Created 2011-03-21T21:35:52.201+11:00</Snippet>
<!-- Normal track style -->
<LookAt><gx:TimeSpan><begin>2011-03-21T21:35:52.201+11:00</begin><end>2011-03-21T21:35:52.807+11:00</end></gx:TimeSpan>
<longitude>148.110275</longitude><latitude>-34.656506</latitude><altitude>100000.0</altitude><range>1800.000000</range>
</LookAt>
<Style id="red_n">
<IconStyle><scale>.5</scale><Icon><href>http://earth.google.com/images/kml-icons/track-directional/track-none.png</href></Icon></IconStyle>
<LabelStyle><scale>0</scale></LabelStyle>
<LineStyle><color>FF1400FF</color><width>6</width></LineStyle>
</Style>
<Style id="red_h">
<IconStyle><scale>1.2</scale><Icon><href>http://earth.google.com/images/kml-icons/track-directional/track-none.png</href></Icon></IconStyle>
<LineStyle><color>FF1400FF</color><width>9</width></LineStyle>
</Style>
<StyleMap id="red">
<Pair><key>normal</key><styleUrl>#red_n</styleUrl></Pair>
<Pair><key>highlight</key><styleUrl>#red_h</styleUrl></Pair>
</StyleMap>
<Style id="orange_n">
<IconStyle><scale>.5</scale><Icon><href>http://earth.google.com/images/kml-icons/track-directional/track-none.png</href></Icon></IconStyle>
<LabelStyle><scale>0</scale></LabelStyle>
<LineStyle><color>FF0078F0</color><width>6</width></LineStyle>
</Style>
<Style id="orange_h">
<IconStyle><scale>1.2</scale><Icon><href>http://earth.google.com/images/kml-icons/track-directional/track-none.png</href></Icon></IconStyle>
<LineStyle><color>FF0078F0</color><width>9</width></LineStyle>
</Style>
<StyleMap id="orange">
<Pair><key>normal</key><styleUrl>#orange_n</styleUrl></Pair>
<Pair><key>highlight</key><styleUrl>#orange_h</styleUrl></Pair>
</StyleMap>
<Schema id="schema">
<gx:SimpleArrayField name="ts" type="long"><displayName>ts</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="la" type="double"><displayName>la</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="lo" type="double"><displayName>lo</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="al" type="double"><displayName>al</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="e" type="float"><displayName>e</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="q" type="float"><displayName>q</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="r" type="float"><displayName>r</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="b" type="float"><displayName>b</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="s" type="float"><displayName>s</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="h" type="float"><displayName>h</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="tk.iff" type="string"><displayName>tk.iff</displayName></gx:SimpleArrayField>
<gx:SimpleArrayField name="tk.c" type="string"><displayName>tk.c</displayName></gx:SimpleArrayField>
</Schema>
<Folder>
<name>TRACKS</name>
<Placemark><name>1300701587528.14</name><styleUrl>#red</styleUrl>
<gx:Track><altitudeMode>absolute</altitudeMode>
<when>2011-03-21T21:35:51.557+11:00</when>
<when>2011-03-21T21:35:52.557+11:00</when>
<gx:coord>148.110275 -34.656506 1201.826400</gx:coord>
<gx:coord>148.109573 -34.656212 1201.826400</gx:coord>
<ExtendedData><SchemaData schemaUrl="#schema">
<gx:SimpleArrayData name="ts">
<gx:value>1300703751557</gx:value>
<gx:value>1300703752557</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="la">
<gx:value>-34.656506</gx:value>
<gx:value>-34.656212</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="lo">
<gx:value>148.110275</gx:value>
<gx:value>148.109573</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="e">
<gx:value>1.0</gx:value>
<gx:value>1.0</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="q">
<gx:value>1.0</gx:value>
<gx:value>1.0</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="r">
<gx:value>0.0</gx:value>
<gx:value>0.0</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="s">
<gx:value>107.5</gx:value>
<gx:value>107.0</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="tk.iff">
<gx:value>6519</gx:value>
<gx:value>6519</gx:value>
</gx:SimpleArrayData>
<gx:SimpleArrayData name="tk.c">
<gx:value>red</gx:value>
<gx:value>red</gx:value>
</gx:SimpleArrayData>
</SchemaData></ExtendedData>
</gx:Track>
v/Placemark>
</Folder>
</Document>
</kml>
parsing
All that has to be parsed is the <Schema> table and the <PlaceMark> and contained elements.
The <Schema> table contains the names of all the attributes associated with a <PlaceMark>, and a <PlaceMark> contains all the reports which are associated with the: timestamp, position, and each attribute.
This means that the style components are ignored because the colour attribute will restore the original colour value on ingestion into IR-GIS.
Those two groups of elements are all that are required to restore a feature's reports. (The display name in the schema elements should probably match the actual attribute display name.)
rework
RIParserContext has been renamed to UpdateContext, and the new name UpdateContextAttributeEntry has been coined for the elements stored in the details, attributes and dynamic collections inside that UpdateContext.
The general parsing infrastructure has been altered to enable the parser to be called multiple times with incomplete Strings, a change enabled by the event-like parsing and by placing the updateFeature(UpdateContext) inside the parser at the appropriate end_element().
This enables the parser to build up associated state over time, such as the parse of the schema table and also to enable it to parse data from multiple lines ingested from the input file.
Thus the new parser framework will work with single amq messages containing complete text, single-lines from files containing complete text, or any combination of messages or records that as a whole form a complete parse specification.
PocketFMS format
This is the KML document format used by PocketFMS, which we have been using in PumpKML, it is a single-placemark containing a LineString of position reports. This form is much easier to process in the pump and easier to generate and parse but all extended data must be associated by some other means (See PumpKML).
<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.0">
<Document>
<name>Your PocketFMS flight</name>
<open>1</open>
<LookAt>
<longitude>151.91625976562500000</longitude>
<latitude>-27.54358863830566406</latitude>
<range>6000</range>
<tilt>45</tilt>
<heading>0</heading>
</LookAt>
<Style id="icon">
<IconStyle>
<Icon>
<href>root://icons/palette-2.png</href>
<y>32</y>
<w>32</w>
<h>32</h>
</Icon>
</IconStyle>
</Style>
<Placemark>
<description><![CDATA[
PocketFMS Breadcrumbs To IGC/KML v0.23 (c) ALiEN
<br><a href="http://www.pocketfms.org/">PocketFMS</a>
]]></description>
<name>Your PocketFMS flight - Extruded path</name>
<styleUrl>#icon</styleUrl>
<Style>
<LineStyle>
<color>7f0000ff</color>
<width>0</width>
</LineStyle>
<PolyStyle>
<color>7fff00aa</color>
<outline>0</outline>
</PolyStyle>
</Style>
<LineString>
<extrude>1</extrude>
<tessellate>0</tessellate>
<altitudeMode>absolute</altitudeMode>
<coordinates>
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
</coordinates>
</LineString>
</Placemark>
<Placemark>
<description><![CDATA[
PocketFMS Breadcrumbs To IGC/KML v0.23 (c) ALiEN
<br><a href="http://www.pocketfms.org/">PocketFMS</a>
]]></description>
<name>Your PocketFMS flight - Line path</name>
<styleUrl>#icon</styleUrl>
<Style>
<LineStyle>
<color>7f0000ff</color>
<width>5</width>
</LineStyle>
</Style>
<LineString>
<extrude>0</extrude>
<tessellate>0</tessellate>
<altitudeMode>absolute</altitudeMode>
<coordinates>
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358863830566406,570
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
151.91625976562500000,-27.54358673095703125,549
</coordinates>
</LineString>
</Placemark>
</Document>
</kml>
