
= XmlCollector
:description: Learn how to configure the XmlCollector in OpenNMS {page-component-title} to collect and extract metrics from XML and JSON documents.

The XmlCollector collects and extracts metrics from XML and JSON documents.

== Collector facts

[options="autowidth"]
|===
| Class Name            | `org.opennms.protocols.xml.collector.XmlCollector`
| Package               | core
| Supported on Minion   | Yes (see limitations)
| Configuration Files   | $\{OPENNMS_HOME}/etc/xml-datacollection-config.xml +
                          $\{OPENNMS_HOME}/etc/xml-datacollection.d/*.xml
|===

== Configuration and use

.Collector-specific parameters for the XmlCollector
[options="header"]
[cols="1,2,3"]
|===
| Parameter
| Description
| Default

3+|  *Required*

| collection
| The name of the XML Collection to use.
| n/a

3+| *Optional*

| handler-class
| Class that performs the collection.
| org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler
|===

The available handlers include:

* org.opennms.protocols.xml.collector.DefaultXmlCollectionHandler
* org.opennms.protocols.xml.collector.Sftp3gppXmlCollectionHandler
* org.opennms.protocols.xml.vtdxml.DefaultVTDXmlCollectionHandler
* org.opennms.protocols.xml.vtdxml.Sftp3gppVTDXmlCollectionHandler
* org.opennms.protocols.json.collector.DefaultJsonCollectionHandler
* org.opennms.protocols.http.collector.HttpCollectionHandler


=== Caveats

The `org.opennms.protocols.json.collector.DefaultJsonCollectionHandler` requires the fetched document to be a single element of type `object` to make xpath query work.
If the root element is an array, it will be wrapped in an object accessible as `/elements`.


== XML collection configuration

Understanding <<operation:deep-dive/performance-data-collection/resource-types.adoc#resource-types, resource types>> helps when editing collector-specific configuration files.

XML collections are defined in `$\{OPENNMS_HOME}/etc/xml-datacollection-config.xml` and groups are defined in `$\{OPENNMS_HOME}/etc/xml-datacollection/`.

.Sample collection definition named `xml-opennms-nodes`
[source, xml]
----
<xml-collection name="xml-opennms-nodes">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:2016</rra>
    <rra>RRA:AVERAGE:0.5:12:1488</rra>
    <rra>RRA:AVERAGE:0.5:288:366</rra>
    <rra>RRA:MAX:0.5:288:366</rra>
    <rra>RRA:MIN:0.5:288:366</rra>
  </rrd>
  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>
</xml-collection>
----

The `url` attribute of the `<xml-source>` supports node and IpInterface <<reference:configuration/filters/parameters.adoc#filter-parameters, field placeholders>>.

NOTE: Metadata expressions can also be used in attributes of the `xml-datacollection-config.xml` configuration file.
This lets the user to also reference credentials stored in the secure credentials.

.The referenced `opennms-nodes.xml` file
[source, xml]
----
<xml-groups>
    <xml-group name="nodes" resource-type="node" resource-xpath="/nodes">
        <xml-object name="totalCount" type="GAUGE" xpath="@totalCount"/>
    </xml-group>
</xml-groups>
----

=== Customize the source request

If you need to customize the request to pull raw data, you can add a `<request>` element within the `<xml-source>` block.

[source, xml]
----
  <xml-source url="http://admin:admin@{ipaddr}:8980/opennms/rest/nodes">
    <request method="GET">
      <parameter name="disable-ssl-verification" value="true"/>
      <parameter name="use-system-proxy" value="true"/>
    </request>
    <import-groups>xml-datacollection/opennms-nodes.xml</import-groups>
  </xml-source>
----

The request element can have the following optional child `<parameter>` elements:

[options="header"]
[cols="1,3,2"]
|===
| Parameter
| Description
| Default

| timeout
| The connection and socket timeout in milliseconds.
| n/a

| retries
| How often to repeat the request in case of an error.
| 0

| disable-ssl-verification
| Do not attempt to verify the name on the SSL certificate against the `xml-source` URL attribute.
| false

| use-system-proxy
| Should the system-wide proxy settings be used?
Configure the system proxy settings via <<operation:deep-dive/admin/configuration/system-properties.adoc#system-properties, system properties>>.
| false

| xslt-source-file
| Full path to XSLT file to transform XML data before processing to collection definitions.
| n/a
|===

=== HTTP headers

If the endpoint being collected requires additional headers, you can add them as part of the `<request>` object.

[source, xml]
----
  <xml-source url="https://{nodelabel}/api/path/to/endpoint">
    <request method="GET">
      <header name="Authorization" value="Bearer static_token_here" />
      <header name="X-Api-Key" value="api-key-here" />
    </request>
    <import-groups>xml-datacollection/file.xml</import-groups>
  </xml-source>
----

=== Retrieve data from a POST request

If your XML data needs to be retrieved via a POST instead of a GET, you can specify the body content of the request.

.`application/form-urlencoded`
[source, xml]
----
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/x-www-form-urlencoded'><![CDATA[
        <form-fields>
          <form-field name='firstName'>John</form-field>
          <form-field name='lastName'>Doe</form-field>
        </form-fields>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>
----

.`application/xml`
[source, xml]
----
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/xml'><![CDATA[
        <person>
          <firstName>John</firstName>
          <lastName>Doe</lastName>
        </person>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>
----

.`application/json`
[source, xml]
----
  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/json'><![CDATA[
        {
          person: {
            firstName: 'John',
            lastName: 'Doe'
          }
        }
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>
----

== Mapping values

Sometimes data is represented as string values.
These values are normally not persisted as time-series data; this means changes are not visible over time.
To circumvent this, we allow mappings defined between input values and values to be persisted.

Let's assume we have the following data input:

[source, xml]
----
<records>
    <record>
        <input>aaa</input>
        <read>123</read>
    </record>
    <record>
        <input>bbb</input>
        <read>456</read>
    </record>
    <record>
        <input>ccc</input>
        <read>789</read>
    </record>
</records>
----

The following group configuration allows you to persist the `input` values as integer values over time:

[source, xml]
----
<xml-group name="xml-mapping" resource-type="input" resource-xpath="/records/record" key-xpath="input">
    <xml-object name="input" type="GAUGE" xpath="input"> <1>
        <xml-mapping from="aaa" to="10" /> <2>
        <xml-mapping from="bbb" to="20" />
        <xml-mapping to="1000" /> <3>
    </xml-object>
    <xml-object name="read" type="GAUGE" xpath="read" />
</xml-group>
----
<1> The data-type is altered in the `xml-object` element from `STRING` to `GAUGE`.
<2> In this example we associate `aaa` to `10` and `bbb` to `20`.
<3> Define a default value by omitting the `from` attribute in a `xml-mapping` definition.
In this example `ccc` will be associated with the default value of `1000`.

== Test XML collection via Karaf

With the configuration in place, you can test it using the `collect` command available in the xref:reference:karaf-shell/karaf-shell.adoc[Karaf shell]:

[source, karaf]
----
opennms:collect -n 1 org.opennms.protocols.xml.collector.XmlCollector 127.0.0.1 collection=xml-opennms-nodes
----

== Examples

See xref:performance-data-collection/collectors/xml-examples.adoc[XmlCollector configuration examples] for end-to-end recipes against common HTTP/JSON APIs.
