
[[flows-basic]]
= Basic Flows Setup
:description: Get started with flows data collection in {page-component-title} including configuring Elasticsearch, Kafka, and protocol listeners.

This section describes how to get started with flows to collect, enrich (classify), persist, and visualize flows.

== Requirements

Make sure that you have the following before you set up flows:

* A configured {page-component-title} instance.
* One or more devices that send flows visible to {page-component-title}, and that are monitored with SNMP.
* Elasticsearch cluster set up with the https://github.com/OpenNMS/elasticsearch-drift-plugin[Elasticsearch Drift plugin] installed on every Elasticsearch node.
** The Drift plugin persists and queries flows that {page-component-title} collects.
The Drift version must match the targeted Elasticsearch version.
** (Optional) Configure Elasticsearch variables like `search.max_buckets` or maximum heap size `ES-JAVA_OPTS` if the default values are not sufficient for your volume of flows or number of nodes.
** (Optional) Create a job to clean the indices so that the disk does not fill up (for example, "keep _X_ days of flows").
Filled disks are a challenging problem to address for those who are not Elasticsearch experts.
We recommend using the https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html[Elasticsearch Curator tool] to do this.
** {page-component-title} set up to monitor the Elasticsearch stack and generate an alarm if it goes down.
* The OpenNMS plugin for Grafana configured to visualize flows.
** Configure the flow and performance data sources.

== Configure {page-component-title} to communicate with Elasticsearch

{page-component-title} must be set up to communicate with Elasticsearch and persist the collected flows data in a defined directory:

. Connect to your {page-component-title} xref:reference:karaf-shell/karaf-shell.adoc[Karaf shell]:
+
[source, console]
----
ssh -p 8101 admin@localhost
----

. Edit `$\{OPENNMS_HOME}/etc/org.opennms.features.flows.persistence.elastic.cfg` to configure flow persistence to use your Elasticsearch cluster:
+
[source, karaf]
----
config:edit org.opennms.features.flows.persistence.elastic
config:property-set elasticUrl http://elastic:9200
config:update
----

. (Optional) Edit or create `$\{OPENNMS_HOME}/etc/org.opennms.features.flows.persistence.elastic.cfg` and configure persistence settings:
+
[source, xml]
----
elasticUrl = http://10.10.3.218:9200 <1>
connTimeout = 30000
readTimeout = 300000
settings.index.number_of_replicas = 0
settings.index.number_of_shards=1
settings.index.refresh_interval=10s
elasticIndexStrategy=daily
----
<1> Replace with comma-separated list of Elasticsearch nodes.

** See <<deep-dive/elasticsearch/introduction.adoc#ga-elasticsearch-integration-configuration, General Elasticsearch configuration>> for a complete set of configuration options.

== Configuring NetFlow Composable Templates

OpenNMS supports (https://www.elastic.co/docs/manage-data/data-store/templates)[composable templates] for more flexible and modular management of Elasticsearch index settings.
To enable composable templates for NetFlow data, update below configuration in Karaf:

[source, karaf]
----
config:edit org.opennms.features.flows.persistence.elastic
config:property-set useComposableTemplates true
config:update
----

=== Managing NetFlow Templates and Policies

NetFlow component and index templates can be managed from the following directory:

`$\{OPENNMS_HOME}/etc/netflow-templates`

You can customize existing templates, add new ones, or define Index Lifecycle Management (ILM) policies within this folder. Any valid Elasticsearch template or ILM JSON placed here will be picked up and applied by the system.

When placing files in the netflow-templates directory:

* Files intended as ILM policies should have names that include keywords such as ilm, lifecycle, or policy.
* Files with names containing setting(s), mapping(s), alias(es), or component will be treated as component templates.
* Files with names containing index or composable will be recognized as index templates.




== Multi-protocol listener

With most tools, if you are monitoring multiple flow protocols, you must set up a listener on its own UDP port for each protocol.
However, {page-component-title} allows a multi-port listener option; this listener, named `Multi-UDP-9999`, is enabled by default and monitors multiple protocols on a single UDP port (`9999`).
The default configuration includes support for Netflow v5, Netflow v9, sFlow, and IPFIX.
You can edit `$\{OPENNMS_HOME}/etc/telemetryd-configuration.xml` to change the port number and add or remove protocols.

IMPORTANT: Make sure your firewall allow list includes the ports that you have configured to receive flow data.

== Enable individual protocols

Follow these steps to enable one or more protocols that you want to handle individually or that are not included in the default multi-protocol listener.

NOTE: This example uses the NetFlow v5 protocol.
You can follow the same steps for any of the other flow-related protocols.

. Edit `$\{OPENNMS_HOME}/etc/telemetryd-configuration.xml` to enable protocols:
+
[source, xml]
----
<listener name="Netflow-5-UDP-8877" class-name="org.opennms.netmgt.telemetry.listeners.UdpListener" enabled="true">
    <parameter key="port" value="8877"/>

    <parser name="Netflow-5-Parser" class-name="org.opennms.netmgt.telemetry.protocols.netflow.parser.Netflow5UdpParser" queue="Netflow-5" />
</listener>

<queue name="Netflow-5">
    <adapter name="Netflow-5-Adapter" class-name="org.opennms.netmgt.telemetry.protocols.netflow.adapter.netflow5.Netflow5Adapter" enabled="true">
    </adapter>
</queue>
----
+
NOTE: The default configuration file provides example configurations for many protocols.
To enable one of these protocols, find the correct example `listener` and `adapter` elements and change their `enabled` attributes to `true`.

. Reload the daemon to apply your changes:
+
[source, console]
${OPENNMS_HOME}/bin/send-event.pl -p 'daemonName Telemetryd' uei.opennms.org/internal/reloadDaemonConfig

This configuration opens a UDP socket bound to `0.0.0.0:8877` to listen and process NetFlow v5 messages that are are forwarded to this port.

== Enable flows on your devices

Refer to your device's manufacturer documentation to learn more about configuring it to send flows.

NOTE: You may need to set up the flow receiver (in this case, {page-component-title}) and allow sending flows per interface on your firewall.

== Link web UI to OpenNMS plugin for Grafana

You must configure a connection to your instance of the OpenNMS plugin for Grafana to access flow-related graphs from the {page-component-title} web UI:

. Connect to your {page-component-title} Karaf shell:
+
[source, console]
ssh -p 8101 admin@localhost

. Configure the plugin settings:
+
[source, karaf]
----
config:edit org.opennms.netmgt.flows.rest
config:property-set flowGraphUrl 'http://grafana:3000/dashboard/flows?node=$nodeId&interface=$ifIndex' <1>
config:update
----
<1> This URL can also point to other tools.
It supports the `$nodeId`, `$ifIndex`, `$start`, and `$end` placeholders.

After the plugin is configured, an icon is displayed at the top-right corner of an SNMP resource graph indicating that flow data is available for the interface.
If you have trouble during or after configuration, refer to xref:deep-dive/flows/troubleshooting.adoc[].

[[kafka-forwarder-config]]
== Configure Kafka forwarder

Flows enriched with {page-component-title} node data can be forwarded to Kafka and persisted.
By default, enriched flows are stored in the `flowsDocument` topic and the payloads are encoded using https://developers.google.com/protocol-buffers/[Google Protocol Buffers].
See `flowdocument.proto` in the corresponding source definition for the model definitions.

To enable JSON support, set `useJson` to `true`.

Follow these steps to configure forwarding flows to Kafka:

. Enable Kafka forwarding:
+
[source, console]
----
$ ssh -p 8101 admin@localhost
...
admin@opennms()> config:edit org.opennms.features.flows.persistence.elastic
admin@opennms()> config:property-set enableForwarding true
admin@opennms()> config:update
----

. Configure the Kafka server for flows:
+
[source, console]
----
$ ssh -p 8101 admin@localhost
...
admin@opennms()> config:edit org.opennms.features.flows.persistence.kafka
admin@opennms()> config:property-set bootstrap.servers 127.0.0.1:9092
admin@opennms()> config:property-set topic opennms-flows
admin@opennms()> config:update
----

== Next steps

After you set up basic flows monitoring, you may want to do some of the following tasks:

* Classify data flows.
{page-component-title} resolves flows to application names.
You can create rules to override the default classifications (see xref:deep-dive/flows/classification-engine.adoc[]).

* xref:deep-dive/flows/distributed.adoc[Enable remote flows data collection] with Minions.
* xref:deep-dive/flows/sentinel/sentinel.adoc[Scale to manage large volumes of flows data] with Sentinels.
* Add https://github.com/OpenNMS/nephron[OpenNMS Nephron] for aggregation and streaming analytics.
