= Tuning ActiveMQ
:description: Learn about default ActiveMQ settings in OpenNMS {page-component-title} and how to change them for better performance.

== Memory limits

The settings for the embedded ActiveMQ broker are found in `$\{OPENNMS_HOME}/etc/opennms-activemq.xml`.
Memory and storage limits are conservative by default and should be tuned to accommodate your workload.
Consider increasing the `memoryUsage` (defaults to 20MB) to `512MB` or greater, assuming you have enough heap available.

[source, xml]
----
<systemUsage>
    <systemUsage>
        <memoryUsage>
            <memoryUsage limit="20 mb"/> <1>
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="1 gb"/> <2>
        </storeUsage>
        <tempUsage>
            <tempUsage limit="100 mb"/> <3>
        </tempUsage>
    </systemUsage>
</systemUsage>
----
<1> The amount of JVM heap memory assigned to the broker to track of destinations, cache messages, etc; `memoryUsage` limit can be increased to `512MB` or more.
<2> The limit of disk storage in `$\{OPENNMS_HOME}/data/` that can be used to store persistent messages; `storeUsage` limit can be increased to `5GB` or more.
<3> The limit of disk storage in `$\{OPENNMS_HOME}/data/` that can be used to store non-persistent messages; `tempUsage` limit can be increased to `1GB` or more.

TIP: If the memory limit is reached, https://activemq.apache.org/producer-flow-control.html[flow control] will prevent messages
from being published to the broker.

== Per destination limits
ActiveMQ provides mechanism to tune memory usage per destination. The default limits are conservative and should be tuned to accommodate your workload.

[source, xml]
----
<policyEntry queue=">" <1>
             producerFlowControl="true"
             memoryLimit="1mb"> <2>
   <deadLetterStrategy>
     <discarding/>
   </deadLetterStrategy>
 </policyEntry>
----
<1> This matches all queues
<2> This is the `memoryLimit` per queue and should be increased.

The `policyEntry/memoryLimit` value prevents any one queue from consuming all broker resources, but should be tuned such that the per-queue limit is a reasonable percentage of the total `systemUsage/memoryUsage` limit.

== KahaDB persistence
KahaDB is a file based persistence database that is local to the ActiveMQ broker.
It has been optimized for fast persistence and is the the default broker storage mechanism.
ActiveMQ spools messages to disk in order to prevent broker running out of memory or to fulfill persistence requirements.

You can optimize the performance of the KahaDB message store by modifying the following attributes:

[source, xml]
----
<persistenceAdapter>
    <kahaDB directory="${activemq.data}/kahadb"
            journalMaxFileLength="128mb" <1>
            journalDiskSyncStrategy="never" <2>
            indexWriteBatchSize="10000" <3>
     />
</persistenceAdapter>
----
<1> Because there is a cost associated with closing a full journal file and opening a new journal file, you can get a slight performance improvement by increasing the journal file size (Default `32MB`)
<2> By default the broker performs an expensive disk sync before sending an acknowledgement back to a producer.
A substantial improvement in broker performance can be achieved by disabling these syncs. (Default: `true`)
<3> Defines the threshold for the number of dirty indexes that are allowed to accumulate, before KahaDB writes the cache to the store.
This consolidates more small journal writes into a single larger write operation. (Default: 1000)

== Monitoring the ActiveMQ broker using the Karaf shell

Use the `opennms:activemq-stats` command available via the Karaf shell to show statistics about the embedded broker:

[source, karaf]
----
opennms:activemq-stats
----

TIP: If the command is not available, try installing the feature using `feature:install opennms-activemq-shell`

This command reports some high level broker statistics as well as message, enqueue and dequeue counts for the available queues.
Pay close attention to the memory usage that is reported.
If the usage is high, use the queue statistics to help isolate which queue is consuming most of the memory.

The `opennms:activemq-purge-queue` command can be used to delete all of the available messages in a particular queue:

[source, karaf]
----
opennms:activemq-purge-queue OpenNMS.Sink.Trap
----

== Authentication and authorization with ActiveMQ

The embedded ActiveMQ broker is preconfigured to authenticate clients using the same authentication mechanisms (JAAS) as the {page-component-title} web application.

Users associated with the `ADMIN` role can read, write or create any queue or topic.

Users associated with the `MINION` role are restricted in such a way that prevents them from making RPC requests to other locations, but can otherwise read or write to the queues they need.

See the `authorizationPlugin` section in `$\{OPENNMS_HOME}/etc/opennms-activemq.xml` for details.

== Multi-tenancy with {page-component-title} and ActiveMQ

The queue names {page-component-title} uses are prefixed with a constant value.
If many {page-component-title} are configured to use the same broker, then these queues would end up being shared among the instances, which is not desired.
In order to isolate multiple instances on the same broker, you can customize the prefix by setting the value of the `org.opennms.instance.id` system property to something that is unique per instance.

.`$\{OPENNMS_HOME}/etc/opennms.properties.d/instance-id.properties`
[source, properties]
----
org.opennms.instance.id=MyNMS
----

Update the Minion's instance ID accordingly to match the {page-component-title} instance.

.`$\{MINION_HOME}/etc/custom.system.properties`.
[source, properties]
----
org.opennms.instance.id=MyNMS
----

CAUTION: If you change the instance ID setting when using the embedded broker, you will need to update the authorization section in the broker's configuration to reflect the updated prefix.
Modify the configuration with the file `$\{OPENNMS_HOME}/etc/opennms-activemq.xml`:

[source, xml]
----
<authorizationPlugin>
  <map>
    <authorizationMap>
      <authorizationEntries>
        <!-- Users in the admin role can read/write/create any queue/topic -->
        <authorizationEntry queue=">" read="admin" write="admin" admin="admin" />
        <authorizationEntry topic=">" read="admin" write="admin" admin="admin"/>
        <!-- Users in the minion role can write/create queues that are not keyed by location -->
        <authorizationEntry queue="<YOUR INSTANCE ID>.*.*" write="minion" admin="minion" />
        <!-- Users in the minion role can read/create from queues that are keyed by location -->
        <authorizationEntry queue="<YOUR INSTANCE ID>.*.*.*" read="minion" admin="minion" />
        <!-- Users in the minion role can read/write/create advisory topics -->
        <authorizationEntry topic="ActiveMQ.Advisory.>" read="minion" write="minion" admin="minion"/>
      </authorizationEntries>
      <!-- Allow all users to read/write/create temporary destinations (by omitting a <tempDestinationAuthorizationEntry>) -->
    </authorizationMap>
  </map>
</authorizationPlugin>
----

== Tuning the RPC client in OpenNMS

The following system properties can be used to tune the thread pool used to issue RPCs:

[options="header, %autowidth", cols="2,1,3"]
|===
| Name
| Default
| Description

| org.opennms.ipc.rpc.threads
| 10
| Number of threads which are always active.

| org.opennms.ipc.rpc.threads.max
| 20
| Maximum number of threads which can be active.
These will exit after remaining unused for some period of time.

| org.opennms.ipc.rpc.queue.max
| 1000
| Maximum number of requests to queue.
Set to `-1` to be unlimited.

| org.opennms.activemq.client.max-connections
| 8
| The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.

| org.opennms.activemq.client.concurrent-consumers
| 10
| The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

|===

TIP: Use the `opennms:stress-rpc` Karaf shell command on the core to help evaluate and tune performance.

== Tuning the RPC client in Minion

The following properties can be added to `org.opennms.minion.controller.cfg` to tune the thread pool used to issue RPCs:

[options="header, %autowidth", cols="2,1,3"]
|===
| Name
| Default
| Description

| broker-max-connections
| 8
| The maximum number of concurrent connections to the ActiveMQ broker; larger numbers allow for increased concurrency and throughput.

| broker-concurrent-consumers
| 10
| The maximum number of concurrent ActiveMQ consumers to spawn; larger numbers can increase concurrency and throughput.

|===
TIP: Use the `opennms:stress-rpc` Karaf shell command on the core to help evaluate and tune performance.

== Troubleshooting RPC failures

Symptoms of RPC failures may include missed polls, missed data collection attempts and the inability to provision or re-scan existing nodes.
For these reasons, it is important to ensure that RPC related communication with Minion at the various monitoring locations remains healthy.

If you want to verify that a specific location is operating correctly make sure that:

1. Nodes exist and were automatically provisioned for all of the Minions at the location
2. The `Minion-Heartbeat`, `Minion-RPC` and `JMX-Minion` services are online for one or more Minions at the location
3. Response time graphs for the `Minion-RPC` service are populated and contain reasonable values
  * These response time graphs can be found under the `127.0.0.1` response time resource on the Minion node
  * Values should typically be under 100ms but may vary based on network latency
4. Resource graphs for the `JMX-Minion` service are populated and reasonable values

To interactively test RPC communication with a remote location use the `opennms:poll` command from the Karaf shell:

[source, karaf]
----
opennms:poll -l LOCATION -c org.opennms.netmgt.poller.monitors.IcmpMonitor 127.0.0.1 <1>
----
<1> Replace `LOCATION` in the command above with the name of the monitoring location you want to test.
