Create an alerting condition

An alerting condition is a set of checks and limits. ITVA uses it to decide when to open an incident for a device, its interfaces, or a probe. By the end of this page, you will have a condition assigned to your devices and probes. For a description of each check and where its data comes from, see the monitoring and alerting overview.

Before you begin

• You need access to the Settings > Alerting page.
• Most device checks only run when a schedule collects new data. Make sure your devices have a schedule with the tasks in When ITVA runs the checks.

How a condition works

You assign one condition to each device and each probe. Many devices and probes can share the same condition. A device or probe without a condition gets no device or probe checks.

Each check in a condition uses some of these fields:

• Enabled: turns the check on or off.
• Interval: how many of the most recent results ITVA looks at, from 1 to 9.
• Threshold: how many of those results must fail to open an incident, from 1 to 9.
• Limit: a percentage from 0 to 100. A result fails when the value is greater than the limit. A value equal to the limit passes.

For example, with a threshold of 3 and an interval of 5, ITVA opens an incident when 3 of the last 5 results fail. The incident resolves when fewer than 3 of the last 5 results fail.

Caution: keep the threshold less than or equal to the interval. ITVA does not stop you from saving a larger threshold, but the check can then never open an incident.

Checks in a condition

A new condition starts with these values:

Device checks

• Device Unreachable counts health check results. ITVA ignores a health check that the device's vendor adapter skipped.
• Device CPU compares the average utilization across all CPUs in each metrics result with the limit.
• Device Memory compares the memory utilization in each metrics result with the limit.
• Device PSU fails when any power supply on the device is unhealthy. It has no threshold.
• Device Ping Monitor fails when the latest ping of any ping monitor on the device gets no replies or is slower than that monitor's latency threshold.
• Vulnerability Detected fails when the device has one or more open vulnerabilities.

Interface checks

Interface checks only run on interfaces that have alerting turned on. Turn it on from the interface details page. The condition only sets the limits.

Interface Down has no Enabled option. It always runs on interfaces with alerting turned on, even when the device has no condition. Without a condition, ITVA uses a threshold of 2 and an interval of 3.

Interface Bandwidth compares inbound and outbound utilization with the limit. When Use circuit CIR instead of interface speed as bandwidth limit for alerting is selected, ITVA uses the lowest CIR of the active circuits on the interface. If no circuit has a CIR, or the option is cleared, ITVA uses the interface speed. If neither is known, ITVA skips the check for that interface. Interface Bandwidth needs a condition on the device.

Probe check

Probe Unreachable counts the results of the platform's probe health checks. It applies only to probes that have this condition assigned. The probe checks in a condition have no effect on a device.

BGP, OSPF, and IS-IS

Neighbor alerting is not part of a condition. You turn it on for each neighbor, and it runs after each BGP, OSPF, or IS-IS sync, with or without a condition on the device. The condition page lists these neighbors so you can see them in one place.

When ITVA runs the checks

ITVA checks a condition when new data arrives, so the interval is a number of runs, not a time. All device checks run each time a Run Health Check or Sync Device Metrics task finishes for the device:

For example, if a schedule runs Run Health Check every 5 minutes, Device Unreachable at 2 of 3 opens an incident 5 to 10 minutes after the device stops responding. Probe Unreachable at 2 of 3 opens an incident 2 to 4 minutes after the probe stops responding.

Device PSU, Device Ping Monitor, and Vulnerability Detected use the latest data from their own tasks. ITVA only checks that data after a health check or a metrics sync. If a device's schedule runs neither of these, ITVA does not run its device checks.

Create a condition

1. Go to Settings > Alerting.
2. In the Conditions card, select New Condition.
3. Enter a Name, for example "Core routers". Each condition in your organization must have a different name.
4. For each check you want, select Enabled.
5. For each check you turned on, set the Threshold, Interval, and Limit, where the check has them.
6. For each check you do not want, clear Enabled. Device PSU and Device Ping Monitor are on by default.
7. Select Save.

Assign the condition

To one device or probe

1. Open the edit form for the device or the probe.
2. In Alerting Condition, select the condition.
3. Save the form.

To all devices in a folder

This sets the condition on the devices that are in the folder now. It does not change devices in subfolders, and devices that you add to the folder later do not get the condition.

1. Open the menu for the folder and select Manage Devices.
2. In Alerting Condition, select the condition. Leave the other fields empty to keep their current values.
3. Save the changes.

Check that it worked

Open the condition from the Conditions card. The page shows the settings for each check, then these cards:

• Devices: every device that uses the condition.
• Interfaces, and the BGP, IS-IS, and OSPF cards: the items on those devices that have alerting turned on.

If a device is missing from the Devices card, assign the condition to it. If an interface is missing, turn on alerting for it. The page does not list probes.

Change or delete a condition

To change a condition, open it and select Edit Condition. ITVA uses the new values the next time it runs the checks. It also applies them to recent results, so an incident can open or resolve at the next run.

If you delete a condition, the devices and probes that used it have no condition. ITVA stops their device and probe checks, but Interface Down and neighbor alerting continue.

Caution: if you turn off a device check, remove a device's condition, or delete the condition while a device incident is open, the incident stays open. Resolve it yourself. Probe and interface incidents resolve at the next run.

Troubleshooting

A check never opens an incident

• The check is not enabled, or the device has a different condition. Check the Devices card on the condition.
• The threshold is greater than the interval.
• No schedule runs the task that the check needs. See When ITVA runs the checks.
• For an interface check, alerting is off on the interface. For Interface Bandwidth, the interface also needs a circuit with a CIR or a known speed.

An incident opened but nobody was notified

A condition opens incidents, but an alerting rule sends the notifications. Make sure a rule matches the device's tags and the incident's priority.

Interface Down alerts on a device with no condition

This is expected. Interface Down runs on every interface that has alerting turned on. To stop it, turn off alerting on the interface.