Create an alerting rule

An alerting rule decides who hears about an incident, and when. It matches incidents by tag and priority, then sends them to channels through one or more escalation levels. By the end of this page, you will have a rule that covers the devices you choose.

Before you begin

• You need access to the Settings > Alerting page.
• Create at least one channel. A level with no channels sends nothing.
• Tag the devices, folders, probes, or alerting monitors that the rule will cover. Rules find incidents only through tags.
• Give each device an Alerting Condition. Without one, ITVA does not run device checks such as Device Unreachable. Interface Down and neighbor alerting still run, but the rule page lists only devices that have a condition.

How a rule matches an incident

A rule matches an incident when both of these are true:

• At least one of the rule's Tags is also on the item the incident is about.
• The incident's priority is one of the priorities you select under Alerting in the rule.

A rule with no tags or no alerting priorities matches no incidents. The tags that ITVA compares depend on what the incident is about:

Tags on an interface do not affect which rules match. ITVA copies them onto the incident, but it uses the device and folder tags to find rules.

More than one rule can match the same incident. Each matching rule sends notifications and escalates on its own. If two matching rules use the same channel, that channel gets the notification twice.

How ITVA sets the incident priority

ITVA has three alerting priorities:

When ITVA opens an incident, it uses the priority set on the item the incident is about:

• For a device or a probe, the Alerting Priority field on its form.
• For an interface, or an OSPF or IS-IS neighbor, the priority you select when you turn on alerting for it.
• For an alerting monitor, the monitor's priority.

If the item has no priority, the incident gets P1. An interface or neighbor does not use its device's priority. For example, if a device is P0 but an interface on it has no priority, an interface down incident is P1.

How levels and escalation work

A rule has one or more levels. Each level has a list of channels and an Escalate After time from 1 to 12 hours. ITVA runs the levels in order:

1. When the incident opens, ITVA notifies the channels in Level 1.
2. When the Level 1 time ends, ITVA checks the incident. If it is still Open, ITVA notifies the channels in Level 2.
3. ITVA continues in the same way until the last level.

For example, a rule has Level 1 set to email the on-call engineer and escalate after 1 hour, and Level 2 set to post to the managers' Teams channel. If nobody acknowledges the incident within 1 hour, the managers see it in Teams.

Escalation stops when someone acknowledges or resolves the incident. The Escalate After time on the last level has no effect, because there is no next level.

When someone acknowledges, resolves, or comments on the incident, ITVA notifies the Level 1 channels only. The channels in later levels do not get these updates.

Maintenance windows

If a device, interface, or probe is in an active maintenance window when an incident opens, ITVA still records the incident. It does not notify any channels and does not escalate the incident.

Rules for tickets

A rule can also send notifications for tickets. Under Ticketing, select one or more Categories and Priorities. The rule matches a ticket when the ticket's category and its priority are both selected. Tags have no effect on tickets.

When someone submits a matching ticket, ITVA notifies the Level 1 channels. It escalates through the levels in the same way as an incident, until someone is assigned to the ticket.

Create a rule

1. Go to Settings > Alerting.
2. In the Rules card, select New Rule.
3. Enter a Name. Each rule in your organization must have a different name.
4. Optional: enter a Description.
5. In Tags, add the tags of the devices, folders, probes, or alerting monitors that the rule covers.
6. Under Alerting, select the incident priorities that the rule handles.
7. Optional: under Ticketing, select ticket categories and priorities.
8. Under Levels, in Level 1, select the Channels to notify first, and an Escalate After time. The default time is 1 hour.
9. Optional: to add a level, select the add button under the last level. Then select its channels and time. Use the up and down arrows on a level to change its position.
10. Select Save.

Check that it worked

Open the rule from the Rules card. The rule page shows what the rule covers:

• The Devices card lists devices whose tags or folder tags match the rule and that have an alerting condition. A matching device without a condition can still send interface down and neighbor incidents through the rule.
• The Interfaces, BGP, IS-IS, and OSPF cards list the items on those devices that have alerting turned on.

If a device you expect is missing, check its tags, its folder's tags, and its alerting condition. The rule page does not list probes or alerting monitors. To test each channel in the rule, open the channel and select Test Channel.

Troubleshooting

An incident opened but nobody was notified

Check these causes in order:

• No rule has a tag that is on the device, its folder, the probe, or the monitor.
• The rule does not include the incident's priority. Check the priority on the incident, then the priorities on the rule.
• Level 1 of the rule has no channels.
• The item was in a maintenance window when the incident opened.
• You created or changed the rule after the incident opened. ITVA sends the first notification and starts escalation only for the rules that match when the incident opens.

A channel gets the same notification twice

Two or more matching rules use the channel. Open the channel: the Rules field lists every rule that uses it. Remove the channel from one of the rules, or change the rules' tags or priorities so that only one matches.

The incident did not escalate

If someone acknowledged or resolved the incident before the Escalate After time ended, ITVA does not escalate it. If the rule has only one level, there is nothing to escalate to.

Interface tags have no effect

Rules use the tags on the device and its folder for interface incidents. To send one interface's incidents to a different rule, give the interface its own priority, and select that priority in the other rule.