spacetab

OpsHorn: incidents and on-call

Prototype, 2020

Our own incident management service: it takes alerts from monitoring, groups and adjusts them by rules and notifies whoever is on call, by schedule and with escalation. Worked out as a clickable prototype — a web console and a mobile app.

Client and task

OpsHorn is the studio's own product: an incident and on-call management service in the spirit of PagerDuty and Opsgenie. It sits between monitoring systems and the team and decides whom to call, how and when, once something breaks.

The service was to collect alerts from Prometheus, Sentry, Amazon CloudWatch and other sources into incidents, group and adjust them by rules and get them to the right person — by email, text, phone call, Telegram, push or the team chat — following the on-call schedule and escalating if nobody responds. We planned to bring OpsHorn to market as a subscription and worked the whole product out, pricing and billing included.

What we worked out

133 prototype pages: the project web console, the personal profile and the mobile app.

Current incidents: groups, an incident card with labels, Acknowledged and Resolved, notifications by escalation level

Incidents: groups and the incident card

Current incidents are gathered into groups, and a whole group can be acknowledged at once. The card shows every alert field (labels, annotations, severity, status), Acknowledged and Resolved buttons, who was notified at each escalation level and when, the next notification and who responded. An explanation shows which integration the incident came through and which rules were applied. Search and an archive filtered by severity, integration and period.

The on-call engineer sees what happened, who has been called and why.

Notification rule: a filter of nested AND/OR groups with regexp, on-call schedules to notify

Notification rules

An incident filter is built from conditions and nested AND/OR groups: integration, name, label, tag, severity, description, a service-specific field — matched by equals, not equals or a regular expression. A rule sends incidents to on-call schedules, the filter is tested on past incidents, and whatever no rule catches goes to the default rule.

An incident reaches the right team instead of waking everyone.

Grouping rule: a chain of group naming rules from the alert name and annotations

Grouping and modification

Grouping joins incidents of one service type and names the group by a chain of rules from the alert name and annotations. Modification changes severity, the alert name and annotations or adds new ones, including from other fields' values; the result is tested on the last 20 incidents.

Similar alerts come together in one group with a clear name.

On-call schedule: a month, two escalation levels, people, channels and groups, overrides on top

On-call schedules

A month's schedule with escalation levels: if the first level doesn't respond in 15 minutes, the notification goes to the second. Recipients are people, channels and groups with schedule templates, and a schedule has its own time zones. Time is set by selecting it on the scale, one-off overrides go on top of the schedule, and a search finds gaps in coverage. Repeated notifications with a cap, and a schedule copy in one click.

At any moment it is clear who is on call and who will be called next.

Prometheus inbound integration: key, endpoint, watchdog, setup guide

Integrations and channels

Alert sources are Prometheus, Sentry, Amazon CloudWatch and a generic integration, each with its own key and endpoint, a watchdog and setup instructions. Team channels are Telegram, Slack and Microsoft Teams.

Connecting monitoring means copying a key and an address.

Profile: contacts, connected devices, personal notification settings by channel and project

Personal notification settings

Everyone decides how to be reached: email, text, call, Telegram and push — always, never or at set hours, with separate settings per project. Email and message templates for each service type with a preview on a real incident; a phone with the app connects by QR code.

A notification arrives where and when the person will notice it.

Project team: roles, invitations, paid calls and texts per memberProject log: who changed what, links to previous versions

Team, plan and log

Project members with roles, invitations and groups; for each member you see whether paid calls and texts are on. Plans: free with limits on projects and people, Premium, and enterprise with SSO and a custom domain; call and text packages by country, usage statistics, the card and invoice recipients. The project log shows who changed what, with a link to the previous version.

The project's team, spending and changes stay under the owner's control.

App, project incidents: groups, current and archiveApp, incident card: labels, Acknowledged and Resolved

Mobile app

Projects with incident counts by severity, current incidents in groups, archive and search, an incident card with Acknowledged and Resolved, and a push notification about an incident.

You can respond to an incident from your phone without opening a laptop.

1 / 8

What it gave

  • The whole incident path on screens

    From a monitoring alert to acknowledgement by the on-call engineer and the archive — with grouping, adjustments, escalation and an explanation of which rules fired.

  • A product with its business model

    Plans, free-plan limits, call and text packages and roles for accounting were worked out together with the features.

  • Fewer needless wake-ups

    Rules, schedules and personal quiet hours decide whom to call and when.

  • Ready to estimate and build

    Screens come with empty lists, errors and variants, with notes on behaviour.

Technologies

Status

The project stopped at the prototype stage in 2020.

Have a similar task?

Tell us about it: we reply within one business day and suggest where to start.

Telegram: @spacetab_bot · Email: bip@spacetab.io