5 Steps to Identify and Address Incident Response Gaps
Our modern world is built on software, but it took a major, global outage on July 19 for many to realize just how major the blast radius can be when things go wrong. Described as the biggest IT outage in history, the incident stemmed from one botched software update. It was enough to ground planes, disrupt emergency services and public transport and even take a television broadcaster offline. The estimated direct financial loss to the U.S. 500 Fortune companies: $5.4 billion.
The lesson is simple: Software isn’t perfect, and as long as tech complexity keeps growing, incidents will keep happening. According to a recent survey, 59% of IT leaders say incidents affecting customers have increased over the past 12 months, growing by an average of 43%.
To better manage risk, organizations need to build more resilient operations and teams. That is the key to recovering from disruptions faster, protecting customer experience and revenue.
Time to Take Five
The July 19 global IT outage is an extreme example of what can happen when digital systems fail. But operational excellence starts with efficiently managing common, day-to-day service disruption with a proactive, not reactive, incident management approach.
Here are some strategies that will help organizations be better prepared for outages:
1. Eliminate Guesswork From Managing Incidents
Defining clear roles (incident commander, scribe, liaison, etc.) and tasks within a response team is a crucial first step to address gaps in accountability and coordination from the get-go. It ensures that everyone knows exactly what their job is and that no critical steps are missed during response.
Teams can also orchestrate the entire incident by looping in those subject matter experts (SMEs) into automated incident workflows built to orchestrate the right response according to the incident’s priority and type. Once again, this will ensure that all critical tasks are performed and no time is spent figuring out what to do or when to accelerate resolution.
2. Drive From Signal to Action
To compress the time it takes to address an incident, it’s not enough to stick to the traditional eyes-on-glass model that network operations centers (NOCs) traditionally privilege. It’s too human-intensive and error-prone to effectively triage an increasingly overwhelming volume of data.
To go from event to resolution with minimal toil and increased speed, teams can leverage AI and automation to deflect noise, surface only the most critical alerts and automate diagnostics and remediations. Generative AI can amplify that effect: For teams collaborating in ChatOps tools, common diagnostic questions can be used as prompts to get context and accelerate action. This lightens the mental load on responders, allowing them to focus on higher-value activities while outsourcing drafting and knowledge-gathering tasks to AI.
3. Deploy GenAI and Automation Throughout the Incident Management Process
When an incident hits, teams spend too much time gathering information and looping in numerous people to tackle it. Generative AI can be used to quickly summarize key data about the incident and provide actionable insights at every step of the incident life cycle.
It can also supercharge the ability to develop and deploy automation jobs faster, even by non-technical teams: Operators can translate conversational prompts into proposed runbook automation or leverage pre-engineered prompts based on common categories.
As always, humans should be kept in the loop to revise and edit the AI-generated prompt, and only then initiate the new working job definition.
4. Proactively Communicate With Stakeholders
Customers hate being kept in the dark. Negative feedback can ramp up the stress levels for teams and hurt the brand’s reputation and trust. To sustain seamless customer experiences, it’s vital to bridge any gaps between support and engineering teams.
Updating internal stakeholders with regular status updates is an industry best practice that ensures all teams have the support they need to resolve faster. To keep customers in the loop, status pages are a great way to provide visual communication into the real-time status of the organization’s operations.
5. Conduct Post-Incident Reviews to Drive Continuous Improvement
Organizations with a mature approach to digital operations and incident response will always look to learn from previous incidents. It’s the best way to keep building resilience and improve processes for the next time.
Ensure the organization has a thorough review process in place and a system to implement changes to prevent similar incidents in the future. The Howie guide is a great place to start as it details all the major stages a post-incident review should run through, from assigning an investigation to distributing its findings.
Plan for the Inevitable
Incidents are inevitable, but serious financial and reputational damage, customer churn and employee burnout don’t have to be the result. Rigorous planning and post-incident learning are the best ways to support business continuity to ensure organizations can respond faster and work smarter, helping to keep customers happy.
