[Deep Dive] Approaches to "Root Cause Analysis" - A Toolbox for Thinking and Analysis

Introduction

We face various "problems" in our daily lives and work.
For example, an IT system suddenly stops working, product sales are sluggish, or communication within a team is not going well.
How should we deal with these problems?

In many cases, we tend to try to remove only the visible phenomena (symptoms). However, that does not lead to a fundamental solution. Even if the symptoms disappear temporarily, the same problem will likely occur again.

This is where "root cause analysis" becomes important.
This is an approach to identify and remove the true root of a problem, that is, the "cause."

In this article, we will take a deep dive into "root cause analysis."

1. What is "Root Cause Analysis"?

1.1 Definition and Content of "Root Cause Analysis"

"Root cause analysis" is "the activity of determining the fundamental reason (cause) why a problem is occurring."
It can be said that it is a task of digging deep into not just "what happened," but "why it happened."

For example, suppose a company printer breaks down.
In this case, the fact that it broke down is the "problem."

In root cause analysis, we think about, "Why did the printer break down?"

Was it paper jams occurring frequently,
Was it out of ink,
Were the parts worn out,
Or was it being used incorrectly,
We explore possible causes one by one.

1.2 The Process of "Problem Analysis" and the Positioning of "Root Cause Analysis"

In the worlds of business and IT, it is common to systematically proceed with a process to solve problems.
This is called "problem analysis."

The general process of "problem analysis" is as follows.

(1) Identification and Definition of the Problem
Clarify what the problem is.

(2) Information Gathering
Collect information related to the problem.

(3) Root Cause Analysis ←This is the theme of this article!
Based on the collected information, explore the root cause of the problem.

(4) Planning Countermeasures
Think of concrete measures to remove the cause.

(5) Execution of Countermeasures
Execute the measures.

(6) Verification of Effectiveness
Check if the measures were effective.

(7) Standardization and Recurrence Prevention
Establish solutions and prevent recurrence.

In this process, "root cause analysis" is the core of problem-solving.
If you cannot identify the cause, you cannot develop effective countermeasures.

1.3 Purpose of Root Cause Analysis

The main purposes of root cause analysis are the following three:

(1) Fundamental Problem Solving
Instead of superficial treatment, remove the true cause so that the same problem never happens again.

(2) Recurrence Prevention
By identifying the cause and taking measures against it, you prevent similar problems from occurring in the future.

(3) Discovery of Areas for Improvement
In the process of exploring the cause, you may discover previously unnoticed issues in business processes or systems, or areas that need improvement.

1.4 Situations Requiring Root Cause Analysis and Concrete Examples

Root cause analysis is useful in various situations such as the following:

[IT Field]

(1) System Failure
"The EC site went down"
-> Why did it go down?
- Server overload
- Program bug
- External attack, etc.

(2) Performance Degradation
"The app is running slowly"
-> Why is it slow?
- Heavy database processing
- Insufficient network bandwidth
- Insufficient memory, etc.

[Business Field]

(1) Sales Slump
"Sales of a specific product are lower than last year"
-> Why are they lower?
- Rise of competing products
- Insufficient promotion
- Quality issues
- Economic downturn, etc.

(2) Increase in Customer Complaints
"There are many inquiries about the product"
-> Why are there many?
- The product is difficult to use
- The manual is unhelpful
- Many initial defects, etc.

(3) Increase in Employee Turnover Rate
"Employees are not staying"
-> Why are they not staying?
- Salary structure
- Interpersonal relationships
- Job content
- Evaluation system, etc.

(4) Project Delays
"The development project is behind schedule"
-> Why is it delayed?
- Incomplete requirements definition
- Lack of skills
- Lack of communication, etc.

1.5 What is a "Cause" in the First Place?

A "cause" is the direct or indirect reason or root that triggered a certain phenomenon (problem).

For example, suppose there is a problem of "feeling unwell".

- Direct causes
- Caught a cold
- Lack of sleep
- Overeating

- Indirect causes
- Work stress
- Irregular lifestyle habits
- Imbalanced nutrition

As shown, there are many layers to a "cause," and a fundamental cause is often hidden deep behind the superficial one.
In root cause analysis, it is important to uncover this "fundamental cause."

2. How to Conduct Root Cause Analysis

2.1 How to Proceed with Analysis

It is recommended to proceed with root cause analysis in the following steps.

(1) Clarifying the Problem

This is the step where you specifically define what the problem is.
It is important to grasp the problem as an objective fact.
Also, organize information from the perspective of the "5W1H" (When, Where, Who, What, How).

Clarifying the problem is the starting point of root cause analysis and one of the most important steps.
If the problem is ambiguous here, subsequent information gathering and hypothesis setting may be off the mark.
You are required to exclude subjectivity and emotions, and describe the problem based on quantitative data and specific events.

[Specific Example]

[Bad Example]
"Recently, there have been many complaints from customers." (Ambiguous and subjective)

[Good Example:]
"Since last week, on the payment page of the EC site, there have been 50 cases per day where users enter their credit card information, an error message is displayed, and the purchase is not completed. This problem is particularly noticeable with access from PCs, and does not occur with access from smartphones."
- When: Since last week
- Where: EC site payment page
- Who: Users
- What: Purchase not completed due to error after entering credit card information
- How: Error message is displayed

(2) Information Gathering and Organization

This is the step where you gather and organize all information related to the problem from multiple angles.
This includes data, logs, interviews with stakeholders, and past cases.
Organize the gathered information chronologically or by category to visualize it.

"All information" includes not only things that seem directly related to the problem, but also information that may seem unrelated at first glance.
This is because causes can sometimes lurk in unexpected places.
Visualizing information helps to grasp the complex situation as a whole and discover patterns or trends.

[Specific Example]

Let's consider the aforementioned EC site payment error case.

[Information Gathering]
- Error logs: Date and time of error, error code, message content
- Server logs: Access status of the payment page, server resource usage rate
- Network logs: User access source, communication status
- Database logs: Recording status of payment-related data
- User interviews: What operations were performed, what error message was displayed
- Developer interviews: Payment system architecture, recent change history
- Past similar cases: Whether similar payment errors have occurred in the past
- External service information: Failure information of the credit card payment agency service being used

[Information Organization and Visualization]
- Graph the frequency of error occurrences by time of day and day of the week
- Summarize the types and counts of error codes in a table
- Show the user's operation flow in a sequence diagram
- Show server resource usage rates in a time-series graph

(3) Setting Hypotheses

This is the step where you formulate several hypotheses based on the information gathered, asking, "Could this be the cause?"
At this stage, do not jump to conclusions; instead, think in terms of "it might be..."

Hypotheses should be derived logically based on information.
Rather than relying solely on intuition or speculation, construct them by identifying correlations or patterns from the gathered information that suggest a potential cause.
By formulating multiple hypotheses, you broaden your perspective and reduce the risk of overlooking the true cause.

[Concrete Example]

Based on the results of information gathering regarding a payment error on an e-commerce site, we formulate the following hypotheses.

・Hypothesis 1: A bug may have been introduced into the payment processing logic due to a recent system update.

・Hypothesis 2: There may be a temporary communication failure in the integration with the credit card payment gateway service.

・Hypothesis 3: Access may be concentrated during specific time periods, causing server resource shortages and stalling processing.

・Hypothesis 4: Database deadlocks may be preventing payment data from being written.

(4) Hypothesis Verification

This is the step where you confirm whether the formulated hypotheses are correct through more detailed information gathering, experiments, or testing.
Repeatedly ask questions such as, "Is that really the cause?" and "Would the problem not occur if that cause were absent?" to dig deeper.

Hypothesis verification is not just a confirmation task; it is a process of gathering evidence to support the hypothesis or searching for counter-evidence.
Experiments and tests are effective for confirming reproducibility or testing whether the problem is resolved when a specific factor is removed.

[Concrete Example]

We verify the aforementioned hypotheses.

・Verification of Hypothesis 1:
　・Review the payment-related code changed in the system update to check for logic errors.
　・Perform comparative testing of the payment flow between the pre-update version and the current version in a test environment to see if the error is reproducible.

・Verification of Hypothesis 2:
　・Check for service outage information from the credit card payment gateway.
　・Analyze API communication logs with the payment gateway in detail to check for error codes and response times.
　・If possible, try using a different payment gateway temporarily to see if the problem is resolved.

・Verification of Hypothesis 3:
　・Monitor server resource usage (CPU, memory, network bandwidth) in detail during the time periods when errors occur.
　・Use load testing tools to apply access loads similar to those during the specific time periods to see if the error is reproducible.

・Verification of Hypothesis 4:
　・Check database deadlock logs and lock information.
　・Monitor the database lock status in real-time during payment processing.

(5) Identifying the Root Cause

This is the step where you verify multiple hypotheses to finally identify the true "root cause" that is triggering the problem.
Sometimes multiple causes are intertwined in a single problem, so you must prioritize them when considering countermeasures.

The root cause is the deepest cause, such that if it is removed, the problem will not recur.
Techniques like the "5 Whys," which repeatedly ask "Why did that happen?" rather than just looking at superficial causes, are effective.
When multiple causes are complexly intertwined, evaluate the impact of each and prioritize them to implement the most effective countermeasures.

[Concrete Example]

As a result of the investigation, the following facts were discovered.

・A minor bug was found in the payment logic during a system update, but it was not the direct cause of the error.
・No outages occurred on the payment gateway service side.
・It was discovered that access spikes during a specific time period (every day from 9:00 PM to 10:00 PM) caused a concentration of database write operations, leading to temporary deadlocks that prevented payment processing from completing. This was the cause of the error message.
・This deadlock occurred because the database settings were not optimized for specific write patterns.

[Identifying the Root Cause]

The root cause of the EC site's payment error was identified as "inadequate database locking mechanism settings that could not handle access concentration during peak times."

[Considering Countermeasures]
・Optimize the database locking mechanism settings.
・Consider transitioning payment processing to batch processing or asynchronous processing.
・Consider scaling up or scaling out the server and database.

2.2 What to do in analysis

(1) Think based on objective facts

It is important to make judgments based on data and evidence rather than emotions or speculation.

Because humans are emotional beings, when a problem occurs, we tend to unconsciously blame someone or jump to conclusions like "it must be this."
However, trying to identify the cause based on emotions or speculation will lead you in the wrong direction, wasting time and effort, and causing you to miss the true cause.
Developing the habit of relying only on objective evidence, such as data, logs, and concrete events, is necessary for accurate root cause analysis.

[Concrete Example]

Instead of judging by speculation that "the system must have stopped because that new employee made an operation error," think based on objective facts like logs: "Looking at the system logs, a database connection error was recorded immediately after the new employee performed the operation. This suggests that the new employee's operation was not the direct cause, but rather that there was a problem with the database process that triggered the operation."

(2) Have a multi-faceted perspective

By incorporating diverse opinions from team members or people from different departments instead of thinking alone, you may notice causes you had overlooked.

No matter how experienced a person is, there is a limit to what one person can think of alone. Problems have various aspects, and by having people with different expertise share opinions from different perspectives, more comprehensive root cause investigation becomes possible.

For example, by having developers, operations staff, sales staff, and customer support staff provide information and point out issues from their respective positions, you can discover connections or blind spots that you would not have noticed alone.

[Concrete Example]

When a defect occurs in a product, instead of just the development team investigating the cause, by gathering information such as "the supplier for a specific part recently changed" from the production line staff, "minor issues had been pointed out in quality inspections for this part for some time" from the quality control department, and "multiple similar defect reports have been received from specific customers" from the sales department, you can arrive at the possibility that the change in the parts supplier is the root cause.

(3) Repeat "Why?"

Even if one cause is found, it is important to dig deeper by asking "Why did that happen?" and continue questioning until you reach the root cause. This is called the "5 Whys" or similar.

If you only address superficial causes, the problem will recur unless the underlying cause is resolved.
By repeatedly asking "Why?", you can unravel the chain of problems and eventually reach the true root cause. This is one of the most powerful tools in problem-solving.

[Concrete Example]

"Why is the website loading slowly?"

QA-1. Why is it slow? → Because the server response is slow.

QA-2. Why is the server response slow? → Because the database query is slow.

QA-3. Why is the database query slow? → Because a specific table lacks an index.

QA-4. Why is there no index? → Because performance testing was insufficient during development.

QA-5. Why was performance testing insufficient? → Because the development schedule was tight, and it was omitted from the test items.

In this case, the root cause can be identified as "inadequate development schedule management and a lax testing process." This leads to fundamental measures, such as reviewing future development processes, rather than simply adding an index.

(4) Visualize information

By organizing information using diagrams, graphs, and tables, complex relationships become easier to understand.

Humans are strongly influenced by visual information. Patterns and trends that might be overlooked in large amounts of text data or lists of numbers can be grasped at a glance when turned into diagrams or graphs. Especially in complex problems where multiple elements are intertwined, visually representing the relationships between information helps facilitate discussion and build a common understanding.

[Concrete Examples]

・Showing the trend of error rates in a line graph reveals a sharp increase during a specific period.

・Showing data flow between systems in a flowchart makes it clear at a glance where data bottlenecks are occurring.

・Showing the time taken for each process in a Gantt chart clarifies which parts are delayed.

・Organizing opinions among stakeholders in a mind map makes it easier to grasp different perspectives and potential connections.

2.3 Points to note in analysis

(1) Do not turn it into a blame game

Root cause analysis is not about blaming someone. Please remember that the goal is to prevent the recurrence of problems and to make improvements.

When a problem occurs, it is easy to fall into the trap of hunting for the culprit, asking "who made the mistake?"
However, hunting for culprits lowers team morale, leads to the concealment of information, and hinders the investigation of the true cause.
The goal is to solve the problem and prevent recurrence, not to identify and blame individuals.
You should focus on fundamental factors such as systems, processes, and the environment, rather than on individuals.

[Concrete Example]

When a system failure occurs, instead of blaming Person A by saying, "It's because Person A made a configuration error!", you should analyze it from the perspective of, "Were there process issues behind Person A's error, such as unclear configuration procedures, a lack of a checking system, or insufficient information during an emergency?"

(2) Do not jump to conclusions

Be careful not to jump at superficial causes and try to implement solutions immediately.

In the rush to solve a problem, one might only address the visible, superficial causes. However, this does not solve the fundamental problem, and the likelihood of the same or similar problems recurring increases.
It is important to have the patience to dig deep until you reach the root cause and to follow the appropriate process to do so.

[Concrete Example]

When inquiries from customers increase, if you immediately conclude that "it's because the staff's response is slow" and simply increase the number of staff, the number of inquiries might decrease temporarily. However, because the root cause of the increase in inquiries themselves (e.g., insufficient product manuals, confusing FAQs on the website, etc.) has not been solved, the inquiries may increase again.

(3) Do not judge based on insufficient information

Trying to identify a cause based only on uncertain or fragmented information can lead to incorrect conclusions.

It is dangerous to reach a conclusion at a stage where information has not been sufficiently gathered.
Judging based on fragmented information or unverified rumors can lead to misguided countermeasures, which not only delays problem-solving but can also create new problems.
You should always be conscious of the comprehensiveness and accuracy of information, and collect additional information if necessary.

[Concrete Example]

When receiving a report of a system failure, you decide to "increase the database specs" based only on fragmented information from an engineer (e.g., "It seems the database crashed"). However, a subsequent detailed investigation reveals that there was actually a problem with the database server's network settings, causing intermittent disconnections, resulting in the spec upgrade being a wasted investment.

(4) Do not try to identify every single cause

Time and resources are limited. It is realistic to focus on root causes that have a high impact and are actionable.

Ideally, you would want to identify all causes and derive a perfect solution, but in reality, resources such as time, budget, and personnel are limited.
Pursuing every possibility is inefficient and can hinder rapid problem-solving. A realistic approach is to focus on the root causes that have the greatest impact or those that can be addressed with relatively little effort, and to prioritize them.

[Concrete Example]

Suppose multiple causes for a decrease in a web service's response speed have been discovered:
- Database queries are not optimized
- Image file sizes are large
- JavaScript file loading order is not appropriate
- Response from some external APIs is slow
- Server hardware is aging

If you determine that "database queries are not optimized" has the greatest impact on performance and can be improved in a relatively short period, you prioritize that by focusing on those countermeasures first and considering other causes in the next phase.

3. Concrete examples and usage of methods/tools for root cause analysis

There are various methods and tools for root cause analysis. Here, I will introduce some of the most representative ones.

3.1 5 Whys

[Overview]
A method used in the Toyota Production System, this is a simple way to dig down to the root cause of a problem by repeatedly asking "Why?"

[How to use]
(1) First, define the problem.

(2) Ask, "Why did that happen?" regarding the problem.

(3) Ask, "Why did that happen?" again regarding the answer.

(4) Repeat this about five times (or until you feel you have reached the root cause).

[Example]
Problem: "System failures occur frequently"

(1) Why do failures occur frequently?
→ Because the server load is high.

(2) Why is the server load high?
→ Because there are many simultaneous accesses.

(3) Why are there many simultaneous accesses?
→ Because we strengthened site promotion.

(4) Why did we strengthen site promotion?
→ Because we started a new campaign.

(5) Why did we start a new campaign?
→ To achieve sales targets.

In this example, the root cause appears to be a "lack of consideration for system load" accompanying the "start of a campaign to achieve sales targets."

3.2 Characteristic Factor Diagram (Fishbone Diagram, Ishikawa Diagram)

[Overview]
Also called a "Fishbone Diagram" because of its shape, which resembles fish bones. It is a tool for systematically organizing and visualizing what factors (causes) are influencing a problem (result).

[How to use]
(1) Write the "problem (result)" at the far right of the diagram.

(2) Set major categories related to the problem as the large bones (main causes), such as "People," "Materials," "Methods," "Equipment," "Environment," and "Measurement."

(3) Write more specific factors (smaller bones) into each main cause.

(4) For each factor, dig deeper by asking, "Why is that a cause?" and add even more detailed factors (branch bones).

[Example]
For the problem of "many defective products," we dig into the causes from perspectives such as people, materials, methods, and equipment.

3.3 Pareto Chart

[Overview]
This is a graph used to identify the most impactful causes among those contributing to a problem.
It combines a bar chart and a line graph, and is based on the "80:20 rule (Pareto Principle)." This is an empirical rule stating that "80% of problems arise from 20% of the causes."

[How to use]
(1) Categorize the occurring problems (types of defects, complaint details, etc.) by cause, and tally the number of occurrences or loss amounts for each.

(2) Sort the aggregated data in descending order of the number of occurrences.

(3) Represent the number of occurrences for each cause with a bar chart, and represent the cumulative composition ratio with a line graph.

(4) By drawing a line where the cumulative composition ratio reaches 80%, identify the causes that should be addressed with priority.

[Example:]
For the problem of "many customer complaints," categorize the complaint details and tally the number of occurrences.
By creating a Pareto chart, you can see that "unclear operation methods" and "initial defects" account for 80% of the total, allowing you to determine that these should be prioritized for countermeasures.

3.4 Logic Tree

[Overview]
This is a method for breaking down complex problems into a tree structure to visually organize the relationships between elements. By breaking things down while keeping MECE (Mutually Exclusive and Collectively Exhaustive) in mind, it becomes easier to grasp the overall picture.

[How to use]
(1) Place the "problem" at the top.

(2) Break the problem down into its constituent elements or causes, creating branches.

(3) Further break down those branches until you reach a concrete level.

[Example]
For the problem of "declining sales," break it down into "decrease in number of customers" and "decrease in average spend per customer," and by further breaking each of these down, you can identify and organize potential causes.

These methods can be used not only individually but also in combination to enable more effective root cause analysis.

Conclusion

"Root cause analysis" is the first step in problem-solving and the most important one.
By not being misled by superficial phenomena and by repeatedly asking "Why?" to find the true root of a problem, we can build a better future.

From solving IT system problems to improving business strategies and dealing with daily life troubles, the mindset of root cause analysis is useful in all situations.

Please try to "deep dive" into the problems in front of you while utilizing the methods introduced here.

If you do, you will surely find the key to a solution.

Book Introductions

・The World's Easiest Problem-Solving Class: Acquire the Power to Think and Act for Yourself (Tankobon) – 2007/6/28
　Kensuke Watanabe (Author), matsu
　Explains logical thinking and problem-solving concepts learned at the world's top consulting firm in a way that even junior and senior high school students can understand.
　You will acquire the "genuine thinking skills" needed to survive in the world!
　The author, who provides highly acclaimed problem-solving training to everyone from new hires to management at leading companies in Japan and around the world, delivers the essence in an easy-to-understand and stimulating way.
　(* Book descriptions, etc., are generally based on the descriptions provided at the link (Amazon). Please refer to the link for detailed content.)

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

・An Illustrated Introduction to Problem Solving: How to Find Problems and How to Take Action (Tankobon) – 2003/11/1
　Junichi Sato (Author)
　Diamond Inc. (2003/11/1)
　Quickly discover problems, correctly grasp the causes, and summarize accurate countermeasures. A guide that uses illustrations to make problem-solving in daily work and life easy to understand.
　Using everyday "problems" as subject matter, this book develops the basic style of thinking and action skills that make goal achievement possible, from problem discovery to resolution, through unique illustrations and an easy-to-read style. It is a definitive edition that promotes a correct understanding and awareness of "goals, policies, strategies, tactics, opportunity utilization, and crisis management."
　(* Book descriptions, etc., are generally based on the descriptions provided at the link (Amazon). Please refer to the link for detailed content.)

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

・Illustrated Business Introduction: A Book to Understand the Latest Logical Thinking (Kindle Edition)
　Nobuyuki Imai (Author) Format: Kindle Edition
　Shuwa System (2023/3/30)
　An introductory book that uses illustrations to explain logical thinking useful for business. It covers basic thinking methods for finding logical and quick problem-solving solutions, specific ways to use them, how to acquire them, and convenient tools.
　(* Book descriptions, etc., are generally based on the descriptions provided at the link (Amazon). Please refer to the link for detailed content.)

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

・New Edition: Problem Solving Professional (Kindle Edition)
　Yoshinori Saito (Author) Format: Kindle Edition
　Diamond Inc. (2010/4/15)
　Systematizes methods for practicing problem-solving in the business field through two types of thinking, two techniques, and one process, such as "zero-based thinking," "hypothesis thinking," "MECE," and "logic trees." The basic concepts of problem-solving are here. While keeping the simple and clear problem-solving theory of the old edition, corporate case studies and exercises have been updated.
　(* Book descriptions, etc., are generally based on the descriptions provided at the link (Amazon). Please refer to the link for detailed content.)

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-