Salesforce Production Incidents: How Handing Off Initial Triage to AI Changed On-Site Response

Do you find yourself staring at debug logs for a long time trying to guess the cause every time an incident occurs in your Salesforce production environment? As a Salesforce engineering manager, I get involved in initial responses to production incidents about once or twice a month.

The time it takes from receiving the initial error report to forming a hypothesis about "what is happening" has been the bottleneck for the entire response process. During incident response, the only thing we can show on the customer inquiry screen is "Currently under investigation," and the feeling of time passing without knowing the cause is a level of anxiety that only those who have experienced it can understand.

In this article, I will write about how we switched to an operation where AI handles the initial triage of debug logs, and how that changed our average response time.

To give you the conclusion first, the time it takes to form a hypothesis for the cause has been reduced from an average of 37 minutes before implementation to an average of 16 minutes. However, it is not as simple as just dumping everything on the AI to solve; there have been times when I took a hypothesis at face value and learned a painful lesson.

From here on, I will disclose the full text of the prompts I actually send to the AI, the preprocessing steps to organize the logs before handing them over, and the judgment checklist I created to improve the accuracy of initial triage. As long as you have the debug logs and Setup Audit Trail output on hand, this is content you can use immediately for your next incident response.

Why initial triage took so long during production incidents

At the scene of an incident response, the work begins by narrowing down "at which layer the issue is occurring." There are usually 3 to 4 candidates each time, such as an Apex exception, an external integration timeout, or a change in permission settings, and it used to take an average of 30 to 40 minutes to narrow this down manually.

To be honest, I didn't feel much value in this narrowing-down process itself. The correction work after the cause is identified is much more intellectually demanding, and the narrowing-down process had become a brute-force task relying on experience and intuition. While a veteran engineer might find the cause in 10 minutes, it could take over an hour if a less experienced member was in charge, making inconsistency in response another issue.

Preprocessing for feeding debug logs to AI

The first thing I did was to ensure that debug logs were not passed to the AI as they were. Salesforce debug logs contain a lot of noise, and if you paste them as they are, important lines get buried.

This is especially noticeable when the log level is set too high toward FINEST, where it is not uncommon for a single request to result in several thousand lines.

Therefore, I decided to mechanically filter the following three things before handing them over.