5 Things I Wish I Had Known Before Using AI Surrogate Models in the Field: Pitfalls in Model Selection, Tool Choice, and Training
In my previous two articles, I discussed accelerating simulations using Neural ODEs and automating calibration with deep reinforcement learning. If you thought, 'I want to try this myself,' this article is written for you. I will honestly summarize five things I wish I had known from the start after actually testing these methods in an AT transmission development environment at an automotive OEM.
1. Why I arrived at Neural ODEs: I tried RNNs and LSTMs first
I didn't choose Neural ODEs from the beginning. I arrived at them after trying RNNs and then LSTMs in that order.
Neural ODEs do not 'memorize patterns' but rather 'learn the parameters of differential equations.' By definition, the solutions result in physically consistent, smooth waveforms. This difference was the deciding factor.
Rather than which model to choose, it is more important to have a decision-making framework for 'why that specific model is necessary.'
2. How to determine the 'scope' of the surrogate (replacement)
What to replace with AI—the accuracy, speed, and training data required all change completely depending on where you set the input/output boundaries.
Decide 'what to make AI' by working backward from 'what is slow.'
3. Why I chose MATLAB: I also considered Python
It was a matter of existing assets. The 1D models in the field were built in Simulink. By using MATLAB's Deep Learning Toolbox, I could connect the trained model directly to Simulink. The value of being able to complete the work without switching environments is significant.
I also considered PyTorch (torchdiffeq). If the existing environment is MATLAB, MATLAB is the only choice; now that MCP environments exist, it is certainly a viable candidate.
Choose tools based on 'what connects to the existing environment' rather than 'what is cutting-edge.'
4. It was accurate in isolation, but it fell apart when integrated into the vehicle model
Although there were no problems in unit testing, the waveform accuracy deteriorated significantly as soon as it was integrated into the actual vehicle model.
During unit testing, oil temperature and source pressure were fixed. In the vehicle model, these change during driving—that difference was the cause of the accuracy degradation. A joint research project with engineers from MathWorks headquarters played a decisive role in identifying and solving the cause.
'Working in isolation' and 'working when integrated into a system' are different things. Verify under conditions close to production as early as possible.
5. What I ultimately wanted to achieve
I will state the results first. I achieved both a waveform accuracy RMSE of 0.02 and a 99% reduction in calculation time simultaneously.
On the other hand, I wasn't aiming for these numbers from the start. Even so, it takes know-how to set KPIs, involve those around you, and continue research. There is a considerable story of struggle behind the results presented at MATLAB EXPO 2023.
How I arrived here—the entire process is in the next paid article.
Accuracy and speed are not mutually exclusive. The order of design and the decisions made when hitting a wall were everything.
For those who want to know more
The following paid article contains everything—the decision-making processes and MATLAB implementation know-how—regarding the "why" behind each of the five points, where I got stuck, and what clues I used to break through.
I hope those who read the free articles ①②③ and thought, "I want to try this myself," will read this as their next step.
→ Paid Article ② "AI Surrogate for Physics Simulation using Neural ODE: From Design to Implementation, an Active Engineer Shows Everything" (¥500)
※ This article represents the author's personal views and is not the official position of their affiliated organization.