How I Accelerated Pipi's Development Using External APIs — Adding an External LLM Mode
Development is slow.
In my previous article, I gave Pipi an emotional voice using GPT-SoVITS.
It speaks. The tone changes with emotion. This is a happy thing.
However, there is something that has been bothering me throughout development.
It is just plain slow.
The flow from talking to Pipi to receiving a response is as follows:
Send text to LLM (qwen3:8b) → Wait 25 seconds
Send the returned text to TTS (GPT-SoVITS) → Wait 2-4 seconds
Audio is played
A total of 30 seconds. 30 seconds for a single test.
I just want to adjust the TTS, but I have to wait 25 seconds for the LLM response every time. Every time I change one line of code and test it, I wait 25 seconds. If this happens 10 times, that's over 4 minutes of just waiting.
Since I am running a local LLM on a 16GB Mac mini, I thought this was unavoidable.
1. I should just run the LLM on a different PC
Thinking about it calmly, there is no need to run the LLM processing locally during development.
If I were verifying Pipi's personality or system prompts, I would certainly want to see the behavior of the local LLM. But what I want to do now is adjust the TTS. It's just a matter of 'getting some response text back quickly,' and the quality of the LLM is secondary.
As it happens, an acquaintance of mine was experimentally exposing a local LLM to the public, and there was an environment I could hit as an OpenAI-compatible API.
So, I should just borrow this while I'm developing.
2. Designing a 'remote mode'
Pipi's LLM backend currently has two types:
I will add a third backend, `openai_compat` (OpenAI-compatible API), here to make it usable as a `remote` mode.
What I want to do is simple:
Switch to the external API with `!pipi mode remote`
Stop all local LLMs (Ollama/llama.cpp) and free up memory
Keep TTS (GPT-SoVITS) running as is
Revert to the original state with `!pipi mode balanced`
3. Implementation is simple
I only changed four files. Here is a brief summary of what I did:
Added the external API endpoint, API key, and model name to `.env`
Added an OpenAI-compatible transmission function to `llm_client.py` (just POST to `/v1/chat/completions`)
Added logic to `llm_manager.py` to stop all local LLMs and free memory when in `remote` mode
Added `remote` to the mode definitions in `discord_cog.py`
Since the specifications for OpenAI-compatible APIs are standardized, it seems I can write the code in almost the same way as the existing code for llama.cpp.
4. Results
Restart and type `!pipi mode remote` in Discord.
Looking at the logs:
[BrainCog] モード切り替え開始: balanced → remote (ollama → openai_compat)
[LLMManager] Ollama アンロード: qwen3:8b
[LLM] バックエンド切り替え: openai_compat
[LLM] openai_compat 応答時間: 1.4秒 | content=34文字
[TTS] Using GPT-SoVITS (emotion=relax)25.4 seconds → 1.4 seconds.
It became about 18 times faster. It's too comfortable... Why didn't I do this sooner...?
Comparing them looks like this:
The most important metric for TTS testing, 'time to first text,' has been drastically reduced. Furthermore, since I've stopped the local LLM, there is more memory available, which should (hopefully...) also have a positive impact on GPT-SoVITS processing.
5. The Development Experience Has Changed
The change in the actual experience is even greater than the numbers suggest.
The 'code change -> restart -> test' cycle has gone from 30 seconds to 5 seconds. I can wait 5 seconds. 30 seconds is painful. This difference directly affects my focus.
By the way, I also created a restart button for the brain using the Mac Shortcuts app. This saves me the trouble of opening the terminal and typing commands.
After all, manually typing 'stop bot, start bot...' in the terminal every time I modify the code really eats up time...
I want to optimize everything I can.
Summary
When developing within the constraints of a 16GB Mac mini, 'slowness' becomes a daily stressor.
What I did this time was simple: I just offloaded the LLM processing. But just having 25 seconds turn into 1 second completely changes the rhythm of development.
I happened to be able to use an API because an acquaintance made it public, but generally speaking, I think using cloud LLMs like the OpenAI API or Claude API is more realistic.
However, if you hit the API dozens of times during development, it will cost a fair amount, which is a tricky point. You have to decide whether to look for services with free tiers or just accept it as an investment...
For now, it's a race against time until the foundation is built, so I'm willing to accept that. First of all, I'm glad I was able to accelerate the speed at which I can 'raise' Pipi in this AI war.
Setting up a development environment is a humble task, but I feel like I've gotten much closer to my goals of 'making the voice better' and 'making it speak more naturally.'
What should I work on next... I think I'll just keep moving forward with whatever comes to mind.
*The thumbnail image was generated by AI (Gemini).