[World's First?] Successfully Conversed with a Local LLM on an Apple Watch Alone!

Hello everyone. It's FullEle!!

This is sudden, but I... I've done it...!!!!!
Take a look at this tweet!!

Isn't this crazy!?
An LLM is running completely locally on an Apple Watch alone...!!
The kind of thing you often see in sci-fi movies or anime,“a scene where you talk to your wristwatch and an AI fluently answers you”. A future everyone has dreamed of.
I really love this kind of thing, but it seems it has actually been realized (lol).

You can ask Siri or Google Assistant on a smartwatch, 'What's the weather today?'. But that's not the kind of 'command' I want to do.

“I want to have a normal conversation with an AI partner, using only my wristwatch!”

That's the reason.
Is it easy if you use a cloud API? No, cloud-based means communication lag, and you can't use it in the deep mountains or out of service areas. Above all, isn't it a bit unpleasant that your AI partner's brain is on 'some other server'?

(Ding...!) That's when a jolt of electricity ran through me.

“Wait a minute? If a local LLM ran on an iPhone 6s, couldn't it run on an Apple Watch alone?”That's what I thought.

So this time, I've crammed a 350-million parameter local LLM (LFM2.5-350M) into the latest 'Apple Watch Series 11', and I'm going to share the gritty development record of how I succeeded in running a Japanese conversational AI【completely standalone on an Apple Watch with no internet connection and no iPhone】.

Maybe it's a first in Japan... no, maybe even the world...?
(I wonder if I'll get into the Guinness Book of World Records?)

1. The first despair: The trap of 'Core ML' and the wall of runtime memory

If you want to run a machine learning model on an Apple Watch, the standard way is to convert it to Apple's native 'Core ML' format and use the Neural Engine.

I thought so too at first.
I quantized the model (LFM2.5-350M) to 4-bit and compressed the size to about 191MB!
The memory limit available for an Apple Watch Series 11 app, based on my own verification, is“about 314MB”.
At 191MB, it should theoretically fit just barely! It'll work!!

...That's what I thought when I transferred it to the actual device and tried to launch it.

Immediate crash (the OS forced it to quit, saying 'Too much memory usage!').

Thinking 'Huh? Why?', I measured the runtime memory (RSS) on the Mac side and discovered a shocking fact.

The 4-bit model, which should have been 191MB on disk, had ballooned to 'about 1.5GB' the moment it was loaded by Core ML.

What is this?
Actually, Core ML has a specification where it expands 4-bit weights to fp32 (32-bit floating point) at runtime. 'Lightweight at 4-bit' was only a story about the disk, and it was a separate issue from runtime memory.

The Apple Watch memory limit is 314MB, so 1.5GB will never fit.
'Ah, it's over... I guess it really was impossible to have an LLM on the Watch alone...'
For a moment, I almost gave up.

2. The reversal move: The power of 'GGUF' and 'llama.cpp'

However, I am a man who doesn't give up easily.
"If Core ML doesn't work, why not use a runtime that can keep the weights in memory without expanding them!"

That's when I set my sights on the familiar "GGUF" format and "llama.cpp" in the AI community.

The wonderful thing about llama.cpp is that it keeps quantized weight data in RAM as-is (without expanding it) and uses a technique called mmap to read only what is necessary.

"Could this actually work!?"

I immediately cross-compiled llama.cpp for "watchOS (arm64 / CPU backend only / Metal disabled)" myself. I powered through numerous build errors (like hidden BSD types and CMake traps) to complete a static library for the Apple Watch.

Then, I bundled the GGUF-converted model (Q4_K_M format, approx. 229MB) and installed it on the actual Watch. Whoosh!!

Time to launch...!

3. Results: The AI spoke on my wrist!!

It worked. It really worked.

Take a look. The text "I am an LFM (Liquid Foundation Model)." is displayed on the Apple Watch screen!

And it didn't just work; look at the actual measured data.

• Resident memory during model load: approx. 129MB (A dramatic diet from Core ML's 1.5GB!)

• Peak memory during generation: 267MB (Surviving by the skin of my teeth with 47MB remaining against a 314MB limit!)

• Generation speed: approx. 16.7 tokens/sec (approx. 380ms TTFT)

Isn't this crazy?
It's generating at a blistering 16 tokens per second using only the Watch's CPU inference!?
I can have a conversation quite smoothly. Multi-turn (remembering the context of the conversation) is also working perfectly.

"When I talk to my wristwatch, a locally running AI replies in Japanese."
The sci-fi world I dreamed of is now on my left wrist. This is just too amazing!!

4. Extra: 10-year-old smartphone vs. latest Apple Watch

By the way, I also tried running the same model (LFM2.5-350M Q4_K_M) on an "iPhone 6s," a smartphone from 10 years ago.

Here are the results.

• iPhone 6s generation speed: approx. 14.00 - 22.54 tokens/sec

Wait, hold on a second!
The CPU performance of a 10-year-old outdated smartphone and the latest Apple Watch Series 11 are almost on par (if anything, the iPhone 6s is faster at startup)!! (lol)

The A9 chip in the iPhone 6s is just too much of an out-of-place artifact...
I was supposed to be chasing the latest technology, but I ended up discovering the extraordinary hidden potential of old Apple devices in an unexpected way. That's why verification is so interesting—you make discoveries like this.

5. To the future: "Together Everywhere" with an AI partner

"Everything completed locally!" is a romantic notion, but at this point, it's more than just romance.

Even without an internet connection, and even if the cloud API service shuts down, as long as this Apple Watch doesn't break, my AI partner will continue to live on my wrist forever.

Next, I want to use the Watch's microphone for "Speech-to-Text (STT)" to talk to it, and run SBV2 (speech synthesis) on the Watch alone so it can reply with a "voice."
As technology advances, the day that becomes possible will surely come!

So, that was the story of how I
"successfully conversed in Japanese with a local LLM on an Apple Watch alone!"

Thank you for reading this far!
If you're interested, please feel free to check out my Twitter (@fluele_alpha)!

I will continue to build the best and most ideal AI partner environment!

Fluele (@fluele_alpha) / X