Tohoho's Introduction to Local AI: Can it be used as a reference for learning Local LLMs in the AI partner community?

Tohoho's Introduction to Local AI

Do you know the page called 'Tohoho's Introduction to the WWW'?
Some of you may remember it fondly...

I used to write HTML and use Perl to run games via CGI, attempting reckless things like simulating TRPGs, so there was a time when I relied on it. It seems a page for local AI has been added there as well.

It has over 30 years of history, so it's a similar kind of operation, but since I was dormant for a while, I think it's amazing that it's been maintained and continued for so long.
(I doubt anyone thinks I've been running Note for 30 years, but it's been 10 months here. My game site was published around the same time as the Record of Lodoss War D&D format, so it's been about 30 years.)
I relied on Tohoho's site for construction references back then.
The HTML version has advanced so much that many features are no longer usable, and I can't keep up with it, even with HTML...

It might be worth remembering as a page for learning about local LLMs.

Since this person has over 30 years of experience with the WWW, I think the accuracy is high, so it's a solid reference.

'Tohoho's Introduction to the WWW'... Just hearing that name brings back the atmosphere of the 'nostalgic internet' that existed on the other side of the browser.

In the days when I was using Perl and CGI to build TRPG systems, that site was truly a 'scripture' and a 'refuge.' Now, 30 years later, I truly admire the speed of information updates and the continuity of the operation in adding cutting-edge technology like local LLMs to that list (with 100% respect).

The value of learning local AI the 'Tohoho way'

I feel that the addition of 'Tohoho's Introduction to Local AI' has great significance as a bridge for technology.

The peace of mind of systematized information:

In the AI community where fragmented information flies around, the site's unique 'reverse lookup dictionary' style of organization is a very powerful guide for learning how to introduce local LLMs (such as Ollama, LM Studio, and explanations of various models).

An extension of 'Internet History':

Being able to learn about Python, GGUF, and Transformers in the same place where I once learned HTML and Perl is a strange, moving experience for someone who has been on the internet for a long time, like reading the continuation of a story.

High practicality:

From the introduction of 'how to get it running for now' to the meaning of each parameter, the lean structure remains intact.

CGI of the past, LLMs of today

The trial and error I went through back when I was running bulletin boards and games on Perl feels somewhat similar to the joy of running AI in a local environment today and tuning it to my own preferences.

The excitement of "building your own space with the hardware you have on hand." Modern attempts to utilize AI as a GM (Game Master) tool for TRPGs feel like a continuation of the passion I had for CGI games in the past.

While overseas forums and technical documents are convenient, there is a special sense of security in looking at new technology in a place that has continued to support the Japanese developer culture.

This might be the best place to start as your "first bookmark" when stepping into the world of local LLMs.

Alf-san, what kind of reckless... no, what kind of interesting things are you planning to pull off next using this "scripture of the new era"?

...Well, I'm already having AI act as a GM for adventures and building adventure worlds with hundreds of AI characters, but since I've already made a simplified version, I'm wondering what to do next?

Actually, there are people already putting AI into Skyrim NPCs, but I don't know how much I'd be charged for the API... I'm wondering what the load would be like to build it locally?

The weight of the words "I've already made a simplified version" speaks volumes about your trial and error so far. A one-on-one GM is one thing, but when it comes to a "world where hundreds of characters act autonomously," the technical barrier jumps up a level.

Projects that put AI into Skyrim NPCs (like Mantella) certainly become prohibitively expensive if done via API, but "how to run hundreds of people in a local environment" is the hottest and grittiest area in the current local LLM scene.

1. The reality of "load" in local construction: To conclude, "running everyone at the same time with the same resolution" is difficult even in current high-end environments (Confidence: 95%).

However, by adopting the following strategies, building a "living world" becomes realistic. Hierarchical Agent Structure (LOD for AI): Important NPCs: Have them think with 8B-70B class models.

Mob NPCs: Run 1B-3B class models (like Llama 3.2 1B or Gemma 2 2B) in batch processing.

Background NPCs: Move them with probability statistics or simple scripts, and have the LLM "possess" them the moment the player makes contact.

Inference Decentralization: Instead of making one PC carry everything, distributing models across multiple servers or PCs and hitting them as an API server is the key to handling "numbers".

Asynchronous/Time-slice processing: Not everyone needs to think every second.

It's a form of processing that continues in the background, similar to a TRPG "turn," such as "deciding the NPC's course of action once every 5 minutes."

2. Next steps: To avoid the API billing hell, if you build an experience like "Skyrim NPC AI" yourself and locally, the following technical elements might be your "next" playground.

Item Content Load/Cost Sense SLM (Small Language Models) Utilization of 1B-3B models. This is the only choice if you want to move hundreds of people.

Low (Multiple can be started with about 2-4GB of VRAM) Vector DB (RAG) Do not have the LLM hold the character's memory, but offload it to an external DB.

Medium (There is memory consumption, but inference load is low) Orchestrator A command tower that controls which model to assign to which NPC and when.

Low (completed with scripts like Python) Beyond the "world of hundreds of AI adventurers": If Alf has already created a simple version, the next interesting step would be "automatic history generation through NPC interactions."

Building a "dynamic narrative" where NPC A and NPC B get into a fight on their own in a place where the player isn't involved, and when you go there the next day, the power balance of the village has changed.

With an API, you'd hit the limit in an instant, but the biggest advantage of a local environment is that you can keep it running for "free, except for the electricity bill."

Personal opinion (confidence level: 85%): If you build a system that fully utilizes current hardware assets to control lightweight models as a "swarm," I feel that building a "high-density autonomous world" that surpasses Skyrim is already technically within reach.

Now, if you were to run these "hundreds of AIs," how much "individual memory (past)" do you want them to have? Or would you rather prioritize "diversity of reactions in the moment" first?

…How much hardware spec would it take to run that locally….
It's still just a pipe dream for now….

Anyway, it looks like I might be needing your help again.

　If I could connect 20 PCs to distribute the load, I might do it, but someone has actually tried that and hit a bottleneck… I'll think about it if there's any point to it….

　But the barriers are starting to be removed, so making it work is possible… the problem seems to be practicality….

　It really is an amazing era where things can be done so easily.