[Distribution] I made a distributed server for edge AI: Building a local AI API with llamacpp x Gemma

Hello, this is Rcat.
The other day, I tried running a quantized small-scale AI on my own computer in this article.
As a result, I was able to confirm performance that is at a practical level even on a CPU.at an operational level if using a GPU.

This time, I have slightly modified the script I created back then to make it a dedicated home AI server, so I would like to provide an explanation and distribute it.

Introduction

Terms of Service

Please check the terms of service in advance when using this information or work.

About comments

Please check the terms of service guidelines before commenting.
Comments that do not comply will be deleted.

Overview

What is edge AI?

Edge refers to the tip. In other words, it means AI that runs within individual devices rather than on a large server. That said, running it on mobile devices is quite a challenge, so this time I will
define a personal computer as the edge.

It does run on smartphones these days, but even with a reasonably high-performance smartphone, there is still some stress. In the first place, it eats up battery like crazy, so I feel there is not much point in running it on a smartphone for now.

Specifications

The specifications of the AI server I created this time are as follows. The delegation of processing is a point I particularly put effort into. By doing this, since processing is slow on a CPU, you can distribute jobs across multiple computers or prioritize using computers equipped with a GPU.

• Language

  • Python

• OS

  • Windows 11

  • Linux (Ubuntu)

• Interface

  • WEB API

  • Web Browser

  • Command Line

• Supported Processors

  • CPU (Confirmed on Intel N100, Ryzen 5 3600, Ryzen AI MAX+ 395)

  • GPU (Confirmed on CUDA, ROCm)

  • (It seems NPU is also supported in terms of libraries)

• Library

  • llamacpp

• Model
  Please check the previous article for details regarding the model.

  • Gemma3 4b-it-qat-Q4_K_M

• Others

  • Scalable
    If there is a computer running this program on the same local network, you can delegate jobs to it.

Feature Introduction

First, I will introduce what you can do with this tool.

Chat using the command line

This is the absolute basic functionality.You can interact via the command line without setting up a server.Note that it does not retain history, so it is a one-time conversation.

R:\Rcat>start.bat -p "こんにちわ" --temperature 0.1 --top_p 0.95 --max_tokens 1024 --model "R:\llamacppエッジAIサーバー\gemma-3-4b-it-qat-Q4_K_M.gguf"
 
2025-07-05 16:09:55,620 INFO llama_cpp_test モデルをロード中: gemma-3-4b-it-qat-Q4_K_M.gguf
2025-07-05 16:09:57,431 INFO llama_cpp_test モデルのロードが完了しました。
2025-07-05 16:09:57,433 INFO llama_cpp_test ストリーミングモードで応答を生成中...

こんにちは！何かお手伝いできることはありますか？ どんなことでもお気軽にご質問ください。 😊

Argument Introduction

• -p Prompt
  Specifies the prompt to pass to the model. This is mandatory.

• --systemprompt System Prompt
  Specifies the system prompt. This is optional.

• --temperature
  This is the temperature parameter. This is optional.

• --top_p
  This is the top_p parameter. This is optional.

• --max_tokens
  This is the max_tokens parameter. This is optional.

• --model
  Select the model to load using the full path.
  If omitted, it will load the data written in the program. Since it likely won't match anything other than my environment, you should almost certainly specify this.

Using it as a Web server

By changing the arguments at runtime, you can run it as a Web server.

Rcat>start.bat -s --port 80 --model "R:\llamacppエッジAIサーバー\gemma-3-4b-it-qat-Q4_K_M.gguf"

2025-07-05 16:14:23,892 INFO llama_cpp_test モデルをロード中: gemma-3-4b-it-qat-Q4_K_M.gguf
2025-07-05 16:14:25,676 INFO llama_cpp_test モデルのロードが完了しました。
この機能はLinux (POSIX) 環境でのみサポートされています。
 * Serving Flask app 'llama_cpp_test'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:80
 * Running on http://192.168.0.199:80
Press CTRL+C to quit

A message saying "Running on http://XXX.XXX.XXX.XXX:XX" will appear, and you can access it from your browser by going to that address.

Argument Introduction

• -s --server
  An option to enable server mode. It is recognized with the highest priority.

• --port
  Specifies the port number. Optional.

• --master
  Starts as a master server.Only servers with this specified can use the delegation function to external nodes can be used.

• --priority
  Priority during distributed processing. A smaller value indicates higher priority and is used for selection when queue conditions are the same. See Distributed Processing for details.

• --cpu
  Linux only. Limits CPU usage using the cpulimit command. Used in conjunction with the --cpucore option. Optional.

• --cpucore
  Used with --cpu. Specifies the number of CPU cores. If the input is incorrect, CPU usage cannot be limited accurately.

Using the Web Interface

When you access it while in server mode, you will see a screen like the one below. By entering text here, you can easily chat with the AI.

Using the Translation Function

There is a small link for translation in the top right; click it to switch to translation mode.
This is just a pre-set instruction in the system prompt, but it is set up so you can translate to English just by entering text. Whether it translates properly depends on the accuracy of the model.

Using the API

This is likely the main purpose of this tool.
By using this, you can completely call the AI from another program locally.

Chat API

This is the CGI for chatting.
POST the required parameters as a JSON body.

Endpoint /CGI_DoChat

All parameters are optional. Please specify only what is necessary.

{
"system_prompt": "システムプロンプト"
"user_prompt": "ユーザープロンプト"
"max_tokens": int #最大トークン数 8192
"temperature": float #温度パラメーター 1.0
"top_p": float #トップPパラメーター 0.95
"top_k": float #トップKパラメーター 40
}

Example of execution in Python

>>> import requests
>>> body = {
... "system_prompt":"ユーザーの入力を中国語に翻訳してください",
... "user_prompt":"ねこはもふもふで癒されます"
... }
>>> resp = requests.post(url="http://localhost/CGI_DoChat",json=body)
>>> print(resp.json().get("response"))
猫很软萌，让人感到放松。(Māo hěn ruǎnméng, ràng rén gǎndào ràngshuǎn.)

または、少しくだけた表現なら：

猫咪超萌，超级治愈！(Māo mī chāo méng, chūjí zhìyù!)

**解説:**

* **猫 (māo):** 猫
* **很 (hěn):** とても
* **软萌 (ruǎnméng):** ふわふわでかわいい、可愛らしい
* **放松 (rǎngshuǎn):** リラックスする、癒し
* **治愈 (zhìyù):** 癒す
* **猫咪 (māo mī):** 猫（可愛らしい言い方）
* **超萌 (chāo méng):** 超可愛（「超」は強調のため）
* **超级 (chūjí):** とても（強調のため）

どちらの翻訳も元の文のニュアンスを捉えています。状況や伝えたい感じによって使い分けてください。

>>> import json
>>> json.dumps(resp.json(),indent=2,ensure_ascii=False)                                                                 '{\n  "first": 0.30786895751953125,\n  "response": "猫很软萌，让人感到放松。(Māo hěn ruǎnméng, ràng rén gǎndào ràngshuǎn.)\\n\\nまたは、少しくだけた表現なら：\\n\\n猫咪超萌，超级治愈！(Māo mī chāo méng, chūjí zhìyù!)\\n\\n**解説:**\\n\\n* *
*猫 (māo):** 猫\\n* **很 (hěn):** とても\\n* **软萌 (ruǎnméng):** ふわふわでかわいい、可愛らしい\\n* **放松 (rǎngshuǎn):** リラックスする、癒し\\n* **治愈 (zhìyù):** 癒す\\n* **猫咪 (māo mī):** 猫（可愛らしい言い方）\\n* **超萌 (chāo méng):** 超可愛（「超」は強調のため）\\n* **超级 (chūjí):** とても（強調のため）\\n\\nどちらの翻訳も元の文のニュアンスを捉えています。状況や伝えたい感じによって使い分けてください。",\n  "time": 5.457850456237793,\n  "tokens": 270\n}'             >>> print(json.dumps(resp.json(),indent=2,ensure_ascii=False))                                                          {
  "first": 0.30786895751953125,
  "response": "猫很软萌，让人感到放松。(Māo hěn ruǎnméng, ràng rén gǎndào ràngshuǎn.)\n\nまたは、少しくだけた表現なら：\
n\n猫咪超萌，超级治愈！(Māo mī chāo méng, chūjí zhìyù!)\n\n**解説:**\n\n* **猫 (māo):** 猫\n* **很 (hěn):** とても\n* **软萌 (ruǎnméng):** ふわふわでかわいい、可愛らしい\n* **放松 (rǎngshuǎn):** リラックスする、癒し\n* **治愈 (zhìyù):** 癒 す\n* **猫咪 (māo mī):** 猫（可愛らしい言い方）\n* **超萌 (chāo méng):** 超可愛（「超」は強調のため）\n* **超级 (chūjí):** とても（強調のため）\n\nどちらの翻訳も元の文のニュアンスを捉えています。状況や伝えたい感じによって使い分けてください 。",                                                                                                                      "time": 5.457850456237793,
  "tokens": 270
}

The response is also in JSON.
The AI's response string is contained within the response key. Other fields include processing time, tokens, etc. In this case, it's 5 seconds.

Translation API

This is an API for performing translations.
If you only need text, GET is also possible.

Endpoint /CGI_Translate

Parameters

{
"text": "翻訳したい文字列"
"max_tokens": int #最大トークン数 4192
"temperature": float #温度パラメーター 0.5
"top_p": float #トップPパラメーター 0.95
}

Example of execution via GET

Information Retrieval API

This is an API for distributed processing, which I will introduce later.
Three types of information are returned: server priority, remaining queue, and execution status.

By the way, AI processing is heavy, so parallel execution is not possible.
If multiple requests are received simultaneously, they are stored in a queue and processed in order from the beginning.
If there are other nodes, it delegates to the one that is most available. If none are available, it executes it itself. It operates to keep the queue counts equal.
If 10 or more accumulate, it is designed to return an error due to overload.

Use Cases

Turning a Discord Bot into an AI

This has to be it.Private messagesWhen you want to perform translations or summaries, there is resistance to outsourcing to an external server.
However, what about your own server?You can execute it without worrying about anything, right?

Example of executing a translation in response to a specific reaction

Installation Method

Checking Package Contents

Distribution is handled at the very bottom.
The package contents are as follows.

Model placement

First, please prepare the AI model initially.
Once downloaded, place it in this folder or wherever you prefer.
You can select the model using the --model argument for the location where you placed it.

This article introduces how to download models Please choose one that matches your computer's specifications.

How to install Python

This project is built with Python, so please be sure to install it.

Environment setup

For Windows, execute start.bat, and for Linux, execute start.sh to automatically create a virtual environment and begin installing libraries.
At this time, if you launch it after properly specifying the parameters, you should be able to confirm normal operation.
If you do not specify parameters, it will crash after the environment setup.

* Note: The distribution package has been confirmed for fully automatic execution on Windows.

For Windows users, a batch file to launch as a server is also included Please use whichever you prefer. Note that even when using it as a server, you need to specify the model, so please rewrite the contents slightly.

When using a GPU

As introduced in the previous article, you need to build the library yourself if you want to use a GPU. The build method is introduced in the article above.
Therefore, do not perform the automatic environment setup; instead, create the virtual environment yourself, build only this library first, and then install the remaining libraries.
Once installed, it can be troublesome due to uninstallation and other issues.

When using with RCOm (Radeon)

This article uses the Radeon 8060S, which is the integrated GPU of the Ryzen AI MAX+ 395.
Depending on how it is used, it is a monster that reportedly exceeds the RTX 4070.

Performing distributed processing

This tool supports distributed processing.
Distributed does not mean splitting a single input for processing, but rather executing on different computers for each job.
The web API introduced earlier is also used for this distribution.

Simply put, this configuration allows for the following.

When used alone
The main AI server works hard, but since it is a CPU, processing is slow.
It feels fine if you only use it occasionally.

When nodes are launched
When sub-AI nodes are running, processing is delegated externally.
Of course, while the sub-nodes are being used, it also processes itself.
Just having one sub-node that uses a GPU for high-speed processing makes a huge difference.

In theory, multiple sub-nodes can be launched.
Parallel processing through sheer numbers is possible even without a GPU.

Enable distributed processing

To use this process, an additional module for searching external nodes is required.
This is the M-Search module distributed in this article.

Alternatively, you can rewrite the source code directly to hardcode the nodes.
If you put a set of IP addresses and port numbers in this list, they will be recognized.

After downloading M-Search, please place it inside the folder of this tool.

To start in distributed processing mode, specify the --master parameter.

>start.bat -s --priority 100 --master --model "もでる☆"

The side sharing the load only specifies the priority.
In the following case, since it is lower than the master, it feels like this side will take on the processing preferentially.

>start.bat -s --priority 0 --model <>

When starting in node mode, the text "M-SEARCH server started" will appear after the model is loaded.
If this appears, it is running as a distributed node.

After that, if you input continuously on the master side, you should be able to confirm that processing is being performed simultaneously on both the distributed side and the master side.

Regarding the criteria for distributed processing

Jobs are distributed based on the following conditions.
The ones at the top have higher priority.

1. Nodes with an empty queue (other than the master)
   If there is already a queue, it will be assigned to the node with the fewest.
   The master will be last because it holds all queues.

2. Nodes that are not in an execution state
   If all queues are not empty, a node that is not in an execution state is selected.

3. Priority settings
   If all conditions are the same, priority is used for distribution.

Summary

This time, I created a personal AI server using a quantized LLM.
The era where LLMs run constantly at home has finally arrived.
With this, you can input any content without worrying about terms of service!
Let's use it while keeping morals in mind.

Distribution Information

Distribution URL

It is distributed from the following URL.
Terms of Service Please agree to the terms before using.

https://script.google.com/macros/s/AKfycbxdcr8pnazR7RbjaSICTtaNWfN7h_rjQrKlZ3h9CZpPRFzRILk1OGc8mZqKbF-NXNO9/exec?name=LocalEdgeAIServer

Link Collection

[Linux] Systemd addition support script

[Windows] How to add to Task Scheduler

How to install LoggingBOT

Installing Linux on a mini PC