Locally Running AI Has Come This Far—A Zero-Cost Coding Environment Built with Gemma 4 and Ollama: Why Local LLMs Now?

AI-powered coding assistance is no longer a niche topic. Many engineers routinely leverage Claude Code or GitHub Copilot to generate, review, and debug code. However, reliance on cloud APIs comes with hidden costs. Token usage fees, security concerns regarding sending internal code to external services, and network latency—these are small, accumulating stresses that can, in some cases, become organizational constraints.

In April 2026, that situation changed. Gemma 4, released by Google, is said to be the first open-weights model to reach a level where "practical coding assistance is possible even when running locally." And the tool to run it on your own machine is Ollama.

In this article, based on the practical guide "Running Google Gemma 4 With Ollama, Claude Code, OpenCode, Codex: Complete Local Setup" published by George Liu on Substack, I will explain how to set up Gemma 4 and Ollama, and integrate them with Claude Code, OpenCode, and the Codex CLI, in a way that intermediate IT users can replicate.

No data sent to the cloud. No API keys or billing required. Let's build a zero-cost local AI development environment that you can start using today.

What is Gemma 4—Why is this time different?

The latest version of the open-weights model released by Google

Gemma 4 is the fourth generation of the open-weights LLM series released by Google on April 2, 2026. "Open-weights" means that the model's weight files (parameters) are publicly available, allowing them to be run freely on your own PC or server. Unlike closed models like ChatGPT or Claude, it operates without an internet connection.

There are multiple models in the Gemma 4 lineup. In terms of parameter count, they are broadly categorized into **2B/4B (lightweight versions) and 26B/31B (high-performance versions)**. What we should focus on here is the architecture of the 26B model.

The "Cost-Performance Reversal" created by Mixture-of-Experts

The 26B model of Gemma 4 adopts an architecture called **Mixture-of-Experts (MoE)**. MoE is a design where only a portion of the model's total parameters are actually activated during inference.

In the case of the Gemma 4 26B-A4B model, the total parameter count is 25.2B, but only the equivalent of 3.8B is actually used when generating a single token. In other words, while it appears to be a 26B model, a "reversal of cost-performance" is occurring where the execution cost is at the 4B class level.

In George Liu's guide, it is reported that this 26B-A4B model runs at approximately 10–15 tokens/second on a MacBook Pro M4 Pro (48GB memory). As for response speed, it feels like "waiting a little," but it is about the same lag as reading while typing, making it sufficiently practical for actual coding assistance.

Common failure pattern: Mistakes in model selection

The first hurdle many people stumble over here is the assumption that "you should just choose the largest model."

If you choose the 31B model, it may run on 16GB of memory depending on the degree of quantization (model compression), but the speed will drop significantly. Also, if memory runs low, the OS will start swapping to disk, and inference speed can drop to a few tokens per second or less. George Liu's experiments also recorded that disk swapping occurred when using the 26B 64K context setting on a 48GB machine.

Recommended criteria for judgment:

• 8–16GB memory → gemma4:4b (quantized version) used

• 24–32GB memory → gemma4:12b is a realistic upper limit

• 48GB memory or more → gemma4:26b is the sweet spot

What is Ollama—The standard tool for running local LLMs

The simplest setup to get started with CLI

Ollama is an open-source tool for running LLMs in a local environment. It supports Mac, Windows, and Linux, and installation is as simple as downloading it from the official website ( ollama.com ).

After installation, simply run the following command in your terminal to launch Gemma 4.

ollama run gemma4

The model will be downloaded automatically upon the first run. By default, the smallest variant is retrieved. If you want to specify a particular size, do it as follows.

ollama pull gemma4:26b

Changing the model storage location

On machines with limited internal SSD space, such as a MacBook, you may want to change the storage location of model files to an external drive or a path with more capacity. Ollama allows you to control this via the `OLLAMA_MODELS` environment variable.

export OLLAMA_MODELS=/Volumes/ExternalDrive/ollama-models

Adding this to your `.zshrc` or `.bashrc` will make it effective for future sessions as well.

Creating custom models with Modelfile

One of Ollama's powerful features is the Modelfile. This is a configuration file similar to a Dockerfile, which allows you to define a new model by overriding parameters on a base model.

A point particularly highlighted in George Liu's guide is the expansion of the Context Window. Ollama's default has only a small Context Window, but you can expand it to 64K tokens using a Modelfile.

printf 'FROM gemma4:26b\nPARAMETER num_ctx 65536' > /tmp/Modelfile-64k
ollama create gemma4-26b-64k -f /tmp/Modelfile-64k

In this command, we are creating a new model `gemma4-26b-64k` based on `gemma4:26b` with `num_ctx 65536` (a 64,000-token context window) set.

Why is the Context Window important?: In coding assistance scenarios, you will be passing long code files or the contents of multiple files to the model at once. If the Context Window is short, the latter part of the files may be truncated, or the model may "forget" past conversations. 64K tokens is a size sufficient to cover even medium-sized codebases.

Integration with three coding agents

Understanding the structure: API endpoints provided by Ollama

When you start Ollama, an OpenAI-compatible API endpoint is launched locally (the default is `http://localhost:11434/v1` ). In other words, for any tool that supports the OpenAI API, you can use Ollama models simply by rewriting the endpoint URL. This mechanism enables integration with Claude Code, OpenCode, and Codex CLI.

The first integration—Claude Code

Claude Code is a CLI-based coding assistant provided by Anthropic. By default, it uses Anthropic's cloud API, but you can switch to the Ollama endpoint by setting the following environment variables.

export ANTHROPIC_BASE_URL=http://localhost:11434/v1
export ANTHROPIC_API_KEY=ollama

The value of `ANTHROPIC_API_KEY` does not need to be an actual key; any string like `ollama` will suffice (because the local Ollama server does not perform key validation).

In Ollama v0.15 and later, you can launch it more easily using the `launch` command.

ollama launch claude --model gemma4-26b-64k

This command launches Claude Code while automatically configuring it to use `gemma4-26b-64k` as the model.

Note: Claude Code expects model responses to follow a specific format. While compatibility with Gemma 4 is generally good, there are times when the response format deviates during tool calls (tool_use). If such issues occur, it is effective to add a system prompt to the Modelfile to adjust the model's behavior.

The Second Integration—OpenCode

OpenCode is a CLI agent gaining attention as an open-source alternative to Claude Code. It works well with Ollama and can be used simply by adding it as a provider in the configuration file (`~/.config/opencode/config.json`).

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "models": ["gemma4:26b"]
    }
  },
  "defaultProvider": "ollama",
  "defaultModel": "gemma4:26b"
}

Since OpenCode's tool-calling format requirements are not as strict as Claude Code's, more stable operation can be expected when combined with Gemma 4. George Liu's guide also records that OpenCode functioned smoothly in its integration with Gemma 4.

The Third Integration—Codex CLI

Codex CLI is a CLI-based coding agent released by OpenAI. Because it supports the OpenAI-compatible API, you can redirect it to Ollama using environment variables.

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=ollama
codex --model gemma4:26b

Among the three agents, this requires the simplest setup. However, because Codex CLI is designed with OpenAI models in mind, there are cases where some advanced tool-calling features may not work with Gemma 4. It is at a level sufficient for basic code generation and explanation tasks.

Points for Choosing an Agent

Which agent you choose depends on your goals and the level of configuration complexity you can tolerate. It is easier to decide if you organize your thoughts based on the following perspectives.

When choosing Claude Code: When you are already accustomed to the Claude Code workflow and want to switch to a local model while maintaining the same UX.

When choosing OpenCode: When you want to prioritize open-source tools or value stability with Gemma 4.

When choosing Codex CLI: When you want to try it out with a simple setup or value compatibility with the OpenAI ecosystem.

Trying It Out—Changes Seen in Before/After

Before: Dependence on Cloud APIs and Hidden Costs

Coding assistance using cloud APIs has problems that are hard to notice once you get used to them.

One is cost. If used daily, API fees can amount to thousands to tens of thousands of yen per month. As projects increase, token consumption grows linearly, and if used by a team, it doubles even further.

Another is data privacy. Sending client code or internal proprietary logic to external services, even if not a problem under terms of service, may have restrictions depending on organizational policies or contracts with customers. The hassle of "checking before using" can often become a source of stress.

Furthermore, there is also the issue of network dependence. In offline environments or places with unstable communication, cloud APIs cannot be used. Working on a Shinkansen, in a basement, or with unstable Wi-Fi during an overseas business trip—once you experience the inability to use AI assistance in these scenes, it becomes surprisingly stressful.

After: What Changes with Local Execution

Let's look at how these challenges change with the combination of Gemma 4 and Ollama through specific scenarios.

[Scenario 1: Internal Tool Development]

Suppose you are writing an API wrapper that connects to an internal payment system. The code contains connection URLs and authentication flows, and you were hesitant to send it to an external service. With a local LLM, you can pass that code directly into the context. A request like 'Add refresh token handling to this authentication flow' can be executed without any security concerns.

[Scenario 2: Coding on the Go]

While traveling on the Shinkansen, you suddenly want to try out an implementation you just thought of. Ollama works even when offline. By launching a lightweight model with `ollama run gemma4:4b` and combining it with OpenCode, you can write code while interacting via the CLI. You can continue pair programming with AI even without a network connection.

[Scenario 3: Cost Management]

For engineers with personal side projects, monthly API costs are not a small burden. By switching to a local LLM, the cost of trial and error becomes zero. A practical approach is to develop locally during the prototyping stage and use cloud APIs only for final verification where quality is paramount.

Moving Toward Implementation—Organizing the Entire System and Next Steps

Summary of Environment Setup

To summarize what we have covered so far, the local Gemma 4 × Ollama environment consists of the following structure.

Infrastructure Layer (Ollama): The LLM execution engine. It handles model management, downloading, and providing the API server.

Model Layer (Gemma 4): The actual model that performs inference. You select a variant based on your hardware. You can adjust parameters like the context window using a Modelfile.

Agent Layer (Claude Code / OpenCode / Codex CLI): The CLI tool responsible for user interaction. You can use it simply by setting the Ollama API endpoint as the destination.

Frequently Asked Questions

Q. Can Gemma 4 be used for things other than coding?

Yes. Since Gemma 4 is a general-purpose LLM, it can be used for a wide range of tasks beyond coding assistance, such as document summarization, translation, and Q&A. However, there is still a gap in coding assistance quality compared to the latest cloud models (Claude Opus, GPT-4o, etc.). It is appropriate to position it as a 'practical-level AI that can be used without billing'.

Q. Are Gemma and Gemini different things?

Yes, they are different. Gemini is a model provided via API as a Google cloud service (which incurs charges). Gemma is an open-weights model with publicly available weight files that you can run for free on your own machine. Be careful, as the names are similar and easy to confuse.

Q. How much memory is needed for practical use?

The practical lower limit is 8GB. In this case, you use a quantized version of gemma4:4b (such as Q4_K_M). It runs comfortably with 16GB. If you have 24GB or more, the 12B model also becomes an option.

Next Steps

Once you have built your environment, I recommend starting with simple tasks. Having it add tests to existing code, write comments, or refactor short scripts—getting a feel for it with these small tasks before expanding to more complex uses is the most reliable approach.

Also, Ollama has an active community, and information on optimal Modelfile settings and how to choose quantization variants is updated daily. Please try searching for settings that suit your environment while also referring to the official library (ollama.com/library/gemma4).

Summary—An Era Where Local AI Becomes an "Option"

The combination of Gemma 4 and Ollama has presented local LLMs as a "realistic option" for the first time. You don't need to complete every task locally. What is important is that the constraint of "only being able to use the cloud" has been removed.

Cost, privacy, offline support—for engineers who have been dissatisfied with cloud APIs for various reasons, now is the best time to try local LLMs.

George Liu's guide goes beyond simple setup procedures, carefully covering how to connect with each agent and the realistic performance for different hardware. If you are interested in the original English text, please be sure to refer to the original article as well.

Your local AI environment begins with the single line: `ollama pull gemma4:4b`.

Reference Article: Running Google Gemma 4 With Ollama, Claude Code, OpenCode, Codex: Complete Local Setup by George Liu