Waiting about 10 seconds for 30,000 tokens. How fast can BaseRT read long documents?
For short questions, Qwen3.6 starts responding quite quickly.
However, I wonder if it can be used with the same feeling when given long documents.
I wanted to look at not just the speed after it starts outputting text, but also the time from finishing reading the document until returning the first character.
This time, I provided documents ranging from about 129 tokens to about 32,813 tokens to Qwen3.6-35B-A3B Q4 running on BaseRT.
The longest document was about 58,000 characters.
To start with the results, the response began in just under 2 seconds for up to about 8,200 tokens.
At about 32,800 tokens, it takes about 10 seconds to get the first character.
The longer the text, the longer the wait time.
On the other hand, the amount that can be read per second increased significantly up to a certain point.
This is the interesting part this time.
🔰 Quick terminology memo first
📝 Token
A small unit used when AI processes text.
In Japanese, the number of characters and the number of tokens do not necessarily match.
📝 TTFT
The time from sending a question until the first character is returned.
I view this as a figure close to the user experience when using long documents.
📝 Prefill
The process where the AI reads the input document before creating an answer.
This time, I also checked how many tokens of long text can be read per second.
📝 Decode
The process of generating the answer text in sequence after reading is finished.
This is a different speed from Prefill.
1. The long text used this time
For the verification, I used fictional notes from my personal experiments with local AI and image generation.
Instead of just repeating the same paragraph, I combined records with slightly different content, such as BaseRT, Ollama, Q4 and Q8, image generation, and article notes.
At the end, I attached the same question to every document.
Based on the entire document, please provide 3 improvement policies for trying local AI personally, answering only in 3 numbered lines.
The prepared input consists of the following 5 levels.
Measured 129 tokens, 12 characters of document text
Measured 518 tokens, 695 characters of document text
Measured 2,051 tokens, 3,426 characters of document text
Measured 8,201 tokens, 14,352 characters of document text
Measured 32,813 tokens, 58,056 characters of document text
The 129-token round has almost no document text, serving as a baseline for comparison.
For the remaining 4 levels, the document length is gradually increased.
Thinking was disabled for Qwen3.6, and each condition was tested 3 times.
To reduce the impact of reading the same input consecutively, the model was reloaded for each measurement.
2. The first character
The median TTFT results were as follows.
129 tokens: 0.111 seconds
518 tokens: 0.187 seconds
2,051 tokens: 0.494 seconds
8,201 tokens: 1.728 seconds
32,813 tokens: 10.161 seconds
For about 500 tokens, it is less than 0.2 seconds.
Even at about 2,000 tokens, it takes about 0.5 seconds, so there is almost no sense of waiting.
At about 8,200 tokens, it took about 1.7 seconds to start the response.
There is a slight pause, but as a result of providing a long document, it feels sufficiently lightweight.
The change was significant at approximately 32,800 tokens.
It took about 10.2 seconds for the first character to appear, and about 11.6 seconds for a 3-line response to be completed.
At this point, I certainly don't feel like it 'came back immediately'.
However, considering that it finished answering in just over ten seconds after reading about 58,000 characters, it is a realistic wait time for personal verification purposes.
3. Does longer mean faster?
In the tentative title for this article, I posed the question, 'Do longer documents become faster?'
Looking only at the elapsed time, the answer is not faster.
The longer the document, the longer the wait time for the first character increased.
However, looking at the Prefill processing speed, a different trend emerges.
128 tokens: 1,963 tokens/s
512 tokens: 4,054 tokens/s
2,048 tokens: 5,655 tokens/s
8,192 tokens: 5,473 tokens/s
32,768 tokens: 3,903 tokens/s
The amount that can be read per second increased by about 2.9 times while increasing from 128 tokens to 2,048 tokens.
It seems that processing power is easier to utilize with a somewhat substantial input rather than one that is too short.
However, at 32,768 tokens, it dropped to 3,903 tokens/s.
Efficiency does not increase the longer you make it.
Under these conditions, the highest Prefill speed was reached around the 2,000 to 8,000 token range.
On the other hand, the decode speed of the built-in benchmark remained almost constant at approximately 160 tokens/s regardless of the input size.
In other words, the increase in time for long texts was mainly the waiting time before the response began.
The speed after the text started flowing was not significantly affected by the length of the input.
4. Responses with too little source material
Beyond speed, let's look at the content of the responses.
First, here is the response for 129 tokens, where the source text was only 12 characters long.
1. Select a high-performance Mac (M1 Pro or later) and maximize memory capacity to ensure model loading and inference speed.
2. Choose an appropriate model size (from around 7B to 70B or more) and quantization level based on the use case to balance local execution load and accuracy.
3. Utilize local inference software such as Ollama or LM Studio to optimize privacy and response speed by keeping everything within the local network rather than via API.
It adheres to the three-line format.
However, this is not a summary of the source material.
It supplements general knowledge not found in the source, such as "M1 Pro or later," "7B to 70B or more," and "Ollama or LM Studio."
Because there was insufficient material, Qwen3.6 generated a plausible response based on the question.
The speed was 0.11 seconds, but what I wanted this time was not a fast general theory.
This serves as an example that shorter input is not always easier to use.
5. Responses consistent at 500 tokens
When the source material was increased to 518 tokens, the response leaned toward the content within the material.
1. Measure the number of input tokens before comparing.
2. Distinguish between violations of response format and generation failures.
3. Clearly state the premises of the comparison in the article.
This response was identical word-for-word all three times.
It concisely captures the improvement points that appear repeatedly in the material and answers the question directly.
It took 0.19 seconds to reach the first character, and about 1.06 seconds for the entire response.
For this task, it seems that about 500 tokens were sufficient to provide the necessary material.
6. Responses from 2,000 tokens onwards
This is a representative response at 2,051 tokens.
1. Even if there is no dramatic difference, state the actual measured results as they are, and list target values alongside measured values to reduce ambiguity in comparisons.
2. To improve reproducibility, try the same question multiple times and look at the median, or do not pad long documents with repeated text.
3. Separate speed figures from text evaluation, treat results including cache as a separate category, and consider suitable use cases rather than model superiority.
As the document became longer, it not only picked up words but also summarized the improvement policies and reasons together.
The three responses were almost identical.
At 8,201 tokens, the following response was returned.
1. Even if there is no dramatic difference, state the actual measured results as they are, and list target values alongside measured values to reduce ambiguity in comparisons.
2. To improve reproducibility, measure the same question multiple times and look at the median to increase confidence in the results.
3. In long-text evaluation and speed comparisons, do not pad with repeated text, but verify by only increasing the input length.
This also did not deviate from the core of the document.
The second and third times were the same response, but only the first time the content of the third point was slightly different.
Even as the document increased, it was organized into the three specified lines.
7. Responses at 30,000 tokens
At the maximum of 32,813 tokens, it did not stop midway in any of the three attempts and returned the three specified lines.
This is a representative response.
1. To improve reproducibility, measure the same question multiple times (three times) and state the median or actual measured values as they are.
2. To eliminate the influence of cache, either terminate the model and restart it, or reload the model.
3. To reduce ambiguity in comparisons, fix the output conditions while changing only the input, and describe the target values alongside the measured values.
The first and second responses were the same.
The third time, it changed to slightly shorter content as follows.
1. State the actual measured results as they are even if there is no dramatic difference
2. Perform multiple measurements to improve reproducibility and look at the median
3. State the premises of the comparison in the article and consider suitable use cases rather than model superiority
Both are contents found within the document.
However, the three items selected from the long document were not completely fixed every time.
The longer the document, the more potential points of discussion there are.
There is still some fluctuation in what it determines to be important from within that.
Being able to read a long text to the end and choosing the same key points every time are different things.
This was a part that could not be understood by speed alone.
8. My verdict this time
The Qwen3.6-35B-A3B Q4 on BaseRT returned the first character within 2 seconds for up to about 8,200 tokens.
Even with about 32,800 tokens, it took about 10 seconds for the first character and about 12 seconds to complete the response.
It seems perfectly usable for purposes like handing over long personal notes all at once to have them organized just once.
On the other hand, using it by attaching 30,000 tokens to everyday casual consultations makes both the wait time and the document preparation burdensome.
Based on my experience this time, the range of 2,000 to 8,000 tokens was the easiest to balance between the amount of material and wait time.
And if the material is too short, even if it is fast, it ends up supplementing with generalities.
Rather than shortening the wait time, including the necessary materials to answer was more important.
On Thursday, I will move away from speed and try long-text summarization, condition organization, and code correction.
I plan to see if a 35B local AI can move actual work forward in a single pass.
Tami
#BaseRT #Qwen3_6 #LocalLLM #MacBookPro #M5Max #GenerativeAI #LongTextSummarization