What I learned from training my own LLMs on a home GPU and conducting performance diagnostic tests on five LLMs

My self-made (AI-created) 130M-size LLM, which I only trained for about 3 hours on an RTX 3090, beat the Japanese GPT-2 type Trans-LG—which is about 3 times the size and had about 12 times the training volume—with 81.27% vs 77.95% on the public grammar benchmark (JBLiMP)... Seriously? The GPT-5.6 design is amazing... Alright, I'm going to continue training this model and help it grow.

I compared four LLMs I created myself and one publicly available model using the same problem set.

The evaluation used 12 types of 4-choice questions, totaling 768 questions, measuring search, information updates, ordering, multi-step reasoning, aggregation, and resistance to distractor information. All models were given the same prompt and options, and the answer was determined by calculating the probability of each option following the prompt. Since there are 4 choices, the expected value for guessing randomly without knowing anything is 25%, or 192 correct answers.

This problem set is a diagnostic test for my own mechanisms and is not a public benchmark for measuring general language ability. Additionally, the number of parameters, training volume, training data, and tokenizer differ for each model. Therefore, the following results should not be read as a fair ranking based solely on architecture, but as a current comparison of models actually built on a personal RTX 3090.

Results

| Model | Parameters | Training tokens | Correct | Accuracy | Training speed |
|---|---:|---:|---:|---:|---:|
| CST Prime | 129.9M | 500.0M | 230/768 | 29.95% | 45,242 tok/s |
| KOTODAMA | 130.0M | 500.0M | 210/768 | 27.34% | 9,381 tok/s |
| SmolLM2 | 135M class | 2T | 198/768 | 25.78% | Unknown due to external training |
| KaiNomos | 718.3M | 45.9M | 191/768 | 24.87% | 3,487 tok/s |
| Deltaxis | 249.7M | 97.5M | 188/768 | 24.48% | 3,585 tok/s |

The highest score this time was CST Prime. It outperformed KOTODAMA, which has almost the same number of parameters and was trained on almost the same 500M tokens, by 20 questions. However, in a test comparing corresponding questions, the result was `p=0.181`, so it is not certain that CST Prime's ability is clearly superior based on these 768 questions alone.

Even so, when viewed as a candidate for a personal development product, the difference is not just in the number of correct answers. CST Prime was able to train at about 4.8 times the speed of KOTODAMA. The GPU computation time recorded up to the 500M point was about 3.0 hours for CST Prime and about 12.4 hours for KOTODAMA. This means CST Prime achieved a higher score in a shorter amount of time.

Total scores alone do not reveal model characteristics

When using the same problem set, the models differed significantly in which questions they were good at and how they made mistakes.

CST Prime was strong at updating to new information

`overwrite_latest` is a problem where you answer the final value after the value for the same subject has been changed multiple times.

• CST Prime: 40/64

• KOTODAMA: 10/64

• SmolLM2: 0/64

Looking at the answers in detail, KOTODAMA chose the initial value before the update in 46 questions, and SmolLM2 chose the initial value in 58 questions. In contrast, CST Prime was able to choose the final updated value in 40 questions. CST Prime not only had a slightly higher total score, but also showed a clear difference in its ability to prioritize new information over old information.

However, it is impossible to tell from these results alone which of CST's rotational residuals, attention gate, Late Value Recall, or multi-activation MLP were effective. What we know this time is that the completed model combining these was strong in information update problems.

KOTODAMA was relatively strong in ordering relationships

KOTODAMA scored 37/64 in `constraint_ordering`, which restores the overall order from multiple conditions like "A is before B," making it the best among the 5 models. It also scored 13/64 in `position_order`, which directly answers the position in a sequence, outperforming CST Prime's 1/64.

KOTODAMA has a sequential state that processes text from left to right and a structure that repeatedly uses the same processing circuit. The trend in this ordering problem is consistent with its design intent. However, it cannot be concluded that the structure is the cause based on only 64 questions from a single checkpoint.

On the other hand, KOTODAMA was strongly pulled toward the pre-update value in the information update problem. It is clear that maintaining order and discarding old information to replace it with new information are not the same memory capability.

SmolLM2 had high resistance to distractor information.

`distractor_stability` is a problem that checks whether correct information can be maintained even when irrelevant sentences are increased around the necessary record.

• SmolLM2: 50/64

• CST Prime: 31/64

• KOTODAMA: 28/64

Although SmolLM2 had an overall score of 25.78%, it won significantly in resistance to distractor information. It is possible that the ability to distinguish important information from irrelevant information within a text was cultivated through a large amount of pre-training of 2T tokens. However, since the training data and tokenizer are also different, this cannot be said to be an effect of the architecture.

Conversely, SmolLM2 had 0/64 for updating to the latest value and 0/64 for multi-step tracking. In this self-made test, even external models trained for a long period did not acquire all operations equally.

It was important not only to look at the scores but also at 'how they got it wrong'.

In the multi-step tracking problem, I had them answer 'the destination after following the arrows from A several times'. However, KOTODAMA and SmolLM2 answered the starting point as is for all 64 questions, and CST Prime also answered the starting point for 60 questions. The models did not choose the options randomly; they were strongly pulled toward the starting point immediately preceding the question sentence without actually following the arrows.

The same phenomenon was seen in the position problem. CST Prime chose the first item in the column instead of the specified position in 46 out of 64 questions, and SmolLM2 did so in 53 questions. Simply summarizing it as 'weak position recognition' causes one to overlook what the model is actually doing. In reality, instead of counting to find the position, they were biased toward the shortcut of answering the beginning of the input.

Also, models that could answer more than 30 questions correctly in the problem of restoring order from constraints were almost completely wiped out in the multi-step arrow tracking. Both look like similar relational reasoning to humans, but for the models, the representation formats are different. The scores by family this time measure not only pure thinking ability but also whether they are accustomed to the format of the question sentences.

For this reason, the overall score of the self-made problem set cannot be treated as a ranking of general intelligence as is. On the other hand, if you compare how they got the same questions wrong, it can be used as a diagnostic tool to investigate which information remains inside the model and where the processing stopped.

Training time increases rapidly for complex models.

The biggest practical difference this time was the learning speed.

CST Prime has a structure that passes through 12 layers at once, allowing it to use GPU-oriented compilation and large microbatches. As a result, it could learn at about 45,000 tok/s, taking about 3 hours to reach 500M tokens.

KOTODAMA has the same ~130M parameters, but it repeats 8 shared blocks multiple times. Even if the number of parameters does not increase, the number of calculations to process 1 token increases. Therefore, the learning speed dropped to about 9,400 tok/s, and it took about 12.4 hours to reach 500M. Furthermore, since the average number of iterations increases in the latter half of training, it is expected to take about 14.8 hours for the additional 500M to reach 1B.

Deltaxis applies 16 iterations of 2 shared blocks in addition to a 14-layer main body. KaiNomos is a large model with 718M parameters and 24 layers. For both, the learning speed is about 3,500 tok/s, and to advance from the current point to 1B, it will require about 70 hours and about 76 hours of additional computation, respectively.

Even with the same number of parameters, training time varies greatly depending on the number of iterations and the implementation method. When comparing performance, it is necessary to look not only at the training tokens and the number of parameters but also at how many hours the GPU was actually used.

Even for a large model, if it is undertrained, it will not show its capabilities.

KaiNomos has 718M parameters, but at the time of evaluation, it had only been trained on 45.9M tokens. Deltaxis also had 97.5M tokens for 249.7M parameters. The total scores for both models were near the level of random chance for a 4-choice test.

This shows that simply building a large model does not immediately make it strong. As you increase the number of parameters, the amount of training data and time required to bring out its capabilities also increase. With a personal GPU, if you make the model too large, you may run out of compute budget before it is sufficiently trained.

KaiNomos scored 15/64 in `multi_hop` and 22/64 in `previous_value`. Deltaxis scored 20/64 in `state_transition`. However, since the total scores are at the level of random chance, these cannot yet be judged as their specialized capabilities after completion. At this stage, these are initial records for comparing future growth.

Which models to keep as a personal developer

Based on these results, I can organize how to handle them going forward as follows.

• CST Prime will continue as the flagship model. It has the highest current score and its training speed is overwhelmingly fast. Even with additional training up to 1B, it only takes about 3 hours.

• KOTODAMA will continue up to 1B, and a final decision will be made then. It is slower than Prime, but its score at the 500M point is close, and it has a different strength in sequential relationships. However, if it does not outperform Prime even at 1B, I will remove it from the flagship lineup considering the compute costs.

• KaiNomos will save its current checkpoint, and long-term training will be stopped. It is not yet sufficiently trained, so it cannot be called a structural failure, but it requires about 76 hours to reach 1B, which is too heavy as a priority product for a personal RTX 3090.

• Deltaxis will also be saved as a research asset, and long-term training will be stopped. Despite 16 iterations of refinement calculations, there are no clear results for the multi-step reasoning I was aiming for at this point. The cost of adding about 70 hours to reach 1B is not worth the current signals.

• SmolLM2 will remain as an external reference line. It is not a fair competitor because the training conditions are different, but it serves as a benchmark to see in which capabilities my custom models win or lose against external finished products.

The models ending here are not 'scientifically disproven' in their architecture. There remains a possibility that they would improve if trained sufficiently. However, in personal development, I cannot add dozens of hours of GPU computation based on possibility alone. I need to decide where to use limited GPU time by combining current performance, signs of growth, and training speed.

What I learned this time

Model performance is not determined solely by the number of parameters or training tokens. Depending on how residuals are used, how information is updated, how sequences are maintained, and how many times the same calculation is repeated, both the problems it excels at and its training speed change.

Also, adding complex mechanisms does not necessarily improve performance. Even if capability increases slightly through iterative calculation, if the training time becomes several to dozens of times longer, it becomes impossible to train a sufficient amount in a personal computing environment. Not just theoretical expressive power, but whether it can be raised to completion is also part of model design.

In this comparison, differences between models appeared more in information updates, sequences, and interference resistance than in simple total scores. And the most realistic one was the model that achieved the highest score while training the fastest.

When building an LLM personally, a good model is not a complex one. It is a model that can be raised to the end on a limited GPU and gain capabilities commensurate with that computation time.

Postscript:

The custom (AI-made) LLM CST Prime-130M recorded 269/331 questions, an 81.27% accuracy rate, on the Japanese grammar benchmark JBLiMP at the 500M token training point. In terms of score, it surpassed the highest value of 77.95% for a Japanese GPT-2 type model published in the JBLiMP paper, and the difference from the human standard of 90.90% was about 9.6 points. However, the breakdown was a big win in Japanese structure and a big loss in instruction relationships.

The Japanese GPT-2 type model 'Trans-LG', which achieved the highest score in the JBLiMP paper, is estimated to have about 400 million parameters.

The paper does not directly state the total number of parameters, but calculating from the published configuration, it is about 406 million. It is about 3.1 times that of CST Prime.

Furthermore, Trans-LG repeatedly uses about 146 million subwords of Japanese data, and since it is 61,440 tokens × 100,000 updates, it has processed a total of about 6.1 billion tokens. On the other hand, CST Prime is about 500 million tokens.Original model paper, JBLiMP paper

Therefore, the current situation is,

which makes for a quite interesting comparison. However, the total parameter count is an estimate based on the configuration described in the paper, not an actual measurement from the checkpoint.