We have released "ELYZA-japanese-Llama-2-7b," a commercially usable Japanese LLM based on Meta's "Llama 2"
Summary of this article
ELYZA has publicly released "ELYZA-japanese-Llama-2-7b," a commercially usable Japanese LLM based on "Llama 2"
Performance is comparable to "GPT-3.5 (text-davinci-003)," reaching the highest level among publicly available Japanese models
Chat format demos and evaluation datasets have also been released
Development of 13 billion and 70 billion parameter models is already underway in-house
Introduction
Hello. We are Sasaki, Nakamura, Hirakawa, and Horie from the ELYZA research and development team.
ELYZA has now publicly released "ELYZA-japanese-Llama-2-7b," a Japanese language model based on Meta's "Llama 2" that has undergone additional pre-training in Japanese; "ELYZA-japanese-Llama-2-7b-instruct," which has been subjected to ELYZA's unique post-training; and "ELYZA-japanese-Llama-2-7b-fast / ELYZA-japanese-Llama-2-7b-fast-instruct," which have been accelerated through the addition of Japanese vocabulary. All are 7 billion parameter models, making them among the largest scale Japanese LLMs currently available to the public.
The license complies with the Llama 2 Community License, and as long as the Acceptable Use Policy is followed, it can be used for research and commercial purposes.
You can use the demo from the links below. Please feel free to try it out.
● ELYZA-japanese-Llama-2-7b-instruct Demo
● ELYZA-japanese-Llama-2-7b-fast-instruct Demo
*The demo was closed on Thursday, April 4, 2024. The models themselves continue to be available on the Hugging Face Hub.
What is Llama 2?
Llama 2 is an English-based large language model released by Meta on July 18, 2023. While the previously released "LLaMA" was limited to research use, Llama 2 is also available for commercial use. Because it is a very high-performance model among those publicly available, it is already becoming the de facto standard for open models in the English-speaking world, competing with closed LLMs such as OpenAI's GPT-4 and Google's PaLM.
There are three sizes: 7 billion, 13 billion, and 70 billion parameters, and for all models, chat models that have undergone Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) have also been released simultaneously.
Our hopes for this model release
Currently in Japan, multiple companies are working on the development of their own Japanese LLMs, but compared to Meta's Llama 2, which was trained on 2 trillion tokens of text, the current reality is that they remain small-scale. The background to this includes a lack of computational resources and a scarcity of text data available in Japanese. Furthermore, because training an LLM from scratch requires enormous costs, only a few large companies and research institutions are able to conduct such research.
Under these circumstances, ELYZA has focused on the Japanization of multilingual LLMs, believing that by inheriting the capabilities of LLMs trained in other languages, such as English, into Japanese, and reducing the amount of training required in Japanese, we can accelerate the research and development of Japanese LLMs.
As one of the results of this project, we have succeeded in developing a model with improved Japanese capabilities based on Meta's Llama 2, and have decided to release a portion of it. In a technical blog to be published soon, we plan to share details about the knowledge and know-how gained while Japanizing Llama 2.
We have already begun development of 13 billion and 70 billion parameter models, and we are considering releasing those as well. Through the release of models and know-how, ELYZA aims to accelerate Japanese LLM research by creating an environment where laboratories, startups, and individuals can also engage in the research and development of Japanese LLMs.
Released models
ELYZA-japanese-Llama-2-7b
This is a model that has undergone additional pre-training on Meta's Llama-2-7b-chat using approximately 18 billion tokens of Japanese text. The Japanese text data used for training includes content from OSCAR, Wikipedia, and other sources.
ELYZA-japanese-Llama-2-7b-instruct
This is a model that has undergone post-training on ELYZA-japanese-Llama-2-7b with the goal of solving various tasks according to user instructions. For post-training, we used ELYZA's own high-quality instruction dataset, which also enables support for multi-turn conversations. Note that the post-training at ELYZA does not include any output from GPT-4, GPT-3.5-turbo, or similar models.
ELYZA-japanese-Llama-2-7b-fast / ELYZA-japanese-Llama-2-7b-fast-instruct
This is a model that has undergone pre-training with added Japanese vocabulary for Llama 2. Because the original Llama 2 has a limited Japanese vocabulary, it required more tokens to represent Japanese sentences compared to English, which led to issues such as shorter input/output sentence lengths and slower inference speeds.
Therefore, we have newly added 13,042 Japanese vocabulary items, and as a result, we were able to reduce the number of tokens required to represent the same Japanese sentences to approximately 55%. In terms of inference speed, this translates to approximately 1.82 times faster, representing a significant success in efficiency. We are also releasing "ELYZA-japanese-Llama-2-7b-fast-instruct," which has undergone post-training on ELYZA-japanese-Llama-2-7b-fast, at the same time.
Performance Evaluation
ELYZA Tasks 100
At ELYZA, we have created and used our own unique dataset consisting of diverse Japanese tasks to evaluate the performance of the Japanese LLMs we develop in-house. The reason we do this is that we believe existing evaluation datasets lack the diversity and complexity of tasks required to fully measure general language capabilities like those of ChatGPT. Furthermore, since automated evaluation metrics cannot accurately assess the performance of generative AI, human evaluation is ultimately necessary. Therefore, at ELYZA, we limited the number of tasks to a manageable amount for human evaluation (100 items) and instead manually created a dataset containing diverse and complex tasks.
We have named this dataset "ELYZA Tasks 100" and are releasing it to the public alongside the models. We have also published detailed evaluation criteria and the code used for inference with each model, so please feel free to make use of them.
"ELYZA Tasks 100" can be accessed from this repository.
The results of a 5-point human evaluation using ELYZA Tasks 100 are shown below. For the evaluation, three people conducted a blind test with the model names hidden and shuffled, and the scores were calculated by averaging their results.
As a result of the human evaluation, ELYZA-japanese-Llama-2-7b-instruct achieved the highest score compared to other publicly available Japanese models. ELYZA-japanese-Llama-2-7b-fast-instruct also achieved a comparable score, making it a viable model choice for applications where speed is a priority.
Below, we show some examples of actual output from ELYZA-japanese-Llama-2-7b-instruct.
The first example is a brainstorming task, where the model successfully provided five different ideas to the user as instructed. Regarding the output format, it uses a numbered list + heading + description format, which is designed to be easy for the user to read.
The second example is a task to determine whether two sentences are paraphrases of each other, with the instructions given in a way close to how an actual user would ask. In response, the ELYZA model accurately interpreted the meanings of sentence 1 and sentence 2 and correctly determined that they were different.
For reference, we also show the scores of closed LLMs such as OpenAI's GPT-4 and GPT-3.5-turbo, and Google's PaLM below.
Although ELYZA-japanese-Llama-2-7b-instruct does not yet reach the level of closed LLMs, it has achieved the highest score among open Japanese models. At ELYZA, we will continue our research and development with the aim of further improving performance.
Note that while the ELYZA Tasks 100 dataset was not used at all in the training of the ELYZA models, the ELYZA Tasks 100 scores were used in part for model selection, so it is possible that this worked slightly in favor of the ELYZA models.
For specific outputs and evaluation results for each model, please refer to this spreadsheet.
lm-evaluation-harness
In addition, the evaluation results using lm-evaluation-harness, which is commonly used for evaluating the performance of Japanese models, are shown below.
ELYZA-japanese-Llama-2-7b-instruct scores higher than the original Llama-2-7b-chat in all categories. This confirms that it has indeed acquired Japanese language capabilities through additional pre-training and post-training in Japanese. Furthermore, it can be seen that the model for which we performed additional pre-training at ELYZA achieves scores comparable to other Japanese models, despite having been trained on fewer Japanese tokens.
On the other hand, our internal verification has shown that while such general datasets can measure Japanese knowledge and basic reading comprehension, they are unsuitable for measuring the ability to follow user instructions. In fact, despite a significant difference in the ability to follow instructions between the post-trained ELYZA-japanese-Llama-2-7b-instruct and the non-post-trained ELYZA-japanese-Llama-2-7b, they achieve similar scores in the lm-evaluation-harness evaluation results.
How to use
All four models released this time are available on the Hugging Face Hub and can be used via the transformers library.For detailed usage instructions, please see the README at the following links.
https://huggingface.co/elyza/ELYZA-japanese-Llama-2-7b
https://huggingface.co/elyza/ELYZA-japanese-Llama-2-7b-instruct
https://huggingface.co/elyza/ELYZA-japanese-Llama-2-7b-fast
https://huggingface.co/elyza/ELYZA-japanese-Llama-2-7b-fast-instruct
Future Outlook
The "ELYZA-japanese-Llama-2-7b" series released this time was developed based on the 7-billion parameter model, which is the smallest size of Llama 2. Llama 2 also has 13-billion and 70-billion parameter models, and ELYZA has already begun working on Japanese versions of those models as well. We will continue development so that we can deliver even more powerful models in the near future. Furthermore, we will continue to invest not only in Llama 2 initiatives but also in the Japanese localization of open models from overseas and the development of our own proprietary large language models.
Moving forward, ELYZA will continue to lead the development of domestic generative AI. As a leading company in domestic generative AI, we will contribute to society by releasing the results we obtain in a commercially available format or through corporate projects, with the goal of accelerating the use of LLMs and improving LLM technical capabilities throughout Japan.
The details of this initiative were also presented at the "18th Symposium of the NLP Young Researchers Association (YANS) (2023)" held on August 30th and 31st.
Related Articles
Our engineers are releasing a series of technical explanation blogs regarding "ELYZA-japanese-Llama-2-7b." Please take a look at these as well. ●Published September 12, 2023
(1) Pre-training Edition
●Published September 26, 2023(2) Evaluation Edition
●Published January 16, 2024(3) English Performance Evaluation Edition
Conclusion
Based on the philosophy of "Creating the ordinary in uncharted territory," ELYZA, Inc. focuses on large language models for the Japanese language, conducting joint research with companies and developing cloud services. Through research and development of cutting-edge technology and consulting, we promote the implementation of generative AI in a way that contributes to corporate growth.
Thank you for reading this far. ELYZA is looking for colleagues to help move our business forward in various roles, such as AI engineers and AI consultants. If you are even slightly interested, please come for a casual interview.
Would you like to talk about LLM development, including Japanese Llama 2?
https://chillout.elyza.ai/