使用huggingface pipeline实现英文到中文的翻译

原创已于 2024-02-16 10:55:36 修改 · 436 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

收录于

huggingface

当前文章被收录于：

huggingface

33 篇文章 0 人学习

huggingface

订阅专栏查看详情

当前文章被以下社区和专栏收录：

于 2024-02-12 11:24:35 首次发布

1 ：以下翻译英文到中文，可以CPU运行。

from huggingface_hub.hf_api import HfFolder
HfFolder.save_token('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')
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_en = "The head of the United Nations says there is no military solution in Syria"
model = MBartForConditionalGeneration.from_pretrained("SnypzZz/Llama2-13b-Language-translate")
tokenizer = MBart50TokenizerFast.from_pretrained("SnypzZz/Llama2-13b-Language-translate", src_lang="en_XX")
model_inputs = tokenizer(article_en, return_tensors="pt")
# translate from English to Chinese
generated_tokens = model.generate(
    **model_inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["zh_CN"]
)
print(tokenizer.batch_decode(generated_tokens, skip_special_tokens=True))