GraphRAG运行报错

Could not automatically map qwen3:0.6b to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect

最近在学习GraphRAG，在配置完本地的环境后，对知识库生成索引时遇到了这个错误。网上搜索了一些没有发现清晰明确的答案，后来通过关键词，查询到了出问题的地方。

MODEL_TO_ENCODING，我查询这个关键词，发现库里面内容是下面这样的。

MODEL_TO_ENCODING: dict[str, str] = {
    # reasoning
    "o1": "o200k_base",
    "o3": "o200k_base",
    # chat
    "gpt-4o": "o200k_base",
    "gpt-4": "cl100k_base",
    "gpt-3.5-turbo": "cl100k_base",
    "gpt-3.5": "cl100k_base",  # Common shorthand
    "gpt-35-turbo": "cl100k_base",  # Azure deployment name
    # base
    "davinci-002": "cl100k_base",
    "babbage-002": "cl100k_base",
    # embeddings
    "text-embedding-ada-002": "cl100k_base",
    "text-embedding-3-small": "cl100k_base",
    "text-embedding-3-large": "cl100k_base",
    # DEPRECATED MODELS
    # text (DEPRECATED)
    "text-davinci-003": "p50k_base",
    "text-davinci-002": "p50k_base",
    "text-davinci-001": "r50k_base",
    "text-curie-001": "r50k_base",
    "text-babbage-001": "r50k_base",
    "text-ada-001": "r50k_base",
    "davinci": "r50k_base",
    "curie": "r50k_base",
    "babbage": "r50k_base",
    "ada": "r50k_base",
    # code (DEPRECATED)
    "code-davinci-002": "p50k_base",
    "code-davinci-001": "p50k_base",
    "code-cushman-002": "p50k_base",
    "code-cushman-001": "p50k_base",
    "davinci-codex": "p50k_base",
    "cushman-codex": "p50k_base",
    # edit (DEPRECATED)
    "text-davinci-edit-001": "p50k_edit",
    "code-davinci-edit-001": "p50k_edit",
    # old embeddings (DEPRECATED)
    "text-similarity-davinci-001": "r50k_base",
    "text-similarity-curie-001": "r50k_base",
    "text-similarity-babbage-001": "r50k_base",
    "text-similarity-ada-001": "r50k_base",
    "text-search-davinci-doc-001": "r50k_base",
    "text-search-curie-doc-001": "r50k_base",
    "text-search-babbage-doc-001": "r50k_base",
    "text-search-ada-doc-001": "r50k_base",
    "code-search-babbage-code-001": "r50k_base",
    "code-search-ada-code-001": "r50k_base",
    # open source
    "gpt2": "gpt2",
    "gpt-2": "gpt2",  # Maintains consistency with gpt-4
}

因为 tiktoken 主要针对 OpenAI 模型提供编码支持，像我在本地部署的deepseek 7b 和 qwen3:0.6b 这类非 OpenAI 模型，tiktoken 一般不提供原生支持。我尝试了一下手动加入，发现可以正常运行了，如果需要embedding的模型也同样加在里面即可。

"qwen3:0.6b":"cl100k_base",
"deepseek-r1:7b":"cl100k_base",
"nomic-embed-text:latest":"cl100k_base"

发现了另外的一种方法，可以在setting.yaml中进行设置。

# encoding_model: cl100k_base # automatically set by tiktoken if left undefined

对model和embedding取消这一部分的注释也可。