CANN/cann-recipes-infer盘古7B配置-CSDN博客

YAML Parameter Description

【免费下载链接】cann-recipes-infer 本项目针对LLM与多模态模型推理业务中的典型模型、加速算法，提供基于CANN平台的优化样例项目地址: https://gitcode.com/cann/cann-recipes-infer

The configuration instructions in the YAML file can be found below.

Basic Config
  model_name: "pangu_7B"                              # The model name. String type
  model_path: "/dev/shm/ckpts/openPangu-Embedded-7B" # The model path. String type
  exe_mode: "acl_graph"                              # The execution mode. Support ["ge_graph", "eager", "acl_graph"]
  world_size: 1                                      # The world size. Int type

Model Config
  mm_quant_mode: A16W16       # Support ["A16W16", "A8W8"]
  gmm_quant_mode: A16W16      # Support ["A16W16", "A8W8"]
  with_ckpt: True             # Whether load ckpt. Support [False, True]
  enable_profiler: False      # Whether enable profiling. Support [False, True]
  enable_cache_compile: False # Whether enable cache compile. Support [False, True]
  enable_weight_nz: False     # Whether enable weight NZ format. Support [False, True]
  enable_online_split_weight: True # Whether enable online split weight. Support [False, True]
  tokenizer_mode: "default"   # Support ["default", "chat"]

Data Config
  dataset: "default"   # Support ["default", "LongBench"]
  input_max_len: 2048  # The input max length
  max_new_tokens: 256  # The max new tokens
  batch_size: 1        # The global batch size

Parallel Config
  attn_tp_size: 1   # Attention TP Number
  moe_tp_size: 1    # MoE TP Number
  embed_tp_size: 1  # Embed TP Number
  lmhead_tp_size: 1 # LMHead TP Number

创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考