【BUG】MMLU数据集huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url

问题

用4张A100跑MMLU数据集有如下报错

[rank2]:   File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 475, in hf_raise_for_status
[rank2]:     raise _format(HfHubHTTPError, str(e), response) from e
[rank2]: huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://hf-mirror.com/api/datasets/hails/mmlu_no_train/tree/b2e1ec9aa795adafe68e8e983248dbd4b52a1c60/machine_learning?recursive=True&expand=False
W0206 22:29:22.137000 3228245 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3228292 closing signal SIGTERM
E0206 22:29:22.231000 3228245 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 3228293) of binary: /data/home/qp252478/.conda/envs/llm_gpu/bin/python3.10
Traceback (most recent call last):
  File "/data/home/**/.conda/envs/llm_gpu/bin/accelerate", line 7, in <module>
    sys.exit(main())
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1165, in launch_command
    multi_gpu_launcher(args)
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher
    distrib_run.run(args)
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data/home/**/.conda/envs/llm_gpu/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
eval_llada.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2026-02-06_22:29:22
  host      : sbg4.apocrita
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 3228294)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2026-02-06_22:29:22
  host      : sbg4.apocrita
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 3228295)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-02-06_22:29:22
  host      : sbg4.apocrita
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 3228293)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

原因

使用 accelerate launch 进行多卡（Distributed）推理。当代码运行到加载数据集这一步时，每一张显卡（每一个进程）都同时向镜像站发送了下载/查询请求。例如，如果你有 8 张卡，就是瞬间发起 8 次完全相同的请求。镜像站（hf-mirror.com）的防火墙把这种并发请求识别为攻击或滥用，从而拒绝了连接。

解决方案

• 激活环境

conda activate llm_gpu

• 设置镜像站（如果需要）：既然你的报错里提到了 hf-mirror.com，说明你可能配置了镜像。为了保证 Login 节点能连上，建议再设一次（以防万一）：

export HF_ENDPOINT=https://hf-mirror.com

• 执行下载代码（单进程）

# 文件名建议：download_mmlu.py
from datasets import load_dataset, get_dataset_config_names

dataset_name = 'hails/mmlu_no_train'

print(f"正在获取 {dataset_name} 的配置列表...")
try:
    # 获取所有子学科列表（如 math, history 等）
    configs = get_dataset_config_names(dataset_name)
except Exception as e:
    print(f"获取列表失败: {e}")
    exit()

print(f"共发现 {len(configs)} 个子集，开始下载...")

# 循环下载每一个子集
for config in configs:
    print(f"正在下载: {config} ...", end="", flush=True)
    try:
        # 因为你刚删了缓存，这里普通的 load_dataset 就会自动触发下载
        load_dataset(dataset_name, config) 
        print(" ✅ 完成")
    except Exception as e:
        print(f" ❌ 失败: {e}")

print("\n🎉 所有数据下载完毕！现在可以在多卡环境使用了。")