在超算上启动h5bench

最新推荐文章于 2026-06-25 20:44:04 发布

原创最新推荐文章于 2026-06-25 20:44:04 发布 · 406 阅读

3 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

标签

#chrome #前端 #javascript

Python3.8

Python 是一种高级、解释型、通用的编程语言，以其简洁易读的语法而闻名，适用于广泛的应用，包括Web开发、数据分析、人工智能和自动化脚本

🚀 东方超算系统上编译安装 HDF5 1.14.6（支持 zlib + szip + MPI）

本教程假设你在东方超算（登录节点）上，默认已加载 gcc 编译器，
并通过 module load 启用 OpenMPI 与 ZLIB 模块。

🧰 1. 环境准备

1.1 加载编译环境模块

# 加载编译器和 MPI
module load compiler/devtoolset/7.3.1
module load mpi/openmpi/4.0.4/gcc-7.3.1

# 加载 ZLIB 库（用于 GZIP 压缩）
module load mathlib/zlib/1.2.8/gcc

验证环境：

mpirun --version

输出示例：

mpirun (Open MPI) 4.1.4rc1

📦 2. 下载源码

在登录节点中执行：

cd ~/hdf5
scp hdf5-1.14.6.tar.gz .
tar -xzf hdf5-1.14.6.tar.gz
cd hdf5-1.14.6

🧩 3. 安装 SZIP 库（启用 szip encoder 所必需）

HDF5 的 SZIP 压缩算法不是默认包含的，需要单独安装。

scp 
cd szip-master
./configure --prefix=$HOME/szip-install
make -j8
make install

验证：

ls $HOME/szip-install/lib

应看到：

libsz.a  libsz.la  libsz.so  libsz.so.2  libsz.so.2.0.0

⚙️ 4. 配置 HDF5 编译选项

在解压的 hdf5-1.14.6 目录中，执行以下命令：

cd ~/hdf5-1.14.6
make clean

CPPFLAGS="-I$HOME/szip-install/include" \
LDFLAGS="-L$HOME/szip-install/lib" \
./configure \
  --prefix=$HOME/hdf5-1.14.6-install \
  --enable-parallel \
  --enable-shared \
  --enable-hl \
  --with-zlib \
  --with-szlib=$HOME/szip-install

我们可以看到

Features:
---------
                     Parallel HDF5: yes
  Parallel Filtered Dataset Writes: yes
                Large Parallel I/O: yes
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib),szip(no encoder)

🏗️ 5. 编译与安装

make -j32
make install
``

成功后提示：

Libraries have been installed in:
/public/home/dfzx202411/hdf5-1.14.6-install/lib


---

## 🔍 6. 验证功能是否启用

```bash
$HOME/hdf5-1.14.6-install/bin/h5pcc -showconfig

🧠 7. 设置环境变量（可添加到 ~/.bashrc）

export HDF5_HOME=$HOME/hdf5-1.14.6-install
export SZIP_HOME=$HOME/szip-install
export PATH=$HDF5_HOME/bin:$PATH
export LD_LIBRARY_PATH=$HDF5_HOME/lib:$SZIP_HOME/lib:$LD_LIBRARY_PATH

验证命令：

which h5pcc

应返回：

/public/home/dfzx202411/hdf5-1.14.6-install/bin/h5cc

🚀 东方超算系统上编译安装H5Bench 并运行

首先把代码放上去，代码链接了四个子模块也要一起放上去。

mv ./E3SM-IO-76a1c2fabd042423493a455ef3ce72577a3848a2/* ./h5bench-master/e3sm/
mv ./MACSio-7f19af30b7753d2980fd6b833e831ff4b42b4fb0/*  ./h5bench-master/macsio/
mv ./openPMD-api-32cb87d1b0012493287204d7a8c78af9a1141710/*  ./h5bench-master/openpmd/
mv ./amrex-9c412974de2ac0496878acf1f7e8d6c426f535e7/* ./h5bench-master/amrex/

然后编译

这里需要一个高版本的cmake

 module load compiler/cmake/3.24.1

mkdir build
cd build

cmake  -DCMAKE_INSTALL_PREFIX=$HOME/h5bench_install ..

make
make install

python3

现在的报错显示缺少python3，好像module load 没有，装个miniconda好了。
但是miniconda不能用最新的，glibc只有2.17，所以需要去找个旧版本，我找到的是4.5.1 2018年的miniconda ，base环境自带一个python3.6.5 够用了

初步成功

salloc 了一个节点，测试单节点写入成功，明天开始测试大规模。

参考脚本

[dfzx202411@login10 scripts]$ cat /public/home/dfzx202411/h5bench_test/configs/task1_1nodes.json
{
    "mpi": {
        "command": "mpirun",
        "ranks": "1",
        "configuration": "-np 1"
    },
    "vol": {      },
    "file-system": {
        "lustre": {
            "stripe-size": "1M",
            "stripe-count": "4"
        }
    },
    "directory": "/public/home/dfzx202411/h5bench_test/data",
    "benchmarks": [
        {
            "benchmark": "write",
            "file": "test.h5",
            "configuration": {
                "MEM_PATTERN": "CONTIG",
                "FILE_PATTERN": "CONTIG",
                "NUM_PARTICLES": "16 M",
                "TIMESTEPS": "5",
                "DELAYED_CLOSE_TIMESTEPS": "2",
                "COLLECTIVE_DATA": "NO",
                "COLLECTIVE_METADATA": "NO",
                "EMULATED_COMPUTE_TIME_PER_TIMESTEP": "1 s",
                "NUM_DIMS": "1",
                "DIM_1": "16777216",
                "DIM_2": "1",
                "DIM_3": "1",
                "MODE": "SYNC",
                "CSV_FILE": "output.csv"
            }
        }
    ]
}

参考效果

[dfzx202411@g09r1n19 ~]$ $HOME/h5bench_install/bin/h5bench -d  /public/home/dfzx202411/h5bench_test/configs/task1_1nodes.json
2025-11-10 23:30:25,817 h5bench - INFO - Starting h5bench Suite
2025-11-10 23:30:25,821 h5bench - WARNING - Base directory already exists: /public/home/dfzx202411/h5bench_test/data
2025-11-10 23:30:25,826 h5bench - INFO - Lustre support not detected
2025-11-10 23:30:25,826 h5bench - DEBUG - LD_LIBRARY_PATH: /public/home/dfzx202411/hdf5-1.14.6-install/lib:/public/home/dfzx202411/szip-install/lib:/public/software/compiler/dtk/dtk-22.10.1/lib:/public/software/compiler/dtk/dtk-22.10.1/lib64:/public/software/compiler/dtk/dtk-22.10.1/hip/lib:/public/software/compiler/dtk/dtk-22.10.1/llvm/lib:/public/software/compiler/dtk/dtk-22.10.1/miopen/lib:/opt/hpc/software/compiler/intel/intel-compiler-2017.5.239/compiler/lib/intel64_lin/:/opt/hpc/software/mpi/hpcx/v2.11.0/gcc-7.3.1/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/sharp/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/hcoll/lib:/opt/hpc/software/mpi/hpcx/v2.11.0/ucx_without_rocm/lib:/opt/rh/devtoolset-7/root/usr/lib64:/opt/rh/devtoolset-7/root/usr/lib:/opt/hpc/software/mpi/pmix/lib:/opt/hpc/software/mpi/hpcx/v2.7.4/ucx_without_rocm/lib:/opt/hpc/software/mpi/hpcx/v2.7.4/ucx_without_rocm/lib/ucx:/opt/hpc/software/mathlib/blas/hygon/0831/lib/:/opt/gridview/slurm/lib:/opt/gridview/slurm/lib64:/opt/gridview/munge/lib:
2025-11-10 23:30:25,827 h5bench - DEBUG - DYLD_LIBRARY_PATH:
2025-11-10 23:30:25,827 h5bench - DEBUG - LD_PRELOAD:
2025-11-10 23:30:25,827 h5bench - INFO - JOBID: 17289244
2025-11-10 23:30:25,827 h5bench - INFO - h5bench [write] - Starting
2025-11-10 23:30:25,827 h5bench - INFO - h5bench [write] - DIR: /public/home/dfzx202411/h5bench_test/data/a57cbf19-17289244/
2025-11-10 23:30:25,830 h5bench - INFO - Parallel setup: mpirun -np 1
2025-11-10 23:30:25,832 h5bench - INFO - mpirun -np 1 /public/home/dfzx202411/h5bench_install/bin//h5bench_write /public/home/dfzx202411/h5bench_test/data/a57cbf19-17289244/h5bench.cfg /public/home/dfzx202411/h5bench_test/data/test.h5
2025-11-10 23:30:33,140 h5bench - INFO - SUCCESS (all output files are located at /public/home/dfzx202411/h5bench_test/data/a57cbf19-17289244)
2025-11-10 23:30:33,140 h5bench - INFO - Runtime: 7.3098042 seconds (elapsed time, includes allocation wait time)
2025-11-10 23:30:33,140 h5bench - INFO - h5bench [write] - Complete
2025-11-10 23:30:33,141 h5bench - INFO - Finishing h5bench Suite