Kylin Cube Build(二)：构造任务链

Kylin Cube Build的任务链构造

• 以MR_V2为例
• 构建过程的时序单位为step，在代码中有些step也称做task

初始化操作

构建cube的程序入口: new BatchCubingJobBuilder2(newSegment, submitter).build()

1. 日志输出: logger.info("MR_V2 new job to BUILD segment " + seg);
2. 实例化结果对象: final CubingJob result = CubingJob.createBuildJob()
3. 设置jobId和cuboidRootpath
4. build函数中的addTask负责把每个step添加到实例化成功的结果对象result的待办List类型的subTasks中，由Java多线程机制负责执行

构建流程

Phase 1: Create Flat Table & Materialize Hive View in Lookup Tables

inputSide.addStepPhase1_CreateFlatTable(result);

final String hiveInitStatements = JoinedFlatTable.generateHiveInitStatements(flatTableDatabase);

// 拼接HiveQL
addStepPhase1_DoCreateFlatTable(jobFlow);

// 物化Lookup Table
addStepPhase1_DoMaterializeLookupTable(jobFlow);

其中, addStepPhase1_DoCreateFlatTable方法调用createFlatHiveTableStep生成主要的HiveQL

private AbstractExecutable createFlatHiveTableStep(String hiveInitStatements, String jobWorkingDir,
                String cubeName) {
    //from hive to hive
    final String dropTableHql = JoinedFlatTable.generateDropTableStatement(flatDesc);
    final String createTableHql = JoinedFlatTable.generateCreateTableStatement(flatDesc, jobWorkingDir);
    String insertDataHqls = JoinedFlatTable.generateInsertDataStatement(flatDesc);

    CreateFlatHiveTableStep step = new CreateFlatHiveTableStep();
    step.setInitStatement(hiveInitStatements);
    step.setCreateTableStatement(dropTableHql + createTableHql + insertDataHqls);
    CubingExecutableUtil.setCubeName(cubeName, step.getParams());
    step.setName(ExecutableConstants.STEP_NAME_CREATE_FLAT_HIVE_TABLE);
    return step;
}

在WebUI上可以看到这一步执行的HiveQL

Phase 2: Build Dictionary

1. 抽取列值: Extract Fact Table Distinct Columns

   result.addTask(createFactDistinctColumnsStepWithStats(jobId));

2. 是否按照某些列来分散数据: Build UHC Dictionary

   if (isEnableUHCDictStep()) {
       result.addTask(createBuildUHCDictStep(jobId));
   }

3. 构建字典: Build Dimension Dictionary

   result.addTask(createBuildDictionaryStep(jobId));

4. 保存统计信息: Save Cuboid Statistics

   result.addTask(createSaveStatisticsStep(jobId));

5. Create HTable: 对HBase进行操作

   outputSide.addStepPhase2_BuildDictionary(result);
   
   public void addStepPhase2_BuildDictionary(DefaultChainedExecutable jobFlow) {
       jobFlow.addTask(steps.createCreateHTableStep(jobFlow.getId()));
   }

Phase 3: Build Cube

MR_V2有两种Cube构建算法, 为分层构建和内存构建, 两种算法的子任务都会被加入任务链, 根据Phase2的统计信息自动选择其中一种且仅有一种算法进行执行

addLayerCubingSteps(result, jobId, cuboidRootPath); // layer cubing, only selected algorithm will execute
addInMemCubingSteps(result, jobId, cuboidRootPath); // inmem cubing, only selected algorithm will execute
outputSide.addStepPhase3_BuildCube(result);

1. Build Base Cuboid Data & Build N-Dimension Cuboid

   层级算法需要构建多层Cuboid

   // base cuboid step
   result.addTask(createBaseCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, 0), jobId));
   // n dim cuboid steps
   for (int i = 1; i <= maxLevel; i++) {
       result.addTask(createNDimensionCuboidStep(getCuboidOutputPathsByLevel(cuboidRootPath, i - 1), getCuboidOutputPathsByLevel(cuboidRootPath, i), i, jobId));
   }

2. 把结果转换成HFile: Convert Cuboid Data to HFile

   jobFlow.addTask(steps.createConvertCuboidToHfileStep(jobFlow.getId()));

3. Load HFile to HBase Table: 对HBase进行操作

   jobFlow.addTask(steps.createBulkLoadStep(jobFlow.getId()));

Phase 4: Update Metadata & Cleanup

• Cube构建完毕后, 先更新元数据
• 调用数据源接口清理临时数据
• 调用存储引擎接口清理临时数据

result.addTask(createUpdateCubeInfoAfterBuildStep(jobId, lookupMaterializeContext));
inputSide.addStepPhase4_Cleanup(result);
outputSide.addStepPhase4_Cleanup(result);

1. Update Cube Info

2. Hive Cleanup

   拼接HiveQL, 主要通过方法cleanUpIntermediateFlatTable:

   private String cleanUpIntermediateFlatTable(KylinConfig config) throws IOException {
       StringBuffer output = new StringBuffer();
       final HiveCmdBuilder hiveCmdBuilder = new HiveCmdBuilder();
       final List<String> hiveTables = this.getIntermediateTables();
       for (String hiveTable : hiveTables) {
           if (config.isHiveKeepFlatTable() == false && StringUtils.isNotEmpty(hiveTable)) {
               hiveCmdBuilder.addStatement("USE " + config.getHiveDatabaseForIntermediateTable() + ";");
               hiveCmdBuilder.addStatement("DROP TABLE IF EXISTS  " + hiveTable + ";");
   
               output.append("Hive table " + hiveTable + " is dropped. \n");
           }
       }
       config.getCliCommandExecutor().execute(hiveCmdBuilder.build());
       rmdirOnHDFS(getExternalDataPaths());
       output.append("Path " + getExternalDataPaths() + " is deleted. \n");
   
       return output.toString();
   }

3. 存储引擎接口清理临时数据方面没有任何操作