oozie hpdl参数分析

最新推荐文章于 2021-09-04 01:24:16 发布

原创最新推荐文章于 2021-09-04 01:24:16 发布 · 1.7k 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

hadoop---oozie 专栏收录该内容

8 篇文章

订阅专栏

本文详细介绍了 Oozie 中参数设置的不同方式，包括客户端调用时的设置方法、全局配置的使用及其实现原理，并探讨了如何通过 Oozie 自带的方法获取这些配置。

一 oozie设置参数方式

1）客户端调用时，例如：

            Properties conf = wc.createConfiguration();
	    conf.setProperty("user.name", "oozie");
	    conf.setProperty("dataflow.taskMangerURL", "http://172.16.11.253:8081/bdi-api/TaskManagerServlet");
	    jobId = wc.run(conf);
2）Global Configurations
3）action的子标签，例：
	<action name='a2'>
		<dataflow xmlns='uri:oozie:dataflow-action:0.1'>
    			<dataflowId>a2</dataflowId>
    		</dataflow>
		<ok to='end'/>
		<error to='fail'/>
	</action>

二全局配置

19 Global Configurations

Oozie allows a global section to reduce the redundant job-tracker and name-node declarations for each action. The user can define aglobal section in the beginning of the workflow.xml . The global section may contain the job-xml, configuration, job-tracker, or name-node that the user would like to set for every action. If a user then redefines one of these in a specific action node, Oozie will update use the specific declaration instead of the global one for that action.

Example of a global element:

<workflow-app xmlns="uri:oozie:workflow:0.4" name="wf-name">
<global>
   <job-tracker>${job-tracker}</job-tracker>
   <name-node>${namd-node}</name-node>
   <job-xml>job1.xml</job-xml>
   <configuration>
        <property>
            <name>mapred.job.queue.name</name>
            <value>${queueName}</value>
        </property>
    </configuration>
</global>

xsd片段：

    <xs:complexType name="GLOBAL">
    	<xs:sequence>
            <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
            <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
            <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element name="configuration" type="workflow:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

注：在实际运行当中<global>的</configuration>子标签将内容将写入活动的配置，例如：

hpdl:

<workflow-app xmlns='uri:oozie:workflow:0.4' name='workflowId1'>
	<global>
		<configuration>
			<property>
				<name>dataflow.taskMangerURL</name>
				<value>http://172.16.11.253:8081/bdi-api/TaskManagerServlet</value>
			</property>
		</configuration>
	</global>

	<start to='a2'/>
	<action name='a2'>
		<dataflow xmlns='uri:oozie:dataflow-action:0.1'>
    		<dataflowId>a2</dataflowId>
    	</dataflow>
		<ok to='end'/>
		<error to='fail'/>
	</action>

通过在action方法中action.getConf()取得的action配置实际值为：

<dataflow xmlns="uri:oozie:dataflow-action:0.1">
  <dataflowId>a2</dataflowId>
  <configuration>
    <property>
      <name>dataflow.taskMangerURL</name>
      <value>http://172.16.11.253:8081/bdi-api/TaskManagerServlet</value>
    </property>
  </configuration>
</dataflow>

三 疑问

除了直接解析action的xml,如何在oozie中通过本身自带的方法取属性值，如下列Property中的值：

            Properties conf = wc.createConfiguration();
	    conf.setProperty("user.name", "oozie");
	    conf.setProperty("dataflow.taskMangerURL", "http://172.16.11.253:8081/bdi-api/TaskManagerServlet");


	    String jobId="";
	    
		try {
			jobId = wc.run(conf);