OpenStack Neutron源码分析：ovs-neutron-agent启动源码解析

声明：

本博客欢迎转载，但请保留原作者信息!

作者：华为云计算工程师 林凯

团队：华为杭州研发中心OpenStack社区团队

本文是在个人学习过程中整理和总结，由于时间和个人能力有限，错误之处在所难免，欢迎指正！

OpenStack Neutron，是专注于为OpenStack提供网络服务的项目。对Neutron各个组件的介绍请看这一篇博客：http://www.openstack.cn/p1745.html。

引用其中对L2 Agent的组件的介绍：L2Agent通常运行在Hypervisor，与neutron-server通过RPC通信，监听并通知设备的变化，创建新的设备来确保网络segment的正确性，应用security groups规则等。例如，OVS Agent，使用Open vSwitch来实现VLAN， GRE，VxLAN来实现网络的隔离，还包括了网络流量的转发控制。

本篇博客将对Neutron中的OVS Agent组件启动源码进行解析。

OVS Agent组件启动大致流程如下图所示：

接下来，让我们真正开始OVS Agent组件启动源码的解析

(1)    /neutron/plugins/openvswitch/agent/ovs-neutron-agent.py中的main()

<span style="font-size:14px;">def main():
    cfg.CONF.register_opts(ip_lib.OPTS)
    common_config.init(sys.argv[1:])
    common_config.setup_logging(cfg.CONF)
    q_utils.log_opt_values(LOG)

    try:
        agent_config = create_agent_config_map(cfg.CONF)
    except ValueError as e:
        LOG.error(_('%s Agent terminated!'), e)
        sys.exit(1)

    is_xen_compute_host = 'rootwrap-xen-dom0' in agent_config['root_helper']
    if is_xen_compute_host:
        # Force ip_lib to always use the root helper to ensure that ip
        # commands target xen dom0 rather than domU.
        cfg.CONF.set_default('ip_lib_force_root', True)

    <span style="color:#ff0000;">agent = OVSNeutronAgent(**agent_config)     (1)</span>
    signal.signal(signal.SIGTERM, agent._handle_sigterm)

    # Start everything.
    LOG.info(_("Agent initialized successfully, now running... "))
    <span style="color:#ff0000;">agent.daemon_loop()                         (2)</span>
</span>

上述代码中，最重要的函数是(1)函数和(2)函数，(1)函数主要的工作是实例化一个OVSAgent，并完成OVS Agent的一系列初始化工作，(2)函数一直在循环检查一些状态，发现状态发生变化，执行相应的操作。

<span style="font-size:14px;">def __init__(self, integ_br, tun_br, local_ip,
                 bridge_mappings, root_helper,
                 polling_interval, tunnel_types=None,
                 veth_mtu=None, l2_population=False,
                 enable_distributed_routing=False,
                 minimize_polling=False,
                 ovsdb_monitor_respawn_interval=(
                     constants.DEFAULT_OVSDBMON_RESPAWN),
                 arp_responder=False,
                 use_veth_interconnection=False):
        super(OVSNeutronAgent, self).__init__()
        self.use_veth_interconnection = use_veth_interconnection
        self.veth_mtu = veth_mtu
        self.root_helper = root_helper
        self.available_local_vlans = set(moves.xrange(q_const.MIN_VLAN_TAG,
                                                      q_const.MAX_VLAN_TAG))
        self.tunnel_types = tunnel_types or []
        self.l2_pop = l2_population
        # TODO(ethuleau): Change ARP responder so it's not dependent on the
        # 			    ML2 l2 population mechanism driver.
        # enable_distributed_routing是否使能分布式路由
        self.enable_distributed_routing = enable_distributed_routing
        self.arp_responder_enabled = arp_responder and self.l2_pop
        self.agent_state = {
            'binary': 'neutron-openvswitch-agent',
            'host': cfg.CONF.host,
            'topic': q_const.L2_AGENT_TOPIC,
            'configurations': {'bridge_mappings': bridge_mappings,
                               'tunnel_types': self.tunnel_types,
                               'tunneling_ip': local_ip,
                               'l2_population': self.l2_pop,
                               'arp_responder_enabled':
                               self.arp_responder_enabled,
                               'enable_distributed_routing':
                               self.enable_distributed_routing},
            'agent_type': q_const.AGENT_TYPE_OVS,
            'start_flag': True}

        # Keep track of int_br's device count for use by _report_state()
        self.int_br_device_count = 0

        self.int_br = ovs_lib.OVSBridge(integ_br, self.root_helper)
        # setup_integration_br：安装整合网桥——int_br
        # 创建patch ports，并移除所有现有的流规则
        # 添加基本的流规则
        <span style="color:#ff0000;">self.setup_integration_br()											(1)</span>
        # Stores port update notifications for processing in main rpc loop
        self.updated_ports = set()
        # setup_rpc完成以下任务：
        # 设置plugin_rpc，这是用来与neutron-server通信的
        # 设置state_rpc，用于agent状态信息上报
        # 设置connection，用于接收neutron-server的消息
        # 启动状态周期上报
        <span style="color:#ff0000;">self.setup_rpc()												(2)</span>
        self.bridge_mappings = bridge_mappings
        # 创建物理网络网桥，并用veth与br-int连接起来
        <span style="color:#ff0000;">self.setup_physical_bridges(self.bridge_mappings)							<span style="white-space:pre">	</span>(3)</span>
        self.local_vlan_map = {}
        self.tun_br_ofports = {p_const.TYPE_GRE: {},
                               p_const.TYPE_VXLAN: {}}

        self.polling_interval = polling_interval
        self.minimize_polling = minimize_polling
        self.ovsdb_monitor_respawn_interval = ovsdb_monitor_respawn_interval

        if tunnel_types:
            self.enable_tunneling = True
        else:
            self.enable_tunneling = False
        self.local_ip = local_ip
        self.tunnel_count = 0
        self.vxlan_udp_port = cfg.CONF.AGENT.vxlan_udp_port
        self.dont_fragment = cfg.CONF.AGENT.dont_fragment
        self.tun_br = None
        self.patch_int_ofport = constants.OFPORT_INVALID
        self.patch_tun_ofport = constants.OFPORT_INVALID
        if self.enable_tunneling:
            # The patch_int_ofport and patch_tun_ofport are updated
            # here inside the call to setup_tunnel_br
            self.setup_tunnel_br(tun_br)

        <span style="color:#ff0000;">self.dvr_agent = ovs_dvr_neutron_agent.OVSDVRNeutronAgent(
            self.context,
            self.plugin_rpc,
            self.int_br,
            self.tun_br,
            self.patch_int_ofport,
            self.patch_tun_ofport,
            cfg.CONF.host,
            self.enable_tunneling,
            self.enable_distributed_routing)										(4)</span>

        self.dvr_agent.setup_dvr_flows_on_integ_tun_br()

        # Collect additional bridges to monitor
        self.ancillary_brs = self.setup_ancillary_bridges(integ_br, tun_br)

        # Security group agent support
        <span style="color:#ff0000;">self.sg_agent = OVSSecurityGroupAgent(self.context,
                                              self.plugin_rpc,
                                              root_helper)					<span style="white-space:pre">			</span>(5)</span>
        # Initialize iteration counter
        self.iter_num = 0
        <span style="color:#ff0000;">self.run_daemon_loop = True										<span style="white-space:pre">	</span>(6)</span>
</span>

<span style="font-size:14px;">def setup_integration_br(self):
        """
                        安装integration网桥
                        创建patch ports，并移除所有现有的流规则
                        添加基本的流规则
        """
        # Ensure the integration bridge is created.
        # ovs_lib.OVSBridge.create() will run
        #   ovs-vsctl -- --may-exist add-br BRIDGE_NAME
        # which does nothing if bridge already exists.
        
        # 通过执行ovs-vsctl中add-br创建int_br
        self.int_br.create()
        self.int_br.set_secure_mode()

        # del-port删除patch 
        self.int_br.delete_port(cfg.CONF.OVS.int_peer_patch_port)
        # 通过ovs-ofctl移除所有流规则
        self.int_br.remove_all_flows()
        # switch all traffic using L2 learning
        # 增加actions为normal，优先级为1的流规则
        # 用L2学习来交换所有通信内容
        self.int_br.add_flow(priority=1, actions="normal")
        # Add a canary flow to int_br to track OVS restarts
        # 添加canary流规则给int_br来跟踪OVS的重启 优先级0级，actions drop
        self.int_br.add_flow(table=constants.CANARY_TABLE, priority=0,
                             actions="drop")
</span>

函数的内容很明显，就是完成安装integration网桥br-int，具体操作内容可以参考代码中的注释。br-int建立完成之后，将原有的流规则删除，并会添加两条基础的流规则，我们来看下这两条流规则的作用是什么？第一条流规则是优先级为1、actions为normal的流规则，这个规则是用来将连接到br-int的网络设备的通信内容进行转发给所有其他网络设备；第二条流规则是优先级为0、actions为drop的流规则，用来跟踪OVS的重启，这个功能在后面循环中会分析到。

之后，我们来看第二个函数self.setup_rpc()的具体内容。

<span style="font-size:14px;">def setup_rpc(self):
        self.agent_id = 'ovs-agent-%s' % cfg.CONF.host
        self.topic = topics.AGENT
        # 设置plugin_rpc，用来与neutron-server通信的
        self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
        # 设置state_rpc，用于agent状态信息上报
        self.state_rpc = agent_rpc.PluginReportStateAPI(topics.PLUGIN)

        # 设置connection，并添加consumers，用于接收neutron-server的消息
        # RPC network init
        self.context = context.get_admin_context_without_session()
        # Handle updates from service
        self.endpoints = [self]
        # Define the listening consumers for the agent
        consumers = [[topics.PORT, topics.UPDATE],
                     [topics.NETWORK, topics.DELETE],
                     [constants.TUNNEL, topics.UPDATE],
                     [topics.SECURITY_GROUP, topics.UPDATE],
                     [topics.DVR, topics.UPDATE]]
        if self.l2_pop:
            consumers.append([topics.L2POPULATION,
                              topics.UPDATE, cfg.CONF.host])
        self.connection = agent_rpc.create_consumers(self.endpoints,
                                                     self.topic,
                                                     consumers)
        
        # 启动心跳周期上报
        report_interval = cfg.CONF.AGENT.report_interval
        if report_interval:
            heartbeat = loopingcall.FixedIntervalLoopingCall(
                self._report_state)
            heartbeat.start(interval=report_interval)
</span>

接下来解析(3)函数self.setup_physical_bridges(self.bridge_mappings)，具体内容如下：

<span style="font-size:14px;">def setup_physical_bridges(self, bridge_mappings):
        '''Setup the physical network bridges.

        Creates physical network bridges and links them to the
        integration bridge using veths.

        :param bridge_mappings: map physical network names to bridge names.
        '''
        """
                        安装物理网络网桥
                        创建物理网络网桥，并用veth/patchs与br-int连接起来
        """
        self.phys_brs = {}
        self.int_ofports = {}
        self.phys_ofports = {}
        ip_wrapper = ip_lib.IPWrapper(self.root_helper)
        ovs_bridges = ovs_lib.get_bridges(self.root_helper)
        for physical_network, bridge in bridge_mappings.iteritems():
            LOG.info(_("Mapping physical network %(physical_network)s to "
                       "bridge %(bridge)s"),
                     {'physical_network': physical_network,
                      'bridge': bridge})
            # setup physical bridge
            if bridge not in ovs_bridges:
                LOG.error(_("Bridge %(bridge)s for physical network "
                            "%(physical_network)s does not exist. Agent "
                            "terminated!"),
                          {'physical_network': physical_network,
                           'bridge': bridge})
                sys.exit(1)
            br = ovs_lib.OVSBridge(bridge, self.root_helper)
            br.remove_all_flows()
            br.add_flow(priority=1, actions="normal")
            self.phys_brs[physical_network] = br

            # 使用veth/patchs使br-eth1与br-int互联
            # 删除原有的patchs，创建int-br-eth1和phy-br-eth1
            # 使用ovs-vsctl show
            # interconnect physical and integration bridges using veth/patchs
            int_if_name = self.get_peer_name(constants.PEER_INTEGRATION_PREFIX,
                                             bridge)
            phys_if_name = self.get_peer_name(constants.PEER_PHYSICAL_PREFIX,
                                              bridge)
            self.int_br.delete_port(int_if_name)
            br.delete_port(phys_if_name)
            if self.use_veth_interconnection:
                if ip_lib.device_exists(int_if_name, self.root_helper):
                    ip_lib.IPDevice(int_if_name,
                                    self.root_helper).link.delete()
                    # Give udev a chance to process its rules here, to avoid
                    # race conditions between commands launched by udev rules
                    # and the subsequent call to ip_wrapper.add_veth
                    utils.execute(['/sbin/udevadm', 'settle', '--timeout=10'])
                # 通过ip netns exec 'namespace' ip link add veth命令添加veth
                int_veth, phys_veth = ip_wrapper.add_veth(int_if_name,
                                                          phys_if_name)
                int_ofport = self.int_br.add_port(int_veth)
                phys_ofport = br.add_port(phys_veth)
            else:
                # Create patch ports without associating them in order to block
                # untranslated traffic before association
                int_ofport = self.int_br.add_patch_port(
                    int_if_name, constants.NONEXISTENT_PEER)
                phys_ofport = br.add_patch_port(
                    phys_if_name, constants.NONEXISTENT_PEER)

            self.int_ofports[physical_network] = int_ofport
            self.phys_ofports[physical_network] = phys_ofport

            # 封锁桥梁之间的所有通信翻译
            # block all untranslated traffic between bridges
            self.int_br.add_flow(priority=2, in_port=int_ofport,
                                 actions="drop")
            br.add_flow(priority=2, in_port=phys_ofport, actions="drop")

            if self.use_veth_interconnection:
                # 使能veth传递通信
                # enable veth to pass traffic
                int_veth.link.set_up()
                phys_veth.link.set_up()
                if self.veth_mtu:
                    # set up mtu size for veth interfaces
                    int_veth.link.set_mtu(self.veth_mtu)
                    phys_veth.link.set_mtu(self.veth_mtu)
            else:
                # 关联patch ports传递通信
                # associate patch ports to pass traffic
                self.int_br.set_db_attribute('Interface', int_if_name,
                                             'options:peer', phys_if_name)
                br.set_db_attribute('Interface', phys_if_name,
                                    'options:peer', int_if_name)
</span>

在setup_physical_bridges这个函数中，完成了物理网桥br-eth*的创建，创建好网桥之后，与安装br-int一样，首先删除了现有的所有流规则，并添加了同样为normal的流规则，用以转发消息，接下来是与br-int不同的地方，根据use_veth_interconnection决定是否使用veth与br-int进行连接，并配置veth或者patch port，然后通过设置drop流规则，封锁桥之间的通信，然后使能veth或者patch ports进行通信。

(4)函数与(5)函数分别是对DVR Agent（分布式路由代理）和Security Group Agent（安全组代理）的初始化工作，用于处理DVR和security group，这部分的内容将在之后的博客介绍。

<span style="font-size:14px;">def rpc_loop(self, polling_manager=None):
        if not polling_manager:
            polling_manager = polling.AlwaysPoll()
		# 初始化设置
        sync = True
        ports = set()
        updated_ports_copy = set()
        ancillary_ports = set()
        tunnel_sync = True
        ovs_restarted = False
		# 进入循环
        while self.run_daemon_loop:
            start = time.time()
            port_stats = {'regular': {'added': 0,
                                      'updated': 0,
                                      'removed': 0},
                          'ancillary': {'added': 0,
                                        'removed': 0}}
            LOG.debug(_("Agent rpc_loop - iteration:%d started"),
                      self.iter_num)
            if sync:
                LOG.info(_("Agent out of sync with plugin!"))
                ports.clear()
                ancillary_ports.clear()
                sync = False
                polling_manager.force_polling()
			# 根据之前在br-int中设置canary flow的有无判断是否进行restart操作
            ovs_restarted = self.check_ovs_restart()							
            if ovs_restarted:
                ......
            # Notify the plugin of tunnel IP
            if self.enable_tunneling and tunnel_sync:
                ......
            if self._agent_has_updates(polling_manager) or ovs_restarted:
                try:
                    LOG.debug(_("Agent rpc_loop - iteration:%(iter_num)d - "
                                "starting polling. Elapsed:%(elapsed).3f"),
                              {'iter_num': self.iter_num,
                               'elapsed': time.time() - start})
                    updated_ports_copy = self.updated_ports
                    self.updated_ports = set()
                    reg_ports = (set() if ovs_restarted else ports)
                    <span style="color:#ff0000;">port_info = self.scan_ports(reg_ports, updated_ports_copy)		(1)</span>
                    LOG.debug(_("Agent rpc_loop - iteration:%(iter_num)d - "
                                "port information retrieved. "
                                "Elapsed:%(elapsed).3f"),
                              {'iter_num': self.iter_num,
                               'elapsed': time.time() - start})
                    # Secure and wire/unwire VIFs and update their status
                    # on Neutron server
                    if (self._port_info_has_changes(port_info) or
                        self.sg_agent.firewall_refresh_needed() or
                        ovs_restarted):
                        LOG.debug(_("Starting to process devices in:%s"),
                                  port_info)
                        # If treat devices fails - must resync with plugin
                        <span style="color:#ff0000;">sync = self.process_network_ports(port_info,
                                                          ovs_restarted)		(2)</span>
                        LOG.debug(_("Agent rpc_loop - iteration:%(iter_num)d -"
                                    "ports processed. Elapsed:%(elapsed).3f"),
                                  {'iter_num': self.iter_num,
                                   'elapsed': time.time() - start})
                        port_stats['regular']['added'] = (
                            len(port_info.get('added', [])))
                        port_stats['regular']['updated'] = (
                            len(port_info.get('updated', [])))
                        port_stats['regular']['removed'] = (
                            len(port_info.get('removed', [])))
                    ports = port_info['current']
                    # Treat ancillary devices if they exist
                    if self.ancillary_brs:
                        port_info = self.update_ancillary_ports(
                            ancillary_ports)
                        LOG.debug(_("Agent rpc_loop - iteration:%(iter_num)d -"
                                    "ancillary port info retrieved. "
                                    "Elapsed:%(elapsed).3f"),
                                  {'iter_num': self.iter_num,
                                   'elapsed': time.time() - start})

                        if port_info:
                            rc = self.process_ancillary_network_ports(
                                port_info)
                            LOG.debug(_("Agent rpc_loop - iteration:"
                                        "%(iter_num)d - ancillary ports "
                                        "processed. Elapsed:%(elapsed).3f"),
                                      {'iter_num': self.iter_num,
                                       'elapsed': time.time() - start})
                            ancillary_ports = port_info['current']
                            port_stats['ancillary']['added'] = (
                                len(port_info.get('added', [])))
                            port_stats['ancillary']['removed'] = (
                                len(port_info.get('removed', [])))
                            sync = sync | rc

                    polling_manager.polling_completed()
                except Exception:
                    LOG.exception(_("Error while processing VIF ports"))
                    # Put the ports back in self.updated_port
                    self.updated_ports |= updated_ports_copy
                    sync = True

            # sleep till end of polling interval
            elapsed = (time.time() - start)
            LOG.debug(_("Agent rpc_loop - iteration:%(iter_num)d "
                        "completed. Processed ports statistics: "
                        "%(port_stats)s. Elapsed:%(elapsed).3f"),
                      {'iter_num': self.iter_num,
                       'port_stats': port_stats,
                       'elapsed': elapsed})
            if (elapsed < self.polling_interval):
                time.sleep(self.polling_interval - elapsed)
            else:
                LOG.debug(_("Loop iteration exceeded interval "
                            "(%(polling_interval)s vs. %(elapsed)s)!"),
                          {'polling_interval': self.polling_interval,
                           'elapsed': elapsed})
            self.iter_num = self.iter_num + 1
</span>

<span style="font-size:14px;">def scan_ports(self, registered_ports, updated_ports=None):
        # 通过ovs-vsctl命令获取数据库中port设置信息
        cur_ports = self.int_br.get_vif_port_set()
        self.int_br_device_count = len(cur_ports)
        port_info = {'current': cur_ports}
        if updated_ports is None:
            updated_ports = set()
        # 获取已经注册的port的更新信息
        updated_ports.update(self.check_changed_vlans(registered_ports))
        if updated_ports:
            # Some updated ports might have been removed in the
            # meanwhile, and therefore should not be processed.
            # In this case the updated port won't be found among
            # current ports.
            updated_ports &= cur_ports
            # 更新updated_ports的数量
            if updated_ports:
                port_info['updated'] = updated_ports

        # FIXME(salv-orlando): It's not really necessary to return early
        # if nothing has changed.
        if cur_ports == registered_ports:
            # No added or removed ports to set, just return here
            return port_info

        # 更新added_ports的数量
        port_info['added'] = cur_ports - registered_ports
        # Remove all the known ports not found on the integration bridge
        # 更新removed_ports的数量，移除所有没有在br-int上发现的已知ports
        port_info['removed'] = registered_ports - cur_ports
        return port_info
</span>

<span style="font-size:14px;">def process_network_ports(self, port_info, ovs_restarted):
        resync_a = False
        resync_b = False
        # 取出更新和添加的prot信息
        devices_added_updated = (port_info.get('added', set()) |
                                 port_info.get('updated', set()))
        if devices_added_updated:
            start = time.time()
            try:
                # treat_devices_added_or_updated根据是否已经存在这个port分别进行添加和更新的操作
                # 添加：skipped_devices.append(device)进行添加之后，将做与update一样的操作
                # 更新：通过treat_vif_port将port添加并且绑定到net_uuid/lsw_id并且 为没有绑定的通信设置流规则
                skipped_devices = self.treat_devices_added_or_updated(
                    devices_added_updated, ovs_restarted)
                LOG.debug(_("process_network_ports - iteration:%(iter_num)d -"
                            "treat_devices_added_or_updated completed. "
                            "Skipped %(num_skipped)d devices of "
                            "%(num_current)d devices currently available. "
                            "Time elapsed: %(elapsed).3f"),
                          {'iter_num': self.iter_num,
                           'num_skipped': len(skipped_devices),
                           'num_current': len(port_info['current']),
                           'elapsed': time.time() - start})
                # Update the list of current ports storing only those which
                # have been actually processed.
                port_info['current'] = (port_info['current'] -
                                        set(skipped_devices))
            except DeviceListRetrievalError:
                # Need to resync as there was an error with server
                # communication.
                LOG.exception(_("process_network_ports - iteration:%d - "
                                "failure while retrieving port details "
                                "from server"), self.iter_num)
                resync_a = True
        if 'removed' in port_info:
            start = time.time()
            # 完成移除port的功能，通过发送RPC命令给Neutron server完成
            resync_b = self.treat_devices_removed(port_info['removed'])
            LOG.debug(_("process_network_ports - iteration:%(iter_num)d -"
                        "treat_devices_removed completed in %(elapsed).3f"),
                      {'iter_num': self.iter_num,
                       'elapsed': time.time() - start})
        # If one of the above operations fails => resync with plugin
        return (resync_a | resync_b)
</span>

从代码的解释可以看到，process_network_ports完成了port的添加，删除和更新的操作。之后循环检测是否已经到了循环间隔，如果还没有到间隔时间就sleep到那个时间，然后继续循环工作。

         至此，我们也就完成OVS Agent的启动源码解析。