apple store 爬虫

原创

于 2017-12-07 21:06:24 发布 · 6k 阅读

本文介绍了一次使用Python的scrapy框架编写爬虫，从Apple Store抓取APP应用信息的经历。作者分享了核心代码，并邀请读者查看、贡献代码，github链接：https://github.com/luotuo/spider-for-apple-store。

前段时间受朋友之托，写了一个爬去Apple store APP应用信息的爬虫。基于scrapy写的。第一次接触scrapy，有很多不太了解的地方。请大家指教。核心代码很短

#! usr/bin/python
# -*- coding: utf-8 -*-

import scrapy
from tutorial.items import TutorialItem
from urllib import unquote
import re


class AppleSpider(scrapy.Spider):
    name = 'apple'
    allowed_domains = ['itunes.apple.com']
    current_category = {}

    def start_requests(self):
        yield scrapy.Request('https://itunes.apple.com/cn/genre/ios/id36?mt=8', self.parse)

    def parse(self, response):
        my_item = TutorialItem()
        app_url = response.url
        app_name = response.xpath('//h1[@itemprop="name"]/text()').extract()
        if len(app_name) > 0:
            category = response.meta['category']
            my_item['app_name'] = app_name[0]
            app_category = response.xpath('//span[@itemprop="applicationCategory"]/text()').extract()
            if len(app_category) > 0:
                my_item['app_category

标签