Walks

网络安全爱好者

Python3学习系列(八):Scrapy用Rules和LinkExtractor爬取多页面

《Python3学习系列(八):Scrapy用Rules和LinkExtractor爬取多页面》

前言:

学习知识是要一步一步来的,下面为大家带来点新知识,爬虫的Rules规则,就是对爬取的网页中的链接进行分析,提取出我们需要的继续爬下去的链接,继续爬取下去。 

目录:

0x01:简析

0x02:Rule与LinkExtractor具体参数

0x03:代码实现与分析

 

0x01 简析

当我们想要提取start_urls中的Url中的链接时,可以加一个Rules,用来提取,该Rules为继承CrawlSpider才有的规则,相比于继承Spider

下面我们先初识一下这个规则

rules = [       
        Rule(LinkExtractor(allow=(r'http://exmaple.com/ershoufang/\d+',)), callback='parse_item'),       
    ]

        

        0x02 Rule与LinkExtractor具体参数

Rule对象

Role对象有下面参数

  • link_extractor:链接提取规则

  • callback:回调函数(如何处理)

  • cb_kwargs:附加参数,可以在回调函数中获取到

  • follow:是否对提取的网页进行进一步提取(默认:False)

  • process_links:处理所有的链接的回调,用于处理从response提取的links,通常用于过滤(参数为link列表)

  • process_request:链接请求预处理(添加header或cookie等)

 LinkExtractor

LinkExtractor常用的参数有:

  • allow:提取满足正则表达式的链接

  • deny:排除正则表达式匹配的链接(优先级高于allow

  • allow_domains:允许的域名(可以是strlist

  • deny_domains:排除的域名(可以是strlist

  • restrict_xpaths:提取满足XPath选择条件的链接(可以是strlist

  • restrict_css:提取满足css选择条件的链接(可以是strlist

  • tags:提取指定标签下的链接,默认从aarea中提取(可以是strlist

  • attrs:提取满足拥有属性的链接,默认为href(类型为list

  • unique:链接是否去重(类型为boolean

  • process_value:值处理函数(优先级大于allow

        

        0x03 代码实现和分析

# -*- coding: utf-8 -*-
#导入接下来要使用的类
from scrapy.spiders import CrawlSpider,Rule
from scrapy.linkextractors import LinkExtractor
#自定义的存放数据的类(容器)
from DoubanSpider.items import DoubanspiderItem


class DoubanSpider(CrawlSpider):
    name = 'douban'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/top250']
    #定义一个爬取规则,从start_urls中的网页提取LinkExtractor上规定的所有链接,callback为对这些链接如何处理
    rules =(
        Rule(LinkExtractor(allow=(r'https://movie.douban.com/subject/\d+')), callback='parse_item'),
        )

    def parse_item(self, response):
        item = DoubanspiderItem()
        item['title'] = "".join(response.xpath('//span[@property="v:itemreviewed"]/text()').extract())

        yield item

转载请注明来自:www.bywalks.com

点赞
  1. ferragamo belts说道:

    I wanted to write a quick message to thank you for the lovely ways you are writing at this website. My considerable internet research has finally been rewarded with pleasant tips to go over with my co-workers. I would mention that most of us visitors actually are unequivocally blessed to be in a remarkable network with so many brilliant professionals with helpful concepts. I feel somewhat privileged to have discovered your entire web site and look forward to many more excellent times reading here. Thanks once more for a lot of things.

  2. My husband and i ended up being absolutely thrilled Ervin could do his basic research through the entire precious recommendations he discovered through your web site. It's not at all simplistic to just continually be handing out tips which often some people have been selling. Therefore we already know we have the writer to thank for that. All of the illustrations you have made, the simple site menu, the relationships you help create - it's got mostly exceptional, and it is letting our son and us consider that this subject matter is excellent, which is especially important. Many thanks for the whole thing!

  3. air jordan 13说道:

    Needed to post you the little note to finally say thank you yet again for your personal pleasing views you've discussed on this page. It was really shockingly open-handed with people like you to provide easily all that most of us would've distributed as an e book in making some money for their own end, mostly now that you could possibly have done it in case you decided. The concepts additionally acted as the great way to understand that most people have the same fervor like my very own to see a little more around this matter. I am sure there are thousands of more pleasant situations in the future for individuals that scan your blog post.

  4. fitflops说道:

    I precisely wanted to say thanks again. I am not sure what I would have done without those tricks revealed by you on my theme. It was actually the challenging matter in my opinion, but finding out a new well-written form you handled the issue made me to cry for gladness. I'm just happy for your information and then sincerely hope you comprehend what an amazing job you are carrying out educating others by way of your webpage. Most likely you've never come across any of us.

  5. golden goose说道:

    Thank you a lot for providing individuals with a very nice chance to read in detail from this blog. It can be very lovely and as well , jam-packed with fun for me and my office co-workers to visit your website minimum three times a week to read the fresh secrets you have. And of course, I'm also always contented with the magnificent opinions you give. Some two tips in this posting are in reality the finest I have had.

  6. I simply desired to thank you very much again. I am not sure the things that I could possibly have taken care of in the absence of the type of basics documented by you regarding that question. Previously it was a real frustrating case in my view, however , finding out the very well-written way you processed it forced me to weep for fulfillment. I'm just grateful for this service and as well , hope that you recognize what a powerful job you were carrying out instructing the mediocre ones through your site. I'm certain you haven't met all of us.

  7. I wish to express appreciation to this writer just for bailing me out of this particular setting. As a result of checking throughout the world wide web and seeing tricks which were not productive, I thought my entire life was done. Being alive without the presence of strategies to the problems you've solved all through your short article is a crucial case, as well as ones that might have badly damaged my entire career if I had not noticed your web blog. Your actual understanding and kindness in dealing with every item was precious. I'm not sure what I would have done if I had not discovered such a step like this. It's possible to now relish my future. Thank you very much for this specialized and sensible help. I won't think twice to suggest the blog to anyone who should receive guide on this problem.

发表评论

电子邮件地址不会被公开。 必填项已用*标注