Walks

网络安全爱好者

Python3学习系列(八):Scrapy用Rules和LinkExtractor爬取多页面

《Python3学习系列(八):Scrapy用Rules和LinkExtractor爬取多页面》

前言:

学习知识是要一步一步来的,下面为大家带来点新知识,爬虫的Rules规则,就是对爬取的网页中的链接进行分析,提取出我们需要的继续爬下去的链接,继续爬取下去。 

目录:

0x01:简析

0x02:Rule与LinkExtractor具体参数

0x03:代码实现与分析

 

0x01 简析

当我们想要提取start_urls中的Url中的链接时,可以加一个Rules,用来提取,该Rules为继承CrawlSpider才有的规则,相比于继承Spider

下面我们先初识一下这个规则

rules = [       
        Rule(LinkExtractor(allow=(r'http://exmaple.com/ershoufang/\d+',)), callback='parse_item'),       
    ]

        

        0x02 Rule与LinkExtractor具体参数

Rule对象

Role对象有下面参数

  • link_extractor:链接提取规则

  • callback:回调函数(如何处理)

  • cb_kwargs:附加参数,可以在回调函数中获取到

  • follow:是否对提取的网页进行进一步提取(默认:False)

  • process_links:处理所有的链接的回调,用于处理从response提取的links,通常用于过滤(参数为link列表)

  • process_request:链接请求预处理(添加header或cookie等)

 LinkExtractor

LinkExtractor常用的参数有:

  • allow:提取满足正则表达式的链接

  • deny:排除正则表达式匹配的链接(优先级高于allow

  • allow_domains:允许的域名(可以是strlist

  • deny_domains:排除的域名(可以是strlist

  • restrict_xpaths:提取满足XPath选择条件的链接(可以是strlist

  • restrict_css:提取满足css选择条件的链接(可以是strlist

  • tags:提取指定标签下的链接,默认从aarea中提取(可以是strlist

  • attrs:提取满足拥有属性的链接,默认为href(类型为list

  • unique:链接是否去重(类型为boolean

  • process_value:值处理函数(优先级大于allow

        

        0x03 代码实现和分析

# -*- coding: utf-8 -*-
#导入接下来要使用的类
from scrapy.spiders import CrawlSpider,Rule
from scrapy.linkextractors import LinkExtractor
#自定义的存放数据的类(容器)
from DoubanSpider.items import DoubanspiderItem


class DoubanSpider(CrawlSpider):
    name = 'douban'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/top250']
    #定义一个爬取规则,从start_urls中的网页提取LinkExtractor上规定的所有链接,callback为对这些链接如何处理
    rules =(
        Rule(LinkExtractor(allow=(r'https://movie.douban.com/subject/\d+')), callback='parse_item'),
        )

    def parse_item(self, response):
        item = DoubanspiderItem()
        item['title'] = "".join(response.xpath('//span[@property="v:itemreviewed"]/text()').extract())

        yield item

转载请注明来自:www.bywalks.com

点赞
  1. nike outlet说道:

    I and also my guys came following the excellent solutions on your web blog then the sudden I had a terrible suspicion I had not expressed respect to you for those secrets. The men appeared to be as a result happy to study all of them and have now actually been having fun with those things. We appreciate you genuinely so considerate and for deciding on such important guides millions of individuals are really desirous to be informed on. My personal honest apologies for not expressing gratitude to you sooner.

  2. I simply needed to say thanks once again. I do not know what I would've achieved in the absence of those basics discussed by you regarding such a area. Entirely was a frustrating dilemma in my position, however , understanding this expert approach you handled the issue forced me to cry for contentment. Now i am happy for the support and then hope that you find out what a great job that you are doing instructing many others using your websites. I am sure you haven't encountered all of us.

  3. adidas nmd说道:

    I wish to show my appreciation to you just for bailing me out of such a scenario. Just after scouting throughout the world wide web and finding methods which were not pleasant, I assumed my life was well over. Living devoid of the solutions to the problems you've fixed by way of your entire short article is a serious case, and those that could have badly affected my career if I had not encountered your website. Your own personal natural talent and kindness in controlling all the stuff was invaluable. I don't know what I would have done if I hadn't come across such a subject like this. I can also now look ahead to my future. Thanks a lot very much for the skilled and result oriented help. I won't hesitate to suggest the sites to anybody who needs to have guidance about this matter.

  4. kobe 9说道:

    I actually wanted to compose a quick word to be able to appreciate you for the wonderful tips you are writing on this site. My time-consuming internet look up has now been paid with reliable insight to share with my good friends. I 'd state that that we readers actually are very much fortunate to exist in a good network with very many brilliant individuals with very helpful tips and hints. I feel rather happy to have used your web page and look forward to really more enjoyable times reading here. Thanks a lot once more for everything.

  5. hermes belt说道:

    I'm just writing to let you know of the amazing experience my child encountered studying your webblog. She mastered a lot of details, not to mention what it's like to have a wonderful teaching nature to have a number of people really easily know just exactly a number of complex matters. You actually exceeded our expectations. I appreciate you for imparting the insightful, safe, educational and as well as cool tips about the topic to Jane.

  6. I together with my guys have already been following the nice information located on the website and instantly developed an awful feeling I had not expressed respect to the site owner for those strategies. All of the people appeared to be certainly glad to study all of them and have now actually been loving these things. Appreciate your genuinely really accommodating and also for finding some notable areas millions of individuals are really needing to be informed on. Our sincere apologies for not expressing appreciation to sooner.

  7. Thank you a lot for providing individuals with an extremely breathtaking chance to read critical reviews from this website. It really is very ideal and also jam-packed with fun for me and my office colleagues to visit your blog minimum thrice per week to study the latest tips you have got. Not to mention, I am just actually fulfilled with your awesome tricks served by you. Certain 4 facts in this posting are in truth the finest I've had.

  8. reebok shoes说道:

    I must show thanks to the writer for bailing me out of such a problem. After looking through the internet and seeing suggestions that were not beneficial, I was thinking my entire life was over. Being alive devoid of the approaches to the problems you have fixed by means of your good short article is a crucial case, as well as the ones which could have in a wrong way affected my entire career if I hadn't noticed the website. Your primary know-how and kindness in playing with the whole lot was excellent. I'm not sure what I would've done if I had not come across such a step like this. I'm able to now look forward to my future. Thank you very much for the reliable and sensible guide. I will not think twice to endorse the website to any individual who should get guide on this subject.

  9. I really wanted to make a simple remark so as to express gratitude to you for those great facts you are showing on this website. My time-consuming internet lookup has finally been honored with brilliant details to talk about with my friends and family. I 'd express that many of us site visitors are undoubtedly fortunate to exist in a good community with many lovely professionals with valuable guidelines. I feel somewhat fortunate to have come across your web page and look forward to so many more cool moments reading here. Thanks a lot once more for everything.

  10. nba jerseys说道:

    I not to mention my pals have already been going through the best strategies located on the blog and then suddenly developed a horrible feeling I never expressed respect to the blog owner for those tips. The boys were for that reason warmed to read them and now have sincerely been making the most of these things. Appreciate your indeed being considerably kind as well as for choosing this kind of notable subjects millions of individuals are really desirous to understand about. My very own honest regret for not expressing appreciation to earlier.

  11. yeezy boost 350说道:

    I must show my thanks to you for rescuing me from this particular scenario. Because of researching through the search engines and seeing ways that were not helpful, I thought my life was over. Existing without the strategies to the problems you've resolved as a result of your good review is a serious case, and the ones that would have badly damaged my entire career if I hadn't noticed your blog. Your natural talent and kindness in controlling all things was invaluable. I don't know what I would've done if I had not come upon such a solution like this. I can now look ahead to my future. Thank you very much for the reliable and results-oriented help. I will not think twice to suggest your blog to anybody who wants and needs counselling about this problem.

  12. kobe shoes说道:

    I definitely wanted to type a quick remark to say thanks to you for those awesome ideas you are writing at this website. My considerable internet research has now been paid with good knowledge to go over with my company. I 'd claim that we site visitors actually are definitely blessed to be in a really good place with many perfect people with great tactics. I feel really fortunate to have encountered your weblog and look forward to plenty of more excellent times reading here. Thanks once more for all the details.

  13. moncler jackets说道:

    Thank you so much for giving everyone remarkably nice chance to check tips from this web site. It is often very terrific plus jam-packed with a great time for me and my office co-workers to search your site at least 3 times in one week to read the fresh guidance you have. And lastly, I'm so actually satisfied with all the attractive strategies you serve. Some two facts in this post are easily the simplest we've ever had.

  14. I wanted to create you that bit of note to be able to say thanks the moment again for your personal unique pointers you have featured in this article. It has been remarkably generous of people like you in giving unhampered what exactly a few people could possibly have distributed for an e book to make some bucks on their own, principally since you might have tried it in the event you desired. These strategies additionally acted like a fantastic way to fully grasp some people have a similar eagerness similar to my own to understand great deal more when it comes to this matter. Certainly there are lots of more fun situations in the future for those who start reading your blog post.

  15. Thank you so much for providing individuals with a very superb possiblity to check tips from here. It's usually so excellent plus full of amusement for me personally and my office fellow workers to visit the blog at minimum thrice in a week to read the fresh guidance you have. Not to mention, I'm always satisfied with the astonishing methods you serve. Selected 4 ideas in this posting are in reality the best we've had.

  16. I would like to show my appreciation to you for rescuing me from this type of predicament. Just after looking out through the world wide web and coming across things which are not helpful, I thought my entire life was well over. Being alive without the answers to the difficulties you've sorted out all through your main blog post is a crucial case, and ones which may have negatively damaged my entire career if I had not discovered the website. That skills and kindness in taking care of all the stuff was important. I am not sure what I would've done if I hadn't discovered such a solution like this. I am able to at this point relish my future. Thanks a lot so much for the impressive and sensible help. I won't be reluctant to refer the blog to any person who would need support about this matter.

  17. goyard tote说道:

    My husband and i got really relieved Jordan managed to do his studies while using the ideas he made while using the web page. It's not at all simplistic to just possibly be handing out ideas most people have been making money from. We really acknowledge we have the writer to appreciate because of that. The type of explanations you have made, the simple website navigation, the friendships you help to create - it's everything incredible, and it is making our son in addition to our family feel that that theme is interesting, which is certainly rather pressing. Thank you for everything!

  18. yeezy boost说道:

    Thank you for all of your hard work on this web site. Betty enjoys carrying out research and it is obvious why. A number of us learn all relating to the compelling manner you make very helpful items on this website and even invigorate response from some other people on this issue and our own child is now learning a lot. Have fun with the rest of the new year. Your performing a glorious job.

  19. I precisely needed to thank you very much again. I am not sure what I might have carried out in the absence of the entire opinions provided by you regarding such a situation. Entirely was the terrifying condition in my position, but taking a look at the very well-written tactic you treated that took me to cry for joy. I'm just happy for your information and then trust you recognize what a great job you have been carrying out training other individuals using your webpage. More than likely you haven't encountered any of us.

  20. jordan 12说道:

    Thanks so much for providing individuals with a very splendid opportunity to read from this blog. It is usually so brilliant and full of a great time for me personally and my office mates to visit your blog more than 3 times in one week to learn the fresh stuff you have. Not to mention, I'm also actually contented for the outstanding hints you serve. Certain two facts on this page are without a doubt the finest we've ever had.

  21. chrome hearts说道:

    I wanted to write a brief word to be able to appreciate you for the awesome points you are sharing at this site. My prolonged internet research has now been honored with excellent ideas to share with my classmates and friends. I 'd assert that we readers are unquestionably blessed to dwell in a remarkable site with so many brilliant individuals with insightful ideas. I feel very happy to have encountered your entire website page and look forward to some more entertaining moments reading here. Thanks once more for everything.

  22. I really wanted to write down a simple remark to appreciate you for all the fantastic pointers you are giving out on this site. My incredibly long internet research has at the end been paid with good concept to exchange with my family and friends. I would tell you that many of us visitors actually are extremely endowed to exist in a decent website with many awesome professionals with very helpful methods. I feel quite privileged to have encountered your entire site and look forward to some more entertaining times reading here. Thank you again for everything.

  23. nmd r1说道:

    I want to show thanks to you for bailing me out of this type of issue. As a result of researching through the the web and obtaining thoughts which were not beneficial, I thought my entire life was gone. Being alive devoid of the approaches to the difficulties you've sorted out by means of your good guideline is a critical case, as well as those that could have in a wrong way affected my career if I had not discovered your blog post. Your good understanding and kindness in playing with every part was helpful. I'm not sure what I would've done if I hadn't encountered such a step like this. I am able to at this moment look ahead to my future. Thanks very much for the expert and sensible guide. I will not think twice to refer your site to any person who needs to have recommendations on this matter.

  24. I would like to show some thanks to this writer just for rescuing me from this problem. After scouting throughout the world-wide-web and meeting tips that were not productive, I assumed my entire life was well over. Existing minus the solutions to the issues you've sorted out as a result of this guideline is a crucial case, and the ones which could have in a negative way affected my career if I had not encountered your site. Your own understanding and kindness in touching all the stuff was tremendous. I am not sure what I would have done if I hadn't come upon such a stuff like this. I can at this moment look forward to my future. Thanks for your time very much for your expert and sensible help. I won't be reluctant to refer your blog to any person who desires direction on this issue.

发表评论

电子邮件地址不会被公开。 必填项已用*标注