Walks

网络安全爱好者

Python3学习系列(十一):Scrapy下载文件和图片

《Python3学习系列(十一):Scrapy下载文件和图片》

前言:

可能在大家学习爬虫的过程中已经学会了如何去下载文件和图片的,无非就是传递一个URL地址然后下载即可,那么在Scrapy这个框架中,又是如何下载文件和图片的呢?

在Scrapy中,存在两个处理管道,分别是FilesPipeline,ImagesPipeline,一个用来下载文件,一个用来下载图片,下面用两个实例来说明这两个管道的用法


FilesPipeline:

下载下图网站的代码文件

《Python3学习系列(十一):Scrapy下载文件和图片》

  • items.py

import scrapy


class DownFilesItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    #pass
    #file_urls用来存放要下载的文件url地址
    file_urls = scrapy.Field()
    files = scrapy.Field()
  • settings.py

BOT_NAME = 'Down_Files'

SPIDER_MODULES = ['Down_Files.spiders']
NEWSPIDER_MODULE = 'Down_Files.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Down_Files (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

ITEM_PIPELINES = {
    #'Down_Files.pipelines.DownFilesPipeline': 300,
    #自动下载文件,本来是FilesPipeline,为了改变下载文件的文件名,重写了FilesPipeline
    #'scrapy.pipelines.files.FilesPipeline':1,
    'Down_Files.pipelines.DownFilesPipeline':1,
}
#下载的文件存放路径
FILES_STORE='Down_Files'
  • pipelines.py

from scrapy.pipelines.files import FilesPipeline
from urllib.parse import urlparse
from os.path import basename,dirname,join
#改变下载的文件名
class DownFilesPipeline(FilesPipeline):
    #Scrapy源代码用这个方法来命名,return(目录名,文件名),我们改变返回值的目录名和文件名即可
    def file_path(self,request,response=None,info=None):
        path =  urlparse(request.url).path
        return join(basename(dirname(path)),basename(path))
  • Down_Files.py

# -*- coding: utf-8 -*-
import scrapy

from scrapy.linkextractors import LinkExtractor
from Down_Files.items import DownFilesItem


class DownFileSpider(scrapy.Spider):
    name = 'Down_File'
    allowed_domains = ['matplotlib.org']
    start_urls = ['http://matplotlib.org/examples/index.html']

    def parse(self,response):
        le = LinkExtractor(restrict_xpaths=('//*[@id="matplotlib-examples"]/div/ul/li/ul/li/a'),deny='/index.html$')
        for link in le.extract_links(response):
            yield scrapy.Request(link.url,callback=self.parse_files)

    def parse_files(self, response):
        href = response.xpath('//div[@class="body"]/div/p/a/@href').extract_first()
        url = response.urljoin(href)
        item = DownFilesItem()
        item['file_urls'] = [url]
        return item

成果图如下

《Python3学习系列(十一):Scrapy下载文件和图片》


ImagesPipeline:

下载So_Image的图片

  • items.py

import scrapy


class DownSoImageItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    #跟下载文件一样,存放图片的url地址
    image_urls = scrapy.Field()
  • settings.py

BOT_NAME = 'Down_So_Image'

SPIDER_MODULES = ['Down_So_Image.spiders']
NEWSPIDER_MODULE = 'Down_So_Image.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Down_So_Image (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

DEFAULT_REQUEST_HEADERS = {
   #'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   #'Accept-Language': 'en',
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1',
}
ITEM_PIPELINES = {
    #'Down_So_Image.pipelines.DownSoImagePipeline': 300,
    'scrapy.pipelines.images.ImagesPipeline':1,
}

IMAGES_STORE = 'Down_Images'
  • Down_Images.py

import scrapy

from scrapy.linkextractors import LinkExtractor
from Down_Files.items import DownFilesItem


class DownFileSpider(scrapy.Spider):
    name = 'Down_File'
    allowed_domains = ['matplotlib.org']
    start_urls = ['http://matplotlib.org/examples/index.html']

    def parse(self,response):
        le = LinkExtractor(restrict_xpaths=('//*[@id="matplotlib-examples"]/div/ul/li/ul/li/a'),deny='/index.html$')
        for link in le.extract_links(response):
            yield scrapy.Request(link.url,callback=self.parse_files)

    def parse_files(self, response):
        href = response.xpath('//div[@class="body"]/div/p/a/@href').extract_first()
        url = response.urljoin(href)
        item = DownFilesItem()
        item['file_urls'] = [url]
        return item

成果图片如下

《Python3学习系列(十一):Scrapy下载文件和图片》

参考资料:《精通Scrapy网络爬虫》

转载请注明来自:www.bywalks.com

点赞
  1. off white说道:

    I definitely wanted to post a simple message in order to say thanks to you for all the fabulous solutions you are writing here. My long internet lookup has at the end of the day been rewarded with reputable insight to talk about with my contacts. I would declare that we site visitors are very fortunate to dwell in a useful place with many marvellous people with very helpful tips. I feel extremely blessed to have used your web page and look forward to some more cool times reading here. Thanks a lot once more for everything.

  2. I'm also commenting to let you be aware of of the really good discovery our child enjoyed using your web site. She noticed a wide variety of issues, which included what it's like to possess an ideal coaching mood to let others effortlessly have an understanding of certain tricky things. You really surpassed readers' expectations. Thank you for delivering these important, safe, edifying as well as easy tips about that topic to Mary.

  3. adidas yeezy说道:

    I simply wanted to say thanks all over again. I'm not certain the things I could possibly have sorted out in the absence of those aspects documented by you on such a area of interest. It was actually a very terrifying dilemma in my position, nevertheless observing the specialized fashion you managed that took me to jump with happiness. I am just happier for the information and then have high hopes you know what an amazing job you happen to be providing instructing other individuals with the aid of your blog post. I am certain you haven't got to know all of us.

  4. nike foamposite说道:

    I have to show my admiration for your kind-heartedness in support of folks that must have assistance with this subject. Your personal dedication to getting the message all through had been quite significant and have consistently helped most people like me to attain their aims. Your entire warm and helpful advice can mean this much to me and additionally to my mates. Regards; from each one of us.

  5. kobe sneakers说道:

    I in addition to my friends ended up reading through the excellent thoughts from your web site while at once developed a horrible feeling I had not thanked the site owner for those secrets. Most of the ladies ended up for this reason thrilled to see all of them and have now very much been taking advantage of those things. Thank you for being indeed kind and for figuring out varieties of outstanding subjects most people are really needing to learn about. Our sincere regret for not expressing appreciation to earlier.

  6. converse outlet说道:

    Thank you a lot for giving everyone an extraordinarily remarkable opportunity to read from this blog. It's usually very kind and also full of fun for me and my office colleagues to visit your website at minimum three times per week to read through the newest stuff you will have. And definitely, I'm always amazed concerning the good hints served by you. Selected 2 points in this posting are particularly the simplest we have all ever had.

  7. lebron 15 shoes说道:

    I enjoy you because of all of the hard work on this site. My mom loves conducting investigation and it's really easy to see why. We all know all of the compelling method you render important information on this web site and in addition recommend contribution from other people on that situation so our favorite daughter is in fact learning a whole lot. Take pleasure in the rest of the year. You are conducting a splendid job.

  8. hermes belt说道:

    I wish to express some appreciation to you for bailing me out of this type of scenario. Just after researching through the world-wide-web and coming across notions that were not powerful, I believed my entire life was over. Being alive minus the solutions to the difficulties you've solved all through this short article is a crucial case, and the ones which may have negatively damaged my career if I hadn't noticed the website. Your own skills and kindness in maneuvering almost everything was vital. I am not sure what I would've done if I had not come across such a step like this. I am able to at this time look forward to my future. Thanks very much for the specialized and results-oriented guide. I won't think twice to suggest your web sites to anyone who should have guide on this subject.

  9. yeezy boost 350说道:

    I enjoy you because of all of your efforts on this blog. Debby delights in making time for research and it's really easy to understand why. All of us notice all regarding the compelling ways you provide helpful ideas through this blog and as well as increase response from visitors on that subject while our favorite girl is without a doubt becoming educated a lot of things. Take advantage of the remaining portion of the new year. You are doing a dazzling job.

  10. I precisely desired to say thanks all over again. I do not know the things I could possibly have handled in the absence of the entire pointers documented by you relating to such field. This has been an absolute scary circumstance for me personally, but spending time with your skilled way you solved that forced me to weep with happiness. I will be thankful for this support and as well , believe you are aware of a powerful job that you're accomplishing training the rest by way of your blog post. Probably you haven't come across all of us.

  11. hermes handbags说道:

    I wish to show thanks to this writer just for bailing me out of such a situation. Because of scouting through the world wide web and getting advice that were not helpful, I figured my entire life was over. Being alive devoid of the answers to the problems you've solved through your good posting is a serious case, and the ones that might have negatively damaged my entire career if I hadn't come across your web site. The knowledge and kindness in dealing with all the things was vital. I'm not sure what I would've done if I had not come across such a thing like this. I'm able to now look ahead to my future. Thanks so much for this specialized and results-oriented guide. I will not hesitate to recommend your web page to anyone who wants and needs guidance on this subject.

  12. yeezy shoes说道:

    Thank you so much for providing individuals with such a nice chance to read in detail from this blog. It's usually so nice and also packed with amusement for me and my office mates to search your site not less than 3 times in one week to see the newest issues you will have. And lastly, I am just always pleased for the incredible things served by you. Certain two ideas on this page are easily the finest we have ever had.

  13. air jordan 11说道:

    I simply wanted to write a remark so as to thank you for all of the remarkable steps you are giving on this website. My prolonged internet research has now been rewarded with sensible concept to share with my visitors. I 'd assume that most of us website visitors actually are rather fortunate to live in a good place with very many marvellous professionals with beneficial secrets. I feel extremely privileged to have used your entire web page and look forward to many more excellent moments reading here. Thanks a lot once more for a lot of things.

  14. I would like to point out my passion for your kind-heartedness for those people that actually need guidance on this topic. Your real commitment to getting the solution up and down had become remarkably invaluable and have in every case made workers like me to achieve their aims. This important suggestions can mean a great deal to me and substantially more to my fellow workers. Best wishes; from everyone of us.

  15. yeezys说道:

    I want to express my thanks to this writer just for bailing me out of such a challenge. After researching through the the web and coming across proposals which were not powerful, I believed my entire life was well over. Existing devoid of the answers to the difficulties you've resolved by means of the short post is a crucial case, and ones which could have badly affected my career if I hadn't discovered your site. Your main understanding and kindness in handling a lot of stuff was valuable. I don't know what I would've done if I had not come across such a step like this. I can at this time relish my future. Thanks for your time so much for your reliable and amazing help. I won't be reluctant to endorse your web site to any individual who ought to have support on this area.

  16. ferragamo belt说道:

    I simply wanted to thank you so much all over again. I do not know the things I would've worked on without these points provided by you on this industry. Completely was an absolute troublesome condition for me personally, nevertheless discovering your specialised approach you handled the issue made me to weep for fulfillment. I'm just happier for this support and as well , pray you know what an amazing job you are getting into educating most people all through your webblog. I am sure you have never got to know any of us.

  17. off white说道:

    I'm also writing to let you know what a terrific discovery my child experienced reading your site. She even learned several issues, including what it is like to possess an incredible coaching spirit to let men and women with no trouble have an understanding of chosen impossible topics. You undoubtedly did more than our own expected results. Many thanks for presenting those practical, trustworthy, explanatory not to mention unique thoughts on the topic to Sandra.

  18. gucci belt说道:

    I enjoy you because of your whole efforts on this web site. Gloria takes pleasure in making time for research and it is simple to grasp why. A lot of people notice all of the compelling manner you provide helpful solutions by means of the web blog and as well improve response from people on this idea while our own girl is actually becoming educated a whole lot. Enjoy the remaining portion of the year. You're performing a powerful job.

  19. nike max说道:

    Thank you for your whole effort on this web page. My aunt delights in managing research and it is easy to understand why. My spouse and i hear all relating to the powerful medium you produce very useful tactics on this website and improve contribution from people on that situation so our favorite simple princess is truly understanding a great deal. Enjoy the remaining portion of the new year. Your performing a good job.

  20. birkin bag说道:

    I'm writing to let you be aware of of the amazing encounter my wife's daughter experienced visiting your blog. She discovered too many issues, not to mention what it's like to possess an ideal coaching mindset to have certain people without problems have an understanding of a variety of problematic issues. You actually did more than visitors' desires. Many thanks for supplying those insightful, trustworthy, explanatory and in addition cool thoughts on the topic to Sandra.

  21. I simply wished to say thanks yet again. I do not know the things I might have accomplished in the absence of the actual smart ideas shown by you regarding that theme. Certainly was the fearsome dilemma in my position, nevertheless looking at a new professional form you dealt with that made me to cry over contentment. I'm just happy for this support and as well , pray you are aware of a powerful job you were getting into instructing most people using your web site. Most probably you've never come across any of us.

  22. I precisely desired to say thanks all over again. I'm not certain the things that I could possibly have tried in the absence of the entire concepts provided by you about this area. It absolutely was a daunting circumstance for me, nevertheless finding out your specialized style you treated it made me to leap with fulfillment. I'm just grateful for this assistance and thus sincerely hope you really know what a powerful job you have been accomplishing instructing other individuals with the aid of your site. Most likely you have never got to know all of us.

  23. I just wanted to construct a brief word to thank you for the wonderful guides you are placing here. My time-consuming internet lookup has finally been paid with sensible details to talk about with my guests. I would tell you that we visitors are unequivocally endowed to exist in a decent community with very many marvellous professionals with interesting hints. I feel quite happy to have discovered your entire website and look forward to many more brilliant minutes reading here. Thank you again for all the details.

  24. ultra boost 3.0说道:

    My husband and i ended up being really happy Albert could finish up his analysis because of the ideas he got out of your web page. It is now and again perplexing to simply continually be releasing strategies a number of people might have been making money from. And now we keep in mind we have got the website owner to give thanks to for this. Most of the illustrations you have made, the easy site navigation, the relationships you help instill - it's everything astounding, and it's really aiding our son and the family consider that this situation is amusing, and that is rather vital. Thanks for everything!

  25. golden goose说道:

    Thanks for all of the hard work on this blog. My aunt takes pleasure in engaging in investigation and it's simple to grasp why. A lot of people hear all regarding the dynamic mode you give valuable strategies via this web blog and cause response from other people on the article plus our simple princess is always learning so much. Have fun with the remaining portion of the year. You have been conducting a terrific job.

  26. adidas gazelle说道:

    I intended to create you a tiny word to help thank you so much as before with your exceptional tricks you have contributed in this article. It is so seriously open-handed of you to supply extensively just what most people might have offered for sale for an e-book to help make some dough on their own, and in particular considering that you could possibly have done it in case you decided. The good ideas also served like the good way to be sure that most people have similar passion similar to mine to know whole lot more in respect of this issue. I believe there are numerous more pleasurable situations ahead for those who discover your website.

  27. Thanks for all your work on this website. My aunt really likes getting into investigations and it's simple to grasp why. My partner and i notice all regarding the lively ways you render important information by means of the web site and in addition attract participation from others on that subject so our favorite daughter is without question becoming educated a whole lot. Enjoy the remaining portion of the new year. Your performing a tremendous job.

  28. hermes birkin说道:

    A lot of thanks for every one of your hard work on this web page. Gloria loves managing investigation and it's easy to see why. My spouse and i know all concerning the dynamic tactic you convey worthwhile suggestions through the blog and in addition encourage participation from others on this matter and our favorite child is certainly studying a lot of things. Take advantage of the rest of the new year. You're conducting a superb job.

  29. red bottom shoes说道:

    I have to express my appreciation to this writer just for rescuing me from this particular crisis. Just after exploring throughout the the net and getting strategies which were not productive, I was thinking my entire life was gone. Existing minus the strategies to the difficulties you've solved through your good short post is a crucial case, and the kind which might have negatively affected my entire career if I had not come across your web blog. Your personal talents and kindness in controlling the whole thing was priceless. I don't know what I would've done if I had not encountered such a subject like this. I can also at this time relish my future. Thank you so much for the skilled and result oriented help. I won't be reluctant to refer the sites to any individual who should get guide on this subject.

  30. I must express appreciation to you for bailing me out of this type of situation. As a result of checking throughout the search engines and coming across concepts that were not helpful, I was thinking my life was over. Existing without the presence of strategies to the difficulties you've fixed through this blog post is a serious case, and the kind that might have in a negative way affected my career if I had not discovered your web blog. Your main natural talent and kindness in taking care of all the stuff was excellent. I'm not sure what I would've done if I had not encountered such a subject like this. I'm able to now relish my future. Thanks so much for this impressive and effective help. I won't hesitate to recommend your site to anybody who would like guide on this subject matter.

  31. I have to show thanks to the writer for rescuing me from this situation. Right after checking through the the web and seeing suggestions that were not helpful, I was thinking my entire life was well over. Existing without the presence of solutions to the problems you've fixed through your entire report is a crucial case, as well as the ones which may have negatively damaged my career if I hadn't noticed your website. Your main capability and kindness in taking care of the whole thing was excellent. I am not sure what I would've done if I had not come across such a point like this. I am able to at this time look ahead to my future. Thanks very much for this high quality and results-oriented help. I won't think twice to endorse your web sites to anybody who requires tips on this issue.

  32. longchamp bags说道:

    I wanted to compose a word so as to say thanks to you for all of the magnificent instructions you are posting on this site. My time-consuming internet search has finally been rewarded with pleasant points to go over with my relatives. I 'd believe that most of us readers actually are unequivocally endowed to be in a remarkable community with so many awesome people with valuable plans. I feel somewhat fortunate to have discovered your entire site and look forward to many more fabulous minutes reading here. Thank you again for a lot of things.

  33. yeezy boost 350说道:

    A lot of thanks for your entire work on this web site. Kate enjoys conducting investigations and it's obvious why. We learn all relating to the compelling manner you produce functional items on the web blog and therefore strongly encourage response from other people on that idea so our own girl has been starting to learn a lot of things. Have fun with the remaining portion of the year. You are always carrying out a glorious job.

  34. Thanks for your whole work on this site. Ellie delights in making time for internet research and it's really simple to grasp why. I hear all regarding the dynamic mode you make good information on the web blog and even invigorate response from some others about this situation while our favorite daughter is understanding a lot of things. Take advantage of the remaining portion of the new year. You are always carrying out a really good job.

  35. jordan 4说道:

    My husband and i felt so contented when Peter managed to complete his survey from the ideas he obtained through the web site. It is now and again perplexing to just be handing out helpful tips which usually some people could have been trying to sell. And now we know we have the website owner to thank because of that. The entire explanations you made, the easy web site menu, the relationships you give support to instill - it's everything sensational, and it is making our son in addition to our family understand the theme is brilliant, and that is quite mandatory. Thanks for all!

  36. adidas ultra说道:

    I as well as my friends ended up looking through the excellent guidelines from your web site and all of the sudden got an awful feeling I had not thanked the web site owner for those strategies. Those boys came certainly very interested to learn them and have in effect honestly been making the most of those things. Appreciation for indeed being very kind and for making a decision on varieties of impressive subjects millions of individuals are really wanting to be aware of. My sincere regret for not expressing appreciation to you earlier.

发表评论

电子邮件地址不会被公开。 必填项已用*标注