Walks

网络安全爱好者

Python3学习系列(六):Scrapy爬取Walks个人博客文章信息并保存

《Python3学习系列(六):Scrapy爬取Walks个人博客文章信息并保存》

前言:

说完理论,总要说说实战,下面说的就是爬取我的个人博客文章标题,链接,访问者等的实战内容 


目录:

0x01:爬取思路

0x02:代码实现与分析

 

0x01 爬取前奏

我们在开始我们的爬虫之前,第一步要做的是什么?当然是分析网站,构思爬取全过程了。

所以,一开始说的就是爬取思路。

对我的Walks个人博客爬取思路:知道我们要爬取什么-》定位要爬取的元素-》访问文章页-》下载HTML-》提取目标数据-》处理爬取数据

0x02 代码实现和分析

  • Items.py

import scrapy

#定义一个存储数据的类,用来存储数据
class MyblogcontentItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    #pass
    #要存储的数据,标题,时间,地址,访问量
    title = scrapy.Field()
    time = scrapy.Field()
    address = scrapy.Field()
    visites = scrapy.Field()

  • Spiders/Walks_Spider.py

#encoding=utf-8
#导入爬虫类
import scrapy
#导入我们刚才建立的存储数据的类
from MyblogContent.items import MyblogcontentItem

class WalksSpider(scrapy.Spider):
    #唯一标识符
    name = "Walks"
    #允许的域名(只可以从这个域名中爬取)
    allowed_domains = ['bywalks.com']
    #开始爬取的地址
    start_urls = [
        "http://www.bywalks.com"
        ]
    
    #对爬取的数据(HTML)进行提取
    def parse(self,response):
        #调用Selector,sel就是返回的HTML
        sel = scrapy.selector.Selector(response)
        #定位爬取数据
        sites = sel.xpath("//div[@id='kratos-blog-post']/div/div/section/article")
        items = []
        for site in sites:
            item = MyblogcontentItem()
            #从HTML中提取数据
            item["title"] =site.xpath("div/div[@class='kratos-post-inner-new']/header/h2/a/text()").extract() 
            item["time"] = site.xpath("div/div[@class='kratos-post-meta-new']/span[1]/a[1]/text()").extract() 
            item["address"] =site.xpath("div/div[@class='kratos-post-meta-new']/span[1]/a[1]/@href").extract()
            item["visites"] = site.xpath("div/div[@class='kratos-post-meta-new']/span[2]/a[1]/text()").extract()
            
            items.append(item)
        return items

  • pipelines.py

class MyblogcontentPipeline(object):
    #对提取出来的数据进行存储
    def process_item(self, item, spider):
        #打开Walks.json文件,不存在就新建    
        with open('Walks.json','a+') as f:
            #写入我们爬取的数据   
            f.write("title = "+str(item["title"])+'\n')
            f.write("time = "+str(item["time"])+'\n')
            f.write("address = "+str(item["address"])+'\n')
            f.write("visites = "+str(item["visites"])+'\n\n')
        #返回item,给下一个处理函数处理(如果有的话)
        return item

  • sittings.py

BOT_NAME = 'MyblogContent'

SPIDER_MODULES = ['MyblogContent.spiders']
NEWSPIDER_MODULE = 'MyblogContent.spiders'
#上面都是直接生成了的,不用添加
#下面使我们自己需要添加的代码,意思就是当我们提取了所需要的数据之后,把数据传入Pipeline里面进行处理,也就是我们上面的py文件
#为什么要添加呢?因为pipeline不是一个爬虫所必须的功能,我们可以自己加也可以不加,所以当需要的时候,我们需要设置下
ITEM_PIPELINES = {  
    'MyblogContent.pipelines.MyblogcontentPipeline':300  
}

#是否遵循网站robots.txt的规则
ROBOTSTXT_OBEY = False

最后附带一张爬取的数据图片

《Python3学习系列(六):Scrapy爬取Walks个人博客文章信息并保存》

 转载请注明来自:www.bywalks.com

点赞
  1. I truly wanted to develop a message in order to thank you for the pleasant guidelines you are placing on this website. My considerable internet research has at the end been rewarded with reliable concept to write about with my family. I 'd state that that we site visitors are unequivocally blessed to live in a remarkable place with many outstanding individuals with very helpful techniques. I feel very privileged to have come across your entire site and look forward to really more amazing times reading here. Thanks a lot again for everything.

  2. lacoste polo说道:

    Thanks for all of the hard work on this site. Kate really loves working on research and it's really easy to see why. A lot of people hear all concerning the dynamic means you render valuable solutions by means of the website and even boost participation from other people on this area of interest and our simple princess is certainly starting to learn so much. Enjoy the rest of the new year. You are always doing a really good job.

  3. yeezy boost说道:

    Thank you for your whole efforts on this web site. My aunt delights in doing investigations and it is simple to grasp why. Most people know all concerning the lively medium you produce invaluable strategies via this web site and therefore strongly encourage response from some other people on that topic plus our simple princess is certainly learning a lot. Have fun with the rest of the year. Your performing a useful job.

  4. I simply wanted to thank you so much again. I'm not certain the things I would have created in the absence of the entire ideas contributed by you over that question. It had been a challenging issue in my view, nevertheless discovering the very skilled style you processed the issue made me to cry over delight. I'm just happier for this information as well as hope you comprehend what a great job your are undertaking training the others using a blog. Most probably you've never got to know any of us.

  5. kobe 9说道:

    I am also commenting to make you know of the exceptional encounter my friend's child undergone visiting your site. She mastered a wide variety of issues, including what it is like to possess a very effective teaching spirit to have certain people without difficulty thoroughly grasp specific grueling topics. You really surpassed our own expected results. Thank you for distributing those great, healthy, informative and even cool thoughts on your topic to Jane.

  6. I definitely wanted to send a brief note to be able to say thanks to you for the lovely information you are showing here. My time-consuming internet search has at the end been compensated with extremely good points to talk about with my two friends. I 'd tell you that we site visitors are very blessed to live in a superb website with so many lovely people with very helpful hints. I feel very privileged to have discovered the web site and look forward to so many more thrilling minutes reading here. Thank you once again for a lot of things.

  7. nike sneakers说道:

    I must express thanks to the writer for rescuing me from such a difficulty. As a result of surfing throughout the online world and getting strategies which are not powerful, I assumed my entire life was over. Living without the answers to the difficulties you have sorted out through your blog post is a critical case, as well as ones which might have in a wrong way damaged my entire career if I hadn't discovered your blog. Your personal know-how and kindness in controlling the whole thing was invaluable. I don't know what I would've done if I hadn't discovered such a stuff like this. I can also at this point relish my future. Thanks a lot so much for this professional and sensible guide. I won't think twice to refer your web page to any individual who needs and wants direction on this subject matter.

  8. I not to mention my buddies came reading the excellent tricks located on the blog then suddenly I got a horrible suspicion I never expressed respect to the web blog owner for those strategies. The guys had been as a consequence passionate to study them and now have absolutely been using these things. We appreciate you getting quite thoughtful and also for figuring out variety of useful useful guides millions of individuals are really eager to know about. My honest regret for not expressing appreciation to you sooner.

  9. vapormax说道:

    I simply desired to thank you so much all over again. I do not know the things that I could possibly have followed in the absence of these strategies discussed by you regarding this topic. This has been an absolute depressing scenario in my position, but discovering your skilled technique you handled it forced me to leap for contentment. I am happy for the work and have high hopes you realize what an amazing job that you are putting in instructing many others via your website. I'm certain you've never got to know all of us.

  10. adidas nmd说道:

    I really wanted to type a note so as to thank you for all the magnificent guidelines you are posting on this site. My time consuming internet search has at the end been paid with incredibly good strategies to talk about with my friends. I 'd claim that we readers are definitely blessed to be in a perfect network with very many outstanding professionals with good opinions. I feel very lucky to have seen your web page and look forward to tons of more thrilling minutes reading here. Thank you once again for everything.

  11. converse shoes说道:

    I want to convey my admiration for your kindness in support of those people that require guidance on the subject matter. Your special dedication to getting the solution all around came to be really invaluable and has continuously empowered many people like me to get to their goals. Your amazing warm and friendly useful information can mean so much a person like me and still more to my mates. Thanks a ton; from everyone of us.

  12. moncler jackets说道:

    Thanks so much for providing individuals with remarkably pleasant possiblity to discover important secrets from this site. It's usually very great and also full of amusement for me and my office fellow workers to visit your web site nearly 3 times weekly to read the newest stuff you have. Not to mention, I'm always amazed for the striking opinions served by you. Selected 2 ideas in this posting are in fact the most efficient we have ever had.

  13. adidas eqt说道:

    I simply needed to appreciate you once again. I am not sure the things that I would have made to happen in the absence of these creative concepts provided by you regarding this concern. It previously was a real scary circumstance in my position, nevertheless discovering the specialized fashion you handled that forced me to jump over happiness. I'm happy for your service and then hope you find out what an amazing job that you're putting in instructing others through your website. More than likely you have never come across all of us.

  14. fila disruptor说道:

    I wish to express thanks to this writer for rescuing me from this setting. Just after surfing through the the web and getting ideas that were not productive, I believed my life was over. Being alive without the presence of answers to the difficulties you've solved as a result of the review is a crucial case, as well as the kind that might have adversely damaged my career if I had not noticed your blog. Your main talents and kindness in dealing with a lot of things was precious. I'm not sure what I would have done if I hadn't encountered such a thing like this. I am able to at this point look ahead to my future. Thanks a lot very much for this expert and result oriented guide. I will not hesitate to endorse your web page to any individual who requires recommendations about this topic.

  15. hermes bags说道:

    Thanks a lot for giving everyone a very terrific possiblity to read critical reviews from here. It's usually so beneficial and as well , packed with amusement for me and my office acquaintances to search the blog not less than 3 times a week to read through the latest things you have got. Not to mention, I'm also usually contented considering the astonishing pointers you serve. Certain two ideas in this article are clearly the finest we have had.

  16. I simply had to appreciate you once more. I do not know what I would have undertaken in the absence of the actual solutions documented by you over such industry. It was actually a real difficult setting for me, nevertheless spending time with your specialized avenue you solved it made me to weep over delight. Now i am grateful for your help and thus wish you know what a powerful job you are always putting in training men and women all through your webblog. I am certain you have never got to know all of us.

  17. moncler说道:

    I have to voice my appreciation for your kind-heartedness in support of those people that have the need for guidance on that situation. Your real commitment to getting the message throughout appeared to be pretty helpful and have surely encouraged many people like me to achieve their targets. Your invaluable tips and hints denotes a great deal to me and somewhat more to my office colleagues. Thank you; from all of us.

  18. yeezy 700说道:

    My wife and i felt absolutely excited Jordan could do his survey by way of the precious recommendations he made through the blog. It's not at all simplistic to simply be freely giving tips and hints which usually many others could have been selling. And we also know we now have the writer to give thanks to for that. All the illustrations you've made, the simple website menu, the relationships you can give support to create - it is everything exceptional, and it's really helping our son in addition to us believe that the article is awesome, and that's really indispensable. Many thanks for everything!

  19. michael kors说道:

    I and my pals were found to be checking out the best information located on the blog and then quickly came up with a terrible feeling I had not expressed respect to the website owner for them. All the boys happened to be very interested to read all of them and now have really been making the most of these things. Thanks for truly being considerably accommodating as well as for picking this form of impressive useful guides millions of individuals are really eager to be informed on. Our own sincere regret for not expressing appreciation to you earlier.

  20. adidas yeezy说道:

    I happen to be writing to make you be aware of what a excellent experience my friend's princess experienced checking your web site. She noticed many things, most notably what it's like to possess a marvelous helping spirit to let many more without difficulty comprehend selected hard to do topics. You undoubtedly exceeded our own expected results. I appreciate you for supplying the interesting, safe, edifying and unique thoughts on the topic to Sandra.

  21. yeezy sneakers说道:

    I want to express appreciation to the writer just for rescuing me from this incident. As a result of searching through the the web and obtaining techniques which are not beneficial, I assumed my life was gone. Being alive without the approaches to the issues you've resolved all through the article content is a serious case, as well as the kind that might have in a wrong way damaged my entire career if I had not encountered your site. Your own personal knowledge and kindness in maneuvering every item was valuable. I am not sure what I would've done if I had not come across such a step like this. I can also at this time look forward to my future. Thanks for your time so much for this expert and effective guide. I won't think twice to recommend your blog post to anyone who needs and wants guidance on this situation.

  22. bape hoodie说道:

    Thank you a lot for providing individuals with an extremely superb chance to read articles and blog posts from this web site. It's always so nice and as well , packed with amusement for me personally and my office mates to visit the blog at the least thrice a week to see the fresh secrets you will have. And of course, I'm so always pleased with your surprising information you serve. Selected 2 facts in this post are absolutely the most suitable I've had.

  23. louboutin shoes说道:

    I really wanted to type a small remark to appreciate you for all of the remarkable tricks you are writing on this site. My particularly long internet research has at the end of the day been paid with sensible points to write about with my partners. I 'd tell you that many of us website visitors are unequivocally fortunate to dwell in a fabulous community with so many wonderful professionals with insightful methods. I feel very much grateful to have discovered your entire website and look forward to really more fabulous minutes reading here. Thank you once again for a lot of things.

发表评论

电子邮件地址不会被公开。 必填项已用*标注