Walks

网络安全爱好者

Python3学习系列(七):Scrapy如何过滤重复数据

《Python3学习系列(七):Scrapy如何过滤重复数据》

前言:

本章内容是如何对爬取的数据进行去重,当你爬取了很多内容之后,这个去重复就尤为重要。 

目录: 

0x01:去重方法

0x02:代码实现 


0x01 去重方法

1:对URL进行去重(判断URL是否重复)

2:对数据进行去重(判断是否是重复数据)


0x02 代码实现

URL去重

#dont_filter=False 就是url是否处理的意思,有一个自带的过滤器
yield scrapy.Request(url, self.parse, dont_filter=False)

数据去重

from scrapy.exceptions import DropItem 

class DuplicatesPipeline(object): 
    def __init__(self):
        #set为集合,集合为数据不重复无排序的 
        self.data_set = set() 
         

    def process_item(self, item, spider):
        data = item['data']
        #如果数据已经存在,抛出DropItem异常,丢弃item数据,不会再传给后面继续处理
        if data in self.data_set: 
            raise DropItem("Duplicate item found: %s" % item) 
        #不存在时,就添加数据 
        self.data_set.add(item['data']) 
        return item

坚持下去吧,相信看完我的文章,你对于Scrapy的运用也会有一个新的理解的

转载请注明来自:www.bywalks.com

点赞
  1. nike vapormax说道:

    I enjoy you because of each of your efforts on this web page. My mother really loves participating in investigation and it is obvious why. My spouse and i notice all relating to the compelling way you make both interesting and useful strategies by means of the web site and as well boost contribution from other individuals about this area of interest while our own daughter is in fact becoming educated so much. Have fun with the remaining portion of the year. You're the one doing a glorious job.

  2. golden goose说道:

    I actually wanted to type a simple note to express gratitude to you for all of the remarkable recommendations you are sharing at this website. My extended internet lookup has at the end been honored with good quality details to talk about with my family members. I would tell you that most of us website visitors are very much fortunate to live in a great network with so many special individuals with valuable secrets. I feel quite blessed to have come across your entire web site and look forward to plenty of more excellent times reading here. Thanks once again for everything.

  3. nike flyknit说道:

    I must show appreciation to this writer for bailing me out of such a problem. Just after looking out through the internet and finding ways which were not productive, I believed my life was over. Living without the answers to the issues you've fixed all through your short article is a crucial case, as well as the ones which might have in a negative way affected my career if I hadn't encountered your web blog. Your primary expertise and kindness in taking care of all the stuff was very useful. I am not sure what I would have done if I had not encountered such a stuff like this. I can at this time look forward to my future. Thanks for your time so much for the reliable and sensible help. I will not think twice to recommend the website to any individual who would like recommendations about this issue.

  4. My spouse and i felt absolutely ecstatic that Albert could do his investigation through the precious recommendations he had out of the web page. It is now and again perplexing just to find yourself giving freely instructions that many the rest might have been making money from. And now we keep in mind we've got the website owner to give thanks to for this. The most important explanations you made, the simple blog navigation, the friendships your site give support to instill - it's got many fantastic, and it is letting our son in addition to the family reckon that this idea is fun, and that is particularly vital. Many thanks for all!

  5. adidas tubular x说道:

    I precisely wished to say thanks once more. I do not know what I would've followed in the absence of these creative ideas discussed by you on my area. This was a very fearsome condition in my opinion, however , discovering a new skilled tactic you handled it made me to leap for joy. I am just happy for the work and then trust you realize what a powerful job that you are doing training the rest via a site. More than likely you haven't met any of us.

  6. I really wanted to develop a small comment so as to appreciate you for all the precious advice you are giving out here. My considerable internet look up has finally been honored with beneficial concept to write about with my pals. I would admit that we visitors are really blessed to dwell in a remarkable community with so many lovely individuals with great plans. I feel pretty grateful to have seen your web pages and look forward to tons of more amazing times reading here. Thank you once more for a lot of things.

  7. SvobodaMooro说道:

    cat 1 keyword is

    9042 is welche dosierung

  8. RebekkaMooro说道:

    is e glicemia

    88fc use of is for men

  9. yeezy boost说道:

    Thanks a lot for providing individuals with an extraordinarily superb possiblity to discover important secrets from this blog. It is often very fantastic and as well , stuffed with a good time for me and my office fellow workers to visit your website more than thrice weekly to learn the new secrets you have got. Of course, I'm usually impressed with all the surprising principles you serve. Certain 4 ideas in this post are undoubtedly the very best we have all ever had.

  10. RebekkaMooro说道:

    acheter is genev

    88fc is rosario argentina

  11. golden goose说道:

    I needed to create you one little bit of note to be able to say thank you the moment again on the exceptional concepts you've shown on this page. It's simply tremendously open-handed with people like you to present easily precisely what numerous people could possibly have distributed for an ebook in making some money for their own end, principally seeing that you might have tried it if you desired. The concepts likewise worked to be a fantastic way to be certain that other individuals have a similar zeal just like my own to figure out a whole lot more pertaining to this problem. I am sure there are lots of more enjoyable instances ahead for those who look over your website.

  12. RebekkaMooro说道:

    is gnc

    88fc is online 25mg

  13. KallinikaMooro说道:

    is e insufficienza renale

    441d is online mastercard

  14. KallinikaMooro说道:

    vente de is au maroc

    441d enter site canadian is

  15. reebok shoes说道:

    I definitely wanted to write a simple message to express gratitude to you for all of the magnificent recommendations you are writing on this site. My considerable internet lookup has finally been rewarded with incredibly good facts to exchange with my friends. I would believe that many of us website visitors are very blessed to exist in a good community with very many awesome people with insightful solutions. I feel truly lucky to have come across the webpages and look forward to so many more exciting moments reading here. Thanks a lot again for all the details.

  16. KallinikaMooro说道:

    is 20 mg 12 compresse

    441d is rezeptfrei online

  17. My wife and i were really lucky when Michael could complete his investigation using the precious recommendations he was given through your weblog. It's not at all simplistic to just be making a gift of helpful tips that many many people may have been making money from. And we also remember we need the blog owner to appreciate for that. These explanations you have made, the straightforward website menu, the friendships you help to instill - it is all sensational, and it is leading our son in addition to our family imagine that this issue is enjoyable, which is tremendously important. Thank you for everything!

  18. KristianaMooro说道:

    is online kaufen express

    b5db compra de is alicante

  19. I simply wished to thank you very much all over again. I am not sure what I would've accomplished without the opinions discussed by you about that question. It has been a very difficult situation in my opinion, but observing your expert technique you treated that took me to cry for delight. Now i'm grateful for this work and then hope you find out what a great job you are always undertaking educating the rest by way of your site. More than likely you haven't got to know any of us.

  20. KristianaMooro说道:

    is e sangue

    b5db is for men works

  21. KristianaMooro说道:

    the best place is purchase

    b5db is is online real

  22. My husband and i got absolutely glad when Jordan could complete his homework from the ideas he grabbed in your weblog. It's not at all simplistic to simply always be giving out secrets and techniques which often other people could have been trying to sell. And we all see we've got the blog owner to give thanks to because of that. Those explanations you've made, the easy blog menu, the friendships you will help foster - it is most awesome, and it's really aiding our son and us understand the idea is fun, which is particularly mandatory. Thanks for the whole thing!

  23. GordianMooro说道:

    ou acheter du is

    be07 where to be safe is

  24. chrome hearts说道:

    I wish to voice my admiration for your kindness in support of individuals that must have assistance with in this topic. Your very own dedication to getting the solution all around has been rather important and has really permitted guys and women like me to get to their pursuits. Your entire useful tutorial implies much to me and somewhat more to my peers. Warm regards; from all of us.

  25. GordianMooro说道:

    be is online in eu

    be07 is legally

  26. ErmilMooro说道:

    order is mexico

    aa72 is y ventas

  27. moncler jackets说道:

    I and my guys came examining the excellent solutions from your web site while instantly developed a horrible feeling I never thanked you for those secrets. All of the young boys are already totally passionate to see all of them and have surely been taking advantage of those things. Appreciate your genuinely very thoughtful and for getting variety of great things most people are really desirous to understand about. Our honest apologies for not expressing gratitude to you earlier.

  28. ErmilMooro说道:

    is und co kaufen

    aa72 at is in europe

  29. ErmilMooro说道:

    billige is kaufen online

    aa72 is free com forum

  30. air jordan说道:

    I want to show appreciation to this writer for rescuing me from this type of predicament. Just after looking out through the the web and coming across strategies which were not helpful, I figured my life was over. Living minus the strategies to the problems you've resolved through your entire guide is a crucial case, as well as the ones that would have negatively affected my career if I hadn't come across your web page. Your own natural talent and kindness in taking care of all the things was invaluable. I'm not sure what I would've done if I hadn't come across such a point like this. I am able to at this point look ahead to my future. Thanks for your time very much for this reliable and result oriented guide. I will not be reluctant to recommend your web page to anyone who would need counselling about this matter.

  31. yeezy boost说道:

    I not to mention my guys ended up analyzing the best hints from the blog while all of the sudden came up with a horrible suspicion I never thanked the web blog owner for those strategies. All the young men are actually certainly happy to read through all of them and have now pretty much been having fun with these things. Many thanks for truly being indeed accommodating and also for going for certain cool subject areas most people are really eager to understand about. Our honest regret for not expressing gratitude to you sooner.

  32. I would like to convey my respect for your kindness supporting individuals who really want help on in this theme. Your personal dedication to getting the message all through came to be astonishingly interesting and has all the time encouraged folks just like me to attain their objectives. Your new helpful key points denotes a lot a person like me and a whole lot more to my mates. With thanks; from all of us.

  33. My wife and i ended up being so more than happy that Michael could carry out his reports through the ideas he made in your web site. It is now and again perplexing to simply possibly be offering helpful tips that some people may have been making money from. Therefore we understand we've got the website owner to give thanks to for that. All the illustrations you've made, the simple site navigation, the relationships you will make it possible to create - it is everything incredible, and it's aiding our son and our family understand this matter is exciting, which is highly mandatory. Thanks for everything!

  34. yeezy boost 350说道:

    I am only commenting to let you know of the wonderful experience my wife's daughter undergone checking your web page. She picked up a lot of details, including what it is like to have an ideal giving character to get a number of people really easily completely grasp specific complex subject matter. You really did more than our expected results. Thanks for providing those powerful, trusted, informative and in addition unique guidance on this topic to Tanya.

  35. longchamp说道:

    I intended to create you a little bit of word to help say thanks again relating to the magnificent secrets you've provided on this site. It is quite tremendously generous of people like you to allow extensively exactly what a few individuals might have made available as an e-book to end up making some profit on their own, particularly since you could have tried it in case you considered necessary. The suggestions additionally served to be the good way to comprehend most people have the identical passion similar to mine to learn a lot more on the topic of this condition. I am sure there are millions of more fun times up front for many who looked at your website.

  36. KlementinaMooro说道:

    where to be is in doha

    d5cb look there online order is

  37. yeezy boost 350说道:

    I must show thanks to the writer for rescuing me from this difficulty. Because of surfing around through the world-wide-web and obtaining proposals which were not pleasant, I believed my life was done. Existing devoid of the strategies to the issues you have fixed by way of your good review is a crucial case, and the kind that would have badly affected my entire career if I hadn't discovered your site. The skills and kindness in playing with all the pieces was very useful. I am not sure what I would have done if I had not come upon such a subject like this. I am able to at this point relish my future. Thanks so much for the expert and result oriented guide. I will not be reluctant to refer your blog post to any individual who should have direction about this area.

  38. I and also my pals came looking through the great items on your website then instantly developed an awful feeling I never thanked the web site owner for them. These men were as a result passionate to see all of them and have surely been taking advantage of those things. Appreciate your getting really kind and then for making a choice on varieties of impressive areas millions of individuals are really desperate to discover. My personal honest regret for not expressing appreciation to earlier.

  39. KlementinaMooro说道:

    be is sample pack

    d5cb is dose should take

  40. yeezy boost 350说道:

    Thanks so much for providing individuals with an extremely spectacular chance to check tips from here. It really is so nice and jam-packed with fun for me and my office fellow workers to visit your blog minimum 3 times every week to read through the fresh stuff you have. And definitely, I am just certainly impressed with your astounding solutions served by you. Some 3 tips on this page are in fact the simplest I've had.

  41. cheappfizer is 100gm

    b950 is different doses

  42. yeezy boost 350说道:

    My wife and i got joyous that Ervin managed to finish off his basic research through the entire precious recommendations he got from your very own web page. It is now and again perplexing to simply find yourself giving for free techniques that many a number of people have been making money from. And we also understand we need the website owner to give thanks to for this. The illustrations you have made, the straightforward web site menu, the friendships you aid to instill - it's got most excellent, and it's assisting our son and our family imagine that this subject is pleasurable, which is especially essential. Thanks for the whole lot!

  43. hermes说道:

    I would like to get across my love for your kindness in support of men and women who really need assistance with in this content. Your special commitment to passing the solution all-around appears to be remarkably useful and have in most cases allowed individuals much like me to attain their ambitions. Your warm and friendly advice entails a lot a person like me and extremely more to my office colleagues. With thanks; from everyone of us.

  44. can i be is at boots uk

    b950 que is yahoo es

  45. I together with my guys have already been going through the nice secrets and techniques found on your site then all of the sudden I got a terrible suspicion I never expressed respect to the web site owner for those tips. All of the women happened to be as a consequence glad to see all of them and already have actually been taking pleasure in those things. Thanks for simply being very thoughtful and then for getting this kind of great tips millions of individuals are really desperate to learn about. My personal sincere regret for not expressing gratitude to you sooner.

  46. My wife and i felt so happy when Jordan could round up his investigation through the entire ideas he received from your very own weblog. It is now and again perplexing just to choose to be giving freely guidelines which often other people may have been trying to sell. And now we see we have the blog owner to appreciate for this. All of the explanations you've made, the simple web site navigation, the relationships you can give support to instill - it is mostly incredible, and it's really leading our son and the family reckon that the theme is satisfying, and that is incredibly indispensable. Many thanks for all!

发表评论

电子邮件地址不会被公开。 必填项已用*标注