Walks

网络安全爱好者

Python3学习系列(五):初识Scrapy框架

《Python3学习系列(五):初识Scrapy框架》  

 

前言:

在第一篇文章写完之后,由于网站备案,所以没有写Scrapy框架的知识,后面的二三四都是在其他PT发了之后转回来的文章,下面来继续谈谈Scrapy框架。Scrapy框架很强大,在学WEB安全方面的话,也是挺有用的,例如写一个批量POC脚本,爬取些网站的信息,都可以用爬虫上面的知识去解决,而Scrapy,正是爬虫上面一个比较强大的框架。

目录: 

0x01:框架概览 

0x02:爬取流程 

0x03:目录结构简析 


0x01 框架概览

下面是一张Scrapy框架的架构图,从Spiders开始,Item Pipeline结束


《Python3学习系列(五):初识Scrapy框架》  

 0x02 爬取流程

上图的爬取流程为 Spiders把Url发给Scheduler(调度器)->调度器发送Requests给DownLoader->返回Response(也就是爬取网页的HTML)给Spiders->从HTML

中提取我们需要的数据(Item内)->传送给Item Pipeline(数据处理器)


0x03 目录结构简析

当我们开始一个建立一个Scrapy项目的时候,例如Scrapy startproject FirstProject(建立第一个Scrapy爬虫目录,FirstProject为文件名)

文件里面的项目结构是

FirstProject/
    __init__.py
    items.py
    pipelines.py
    settings.py
    spiders/
        __init__.py
        spider.py

下面说一下每一个py文件的作用。我们爬取一个网站的过程是:访问URL-》下载访问地址-》提取数据-》处理数据

items.py:建立一个存放我们想提取的数据的类

spiders/spider.py:下载URL,并且提取数据放入刚才建立的存放数据的类

pipelines:处理数据(保存等)

settings.py:配置文件(等同于软件的设置)

转载请注明来自:www.bywalks.com

点赞
  1. yeezy boost 350说道:

    I would like to voice my love for your kindness for persons who need help on in this area. Your real dedication to passing the message around ended up being exceedingly informative and have without exception encouraged women just like me to reach their pursuits. Your entire important facts entails this much a person like me and even further to my office workers. Thank you; from everyone of us.

  2. I as well as my pals were found to be digesting the good secrets located on the website then immediately I had an awful suspicion I had not thanked you for those tips. All of the boys happened to be for that reason thrilled to read through them and already have in reality been taking advantage of these things. I appreciate you for actually being well helpful and for picking out this kind of fantastic subjects millions of individuals are really desirous to discover. Our own honest apologies for not expressing appreciation to you sooner.

  3. adidas yeezy说道:

    I actually wanted to type a small message in order to appreciate you for these amazing techniques you are showing here. My time-consuming internet lookup has at the end of the day been rewarded with awesome know-how to write about with my pals. I 'd suppose that many of us readers actually are undeniably endowed to dwell in a fabulous place with so many awesome individuals with interesting things. I feel quite happy to have come across the weblog and look forward to really more amazing moments reading here. Thanks again for everything.

  4. I am only commenting to make you know what a exceptional experience my cousin's child obtained going through your web site. She came to understand a wide variety of pieces, including what it's like to have a marvelous giving spirit to have certain people really easily know precisely specified advanced subject matter. You truly did more than her expected results. Many thanks for rendering those productive, healthy, educational and also cool guidance on this topic to Lizeth.

  5. ferragamo belt说道:

    I intended to put you a very little note in order to say thanks over again on the nice basics you have discussed on this site. It has been certainly unbelievably open-handed with people like you to present freely all that many of us would've supplied for an e-book to make some profit for their own end, chiefly considering that you could have done it in case you wanted. The things also acted to become a easy way to fully grasp that other people have the same dream just as my own to learn great deal more with regard to this issue. I know there are several more pleasurable times up front for individuals who looked at your blog.

  6. air max 90说道:

    I as well as my friends appeared to be looking through the great tips located on your site while quickly got a terrible feeling I had not thanked the blog owner for those tips. These ladies came totally excited to learn them and have in effect clearly been taking pleasure in them. Appreciate your genuinely simply accommodating as well as for figuring out variety of magnificent things millions of individuals are really desirous to know about. Our sincere apologies for not expressing appreciation to earlier.

  7. links of london说道:

    I together with my pals ended up checking out the excellent procedures from your site then before long got an awful suspicion I had not thanked the website owner for those tips. The women appeared to be so warmed to learn all of them and already have extremely been using those things. We appreciate you turning out to be really helpful and also for picking this sort of remarkable areas most people are really needing to be aware of. Our honest apologies for not expressing appreciation to earlier.

  8. adidas eqt说道:

    I definitely wanted to make a remark to be able to say thanks to you for these splendid guides you are sharing here. My particularly long internet look up has finally been recognized with sensible ideas to go over with my pals. I 'd assert that most of us website visitors are unequivocally lucky to exist in a fantastic community with many lovely individuals with very helpful strategies. I feel pretty grateful to have encountered your website and look forward to some more awesome minutes reading here. Thank you again for a lot of things.

  9. adidas yeezy说道:

    My spouse and i got lucky that John managed to deal with his analysis through the entire precious recommendations he grabbed from your own site. It is now and again perplexing to simply happen to be making a gift of solutions that people have been making money from. So we remember we need the writer to thank for this. The main illustrations you made, the simple website menu, the friendships you assist to foster - it's all sensational, and it is leading our son and us know that the situation is exciting, and that's pretty fundamental. Many thanks for everything!

  10. I in addition to my pals happened to be taking note of the best tips from your web page and so unexpectedly got a terrible feeling I had not expressed respect to the blog owner for those tips. The boys are already totally stimulated to read through them and already have in reality been loving them. Thanks for being quite accommodating and for having these kinds of impressive themes millions of individuals are really eager to understand about. Our honest apologies for not expressing appreciation to you earlier.

  11. nmd r1说道:

    I simply wanted to thank you very much again. I'm not certain the things I might have gone through without the type of information shared by you directly on this subject matter. It seemed to be the challenging scenario in my position, but seeing a well-written style you resolved it made me to cry for fulfillment. I'm happier for your support and as well , sincerely hope you recognize what a powerful job that you are getting into educating many others with the aid of your website. I know that you have never encountered any of us.

  12. I as well as my pals were reviewing the great suggestions from your web page and then quickly I got a terrible feeling I had not expressed respect to the blog owner for them. Most of the guys are already as a result warmed to learn all of them and have now absolutely been taking pleasure in them. Many thanks for really being indeed considerate and for choosing some exceptional information millions of individuals are really desirous to be aware of. Our honest regret for not saying thanks to you sooner.

  13. yeezy boost 350说道:

    I actually wanted to compose a remark to express gratitude to you for some of the magnificent guidelines you are placing on this site. My time consuming internet lookup has finally been honored with extremely good facts to talk about with my two friends. I 'd claim that we readers are undoubtedly lucky to live in a good network with so many marvellous professionals with valuable tricks. I feel quite happy to have encountered your website page and look forward to many more cool moments reading here. Thanks a lot once more for everything.

  14. yeezys说道:

    Thanks so much for providing individuals with such a terrific chance to read from this website. It is usually so sweet and also packed with a lot of fun for me personally and my office acquaintances to visit the blog at the very least three times in a week to read the new guides you have got. And definitely, we are actually astounded with the beautiful creative ideas you serve. Certain 4 tips in this posting are basically the most efficient I have had.

  15. hyperdunks说道:

    I am glad for writing to let you be aware of what a fantastic discovery my friend's daughter undergone browsing yuor web blog. She noticed so many issues, which include what it is like to possess an awesome teaching spirit to have many others easily fully grasp specific very confusing things. You actually exceeded her expected results. Many thanks for producing these important, dependable, explanatory and as well as cool thoughts on the topic to Gloria.

  16. I and also my buddies appeared to be checking the nice recommendations from the blog and immediately I had a horrible feeling I had not expressed respect to the web blog owner for them. All the women came as a result stimulated to study all of them and have in effect absolutely been tapping into those things. Thanks for really being so kind and then for utilizing varieties of helpful tips most people are really eager to understand about. My very own honest apologies for not expressing appreciation to sooner.

  17. retro jordans说道:

    I really wanted to compose a word to express gratitude to you for these awesome tips and tricks you are posting at this website. My rather long internet look up has finally been honored with reputable content to go over with my friends and classmates. I would declare that many of us visitors are extremely fortunate to live in a notable community with many lovely professionals with valuable principles. I feel quite privileged to have seen the website and look forward to some more exciting minutes reading here. Thank you again for everything.

  18. nmd uk说道:

    I and also my guys have already been checking out the excellent tips located on the blog while unexpectedly I got a terrible feeling I had not thanked the site owner for those strategies. All of the women had been very interested to study all of them and have now certainly been making the most of these things. Many thanks for simply being considerably considerate and for finding varieties of brilliant resources most people are really eager to be informed on. My personal sincere apologies for not expressing gratitude to sooner.

  19. yeezy boost 350说道:

    I must express some thanks to the writer just for rescuing me from this particular dilemma. Right after looking throughout the world wide web and meeting tips that were not powerful, I believed my entire life was done. Existing minus the strategies to the difficulties you have resolved through your posting is a critical case, as well as ones which may have adversely affected my career if I hadn't noticed the blog. The mastery and kindness in taking care of everything was precious. I'm not sure what I would've done if I had not come across such a step like this. I'm able to at this moment relish my future. Thanks very much for this high quality and effective guide. I won't hesitate to propose your web site to any individual who should receive assistance on this matter.

  20. Thanks a lot for providing individuals with remarkably superb chance to read from here. It is often so kind and as well , full of fun for me and my office co-workers to visit your site on the least 3 times a week to study the fresh secrets you will have. Not to mention, I'm usually fulfilled with the dazzling information you serve. Selected 3 areas in this article are certainly the most beneficial we've had.

  21. I wish to point out my passion for your kindness for persons who absolutely need assistance with the subject matter. Your real dedication to passing the message along was incredibly informative and have really allowed associates like me to attain their ambitions. The useful guidelines can mean a lot to me and substantially more to my fellow workers. Many thanks; from all of us.

  22. bape hoodie说道:

    Thank you for each of your labor on this web site. My aunt take interest in going through internet research and it's really easy to understand why. I notice all relating to the powerful way you make insightful tips by means of your web site and as well encourage contribution from other people on that subject matter then my child is always becoming educated so much. Take pleasure in the remaining portion of the year. You're the one doing a tremendous job.

  23. I intended to send you that very small observation to finally thank you so much yet again considering the stunning tips you have shared above. It was really shockingly open-handed with you to supply unhampered just what a lot of people might have made available as an ebook in making some cash for themselves, most importantly given that you could possibly have done it in case you considered necessary. These ideas in addition acted as the fantastic way to recognize that the rest have the same dream just as my personal own to know way more in terms of this issue. I think there are millions of more enjoyable moments up front for many who look over your blog post.

  24. timberland boots说道:

    I wanted to create you this tiny remark to finally give many thanks over again for the unique solutions you have documented on this site. This is really surprisingly generous with people like you to provide easily what exactly a number of people would've offered for sale for an e book to make some money on their own, precisely now that you might well have done it in case you wanted. The inspiring ideas additionally acted to become easy way to be certain that other people online have the same desire just like my very own to find out good deal more around this problem. I am sure there are a lot more fun instances ahead for many who look over your blog post.

  25. jordan retro说道:

    I am just commenting to let you know of the extraordinary encounter my friend's girl obtained browsing your webblog. She picked up so many issues, not to mention what it is like to possess a great coaching heart to have most people quite simply learn about several complex issues. You actually surpassed her expected results. Thank you for coming up with such important, safe, edifying and easy guidance on that topic to Julie.

  26. nfl jerseys说道:

    I simply had to appreciate you all over again. I'm not certain what I might have gone through without the opinions discussed by you about my field. Completely was the fearsome issue for me, nevertheless encountering the professional avenue you processed the issue forced me to leap for gladness. I am happier for your assistance and in addition believe you know what a powerful job you're carrying out educating others all through your blog post. I am sure you haven't got to know any of us.

  27. ferragamo belt说道:

    I wanted to send a simple remark in order to express gratitude to you for all of the amazing tricks you are posting here. My particularly long internet look up has at the end of the day been honored with extremely good points to exchange with my colleagues. I would believe that many of us readers are very blessed to dwell in a fabulous site with so many special people with beneficial concepts. I feel quite grateful to have encountered your web pages and look forward to some more enjoyable minutes reading here. Thanks once again for a lot of things.

  28. cheap jordans说道:

    I must express my thanks to you just for rescuing me from such a matter. After exploring throughout the world-wide-web and coming across methods which were not beneficial, I was thinking my entire life was done. Being alive without the presence of answers to the issues you have fixed through your short article is a critical case, and the kind that might have negatively damaged my entire career if I hadn't come across the blog. Your personal understanding and kindness in controlling a lot of things was tremendous. I'm not sure what I would've done if I hadn't come upon such a stuff like this. I am able to now relish my future. Thanks a lot so much for this impressive and result oriented guide. I won't hesitate to recommend your web page to anybody who would need recommendations on this subject matter.

  29. vans outlet说道:

    I just wanted to jot down a brief comment to express gratitude to you for these magnificent guidelines you are giving out on this site. My time intensive internet research has now been honored with wonderful concept to exchange with my two friends. I 'd state that that most of us readers actually are really blessed to be in a really good site with many marvellous professionals with very beneficial suggestions. I feel somewhat privileged to have used the website and look forward to many more entertaining times reading here. Thanks a lot again for everything.

  30. air jordan 12说道:

    I am only commenting to let you understand of the magnificent experience my friend's girl went through reading through your web page. She realized such a lot of pieces, including how it is like to possess an awesome giving heart to have certain people very easily know precisely selected problematic things. You actually did more than visitors' expected results. Many thanks for rendering those priceless, healthy, explanatory and in addition cool guidance on the topic to Janet.

  31. nike cortez men说道:

    Thank you for all your efforts on this website. Gloria enjoys engaging in investigations and it's easy to see why. Almost all notice all concerning the powerful tactic you convey functional items via your web site and as well encourage participation from people on this point then our favorite princess is without question being taught a whole lot. Take advantage of the rest of the year. You're the one conducting a superb job.

  32. nike huarache说道:

    I am also commenting to let you know of the remarkable encounter my daughter undergone viewing your site. She learned some things, which include what it's like to possess an excellent teaching mindset to get many people completely comprehend a variety of grueling topics. You actually surpassed my expected results. Thanks for offering such interesting, safe, revealing and as well as fun tips on your topic to Lizeth.

  33. I together with my guys were found to be following the excellent secrets and techniques on the blog and then all of the sudden I had an awful suspicion I never expressed respect to the web blog owner for those secrets. These young boys happened to be totally excited to learn them and now have truly been having fun with those things. Many thanks for turning out to be very accommodating and then for deciding on this form of incredibly good subjects millions of individuals are really eager to discover. My very own honest apologies for not expressing gratitude to earlier.

  34. yeezy boost说道:

    I would like to show my affection for your generosity supporting those individuals that should have guidance on in this area of interest. Your real commitment to passing the solution throughout became exceptionally insightful and have usually permitted ladies just like me to reach their dreams. Your personal insightful useful information indicates so much to me and still more to my office colleagues. With thanks; from everyone of us.

  35. chrome hearts说道:

    I enjoy you because of all your hard work on this site. Gloria delights in participating in research and it's easy to understand why. Many of us notice all concerning the dynamic ways you give reliable guidelines via the web blog and even foster participation from other people about this point and my princess is undoubtedly becoming educated a whole lot. Enjoy the rest of the new year. You are carrying out a superb job.

  36. ultra boost说道:

    Thank you so much for providing individuals with an exceptionally special possiblity to read critical reviews from this blog. It's usually so ideal plus stuffed with amusement for me personally and my office acquaintances to search your web site particularly 3 times in 7 days to learn the newest secrets you have. And lastly, we are certainly pleased with your amazing knowledge you serve. Certain 4 tips in this posting are unquestionably the best I've had.

  37. adidas yeezy说道:

    Thank you for all your effort on this blog. Kate really likes setting aside time for research and it's simple to grasp why. I learn all about the dynamic ways you convey both interesting and useful steps on this web blog and improve response from the others on the point and our child is without question being taught a lot of things. Take advantage of the remaining portion of the year. You are conducting a powerful job.

  38. asics shoes说道:

    I would like to show my passion for your generosity for those people that must have assistance with in this subject. Your very own commitment to passing the message all over appears to be extremely powerful and has really allowed folks much like me to realize their aims. Your entire informative guideline implies a whole lot to me and somewhat more to my peers. Best wishes; from everyone of us.

  39. nike polo说道:

    I simply wanted to post a comment in order to thank you for all of the amazing secrets you are placing here. My time-consuming internet investigation has now been paid with good content to talk about with my contacts. I 'd point out that most of us readers are definitely endowed to exist in a good network with so many brilliant people with very helpful concepts. I feel very happy to have discovered your entire webpage and look forward to plenty of more cool minutes reading here. Thanks a lot once more for all the details.

  40. jordan 11 retro说道:

    Thank you a lot for providing individuals with an extraordinarily terrific possiblity to check tips from this blog. It really is very pleasurable and as well , stuffed with amusement for me and my office mates to search your website not less than three times in 7 days to see the fresh issues you will have. And indeed, I am also at all times fulfilled with the magnificent tactics you give. Selected 4 points in this article are undeniably the simplest I've had.

  41. michael kors说道:

    I precisely wished to say thanks all over again. I'm not certain what I would have followed without these methods provided by you regarding such question. It truly was an absolute intimidating condition in my view, nevertheless taking a look at this specialized style you processed the issue made me to leap over delight. Extremely happy for the work and even wish you recognize what a great job that you are getting into teaching people with the aid of your websites. I know that you've never met any of us.

发表评论

电子邮件地址不会被公开。 必填项已用*标注