Skip to content

Instantly share code, notes, and snippets.

@ccckmit

ccckmit/scrapy_crawl_quotes

Last active November 10, 2022 13:14
Show Gist options
  • Save ccckmit/f062ea7613f804050088042deed95786 to your computer and use it in GitHub Desktop.
Save ccckmit/f062ea7613f804050088042deed95786 to your computer and use it in GitHub Desktop.
Download ZIP
scrapy crawl quotes
Raw
scrapy_crawl_quotes
# scrapy crawl quotes
```
Hero3C@DESKTOP-O093POU MINGW64 /c/tim/prog10_4
$ scrapy crawl quotes
2022-11-10 21:04:58 [scrapy.utils.log] INFO: Scrapy 2.7.1 started (bot: prog10_4)
2022-11-10 21:04:58 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0,
libxml2 2.9.12, cssselect 1.2.0, parsel 1.7.0, w3lib 2.0.1, Twisted 22.10.0, Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.3, Platform Windows-10-10.0.19043-SP0
2022-11-10 21:04:58 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'prog10_4',
'NEWSPIDER_MODULE': 'prog10_4.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['prog10_4.spiders']}
2022-11-10 21:04:58 [py.warnings] WARNING: C:\Users\Hero3C\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\utils\request.py:231: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.
It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.
See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
return cls(crawler)
2022-11-10 21:04:58 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-11-10 21:04:58 [scrapy.extensions.telnet] INFO: Telnet Password: c026ecd4fa77d643
2022-11-10 21:04:58 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2022-11-10 21:04:58 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-11-10 21:04:58 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-11-10 21:04:59 [scrapy.middleware] INFO: Enabled item pipelines:['prog10_4.pipelines.Prog104Pipeline']
2022-11-10 21:04:59 [scrapy.core.engine] INFO: Spider opened
2022-11-10 21:04:59 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-11-10 21:04:59 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-11-10 21:05:00 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2022-11-10 21:05:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)
2022-11-10 21:05:01 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://quotes.toscrape.com/author/Albert-Einstein> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Jane-Austen/> from <GET http://quotes.toscrape.com/author/Jane-Austen>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Andre-Gide/> from <GET http://quotes.toscrape.com/author/Andre-Gide>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Eleanor-Roosevelt/> from <GET http://quotes.toscrape.com/author/Eleanor-Roosevelt>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Thomas-A-Edison/> from <GET http://quotes.toscrape.com/author/Thomas-A-Edison>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Marilyn-Monroe/> from <GET http://quotes.toscrape.com/author/Marilyn-Monroe>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Steve-Martin/> from <GET http://quotes.toscrape.com/author/Steve-Martin>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Albert-Einstein/> from <GET http://quotes.toscrape.com/author/Albert-Einstein>
2022-11-10 21:05:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/J-K-Rowling/> from <GET http://quotes.toscrape.com/author/J-K-Rowling>
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Andre-Gide/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Thomas-A-Edison/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Jane-Austen/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Eleanor-Roosevelt/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Marilyn-Monroe/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/J-K-Rowling/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Steve-Martin/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:05:32 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Andre-Gide/>
{'author': 'André Gide',
'birthday': 'November 22, 1869',
'bornplace': 'in Paris, France',
'quote': '“It is better to be hated for what you are than to be loved for '
'what you are not.”'}
2022-11-10 21:06:02 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Thomas-A-Edison/>
{'author': 'Thomas A. Edison',
'birthday': 'February 11, 1847',
'bornplace': 'in Milan, Ohio, The United States',
'quote': "“I have not failed. I've just found 10,000 ways that won't work.”"}
2022-11-10 21:06:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Albert-Einstein/> (referer: http://quotes.toscrape.com/)
2022-11-10 21:06:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Jane-Austen/>
{'author': 'Jane Austen',
'birthday': 'December 16, 1775',
'bornplace': 'in Steventon Rectory, Hampshire, The United Kingdom',
'quote': '“The person, be it gentleman or lady, who has not pleasure in a '
'good novel, must be intolerably stupid.”'}
2022-11-10 21:06:33 [scrapy.extensions.logstats] INFO: Crawled 11 pages (at 11 pages/min), scraped 3 items (at 3 items/min)
2022-11-10 21:07:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Eleanor-Roosevelt/>
{'author': 'Eleanor Roosevelt',
'birthday': 'October 11, 1884',
'bornplace': 'in The United States',
'quote': '“A woman is like a tea bag; you never know how strong it is until '
"it's in hot water.”"}
2022-11-10 21:07:33 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Marilyn-Monroe/>
{'author': 'Marilyn Monroe',
'birthday': 'June 01, 1926',
'bornplace': 'in The United States',
'quote': "“Imperfection is beauty, madness is genius and it's better to be "
'absolutely ridiculous than absolutely boring.”'}
2022-11-10 21:08:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/J-K-Rowling/>
{'author': 'J.K. Rowling',
'birthday': 'July 31, 1965',
'bornplace': 'in Yate, South Gloucestershire, England, The United Kingdom',
'quote': '“It is our choices, Harry, that show what we truly are, far more '
'than our abilities.”'}
2022-11-10 21:08:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Steve-Martin/>
{'author': 'Steve Martin',
'birthday': 'August 14, 1945',
'bornplace': 'in Waco, Texas, The United States',
'quote': '“A day without sunshine is like, you know, night.”'}
2022-11-10 21:08:34 [scrapy.extensions.logstats] INFO: Crawled 11 pages (at 0 pages/min), scraped 7 items (at 4 items/min)
2022-11-10 21:09:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Albert-Einstein/>
{'author': 'Albert Einstein',
'birthday': 'March 14, 1879',
'bornplace': 'in Ulm, Germany',
'quote': '“The world as we have created it is a process of our thinking. It '
'cannot be changed without changing our thinking.”'}
2022-11-10 21:09:04 [scrapy.extensions.logstats] INFO: Crawled 11 pages (at 0 pages/min), scraped 8 items (at 1 items/min)
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://quotes.toscrape.com/author/Douglas-Adams> (failed 1 times): User timeout caused connection failure: Getting http://quotes.toscrape.com/author/Douglas-Adams took longer than 180.0 seconds..
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://quotes.toscrape.com/author/Dr-Seuss> (failed 1 times): User timeout caused connection failure: Getting http://quotes.toscrape.com/author/Dr-Seuss took longer than 180.0 seconds..
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://quotes.toscrape.com/author/Bob-Marley> (failed 1 times): User timeout caused connection failure: Getting http://quotes.toscrape.com/author/Bob-Marley took longer than 180.0 seconds..
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Allen-Saunders/> from <GET http://quotes.toscrape.com/author/Allen-Saunders>
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Friedrich-Nietzsche/> from <GET http://quotes.toscrape.com/author/Friedrich-Nietzsche>
2022-11-10 21:09:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Mark-Twain/> from <GET http://quotes.toscrape.com/author/Mark-Twain>
2022-11-10 21:09:04 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Elie-Wiesel/> from <GET http://quotes.toscrape.com/author/Elie-Wiesel>
2022-11-10 21:09:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Friedrich-Nietzsche/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Allen-Saunders/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:05 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Dr-Seuss/>
from <GET http://quotes.toscrape.com/author/Dr-Seuss>
2022-11-10 21:09:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Mark-Twain/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:05 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Bob-Marley/> from <GET http://quotes.toscrape.com/author/Bob-Marley>
2022-11-10 21:09:05 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (308) to <GET http://quotes.toscrape.com/author/Douglas-Adams/> from <GET http://quotes.toscrape.com/author/Douglas-Adams>
2022-11-10 21:09:35 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Friedrich-Nietzsche/>
{'author': 'Friedrich Nietzsche',
'birthday': 'October 15, 1844',
'bornplace': 'in Röcken bei Lützen, Prussian Province of Saxony, Germany',
'quote': '“It is not a lack of love, but a lack of friendship that makes '
'unhappy marriages.”'}
2022-11-10 21:09:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Dr-Seuss/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Bob-Marley/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Elie-Wiesel/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:09:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/author/Douglas-Adams/> (referer: http://quotes.toscrape.com/page/2/)
2022-11-10 21:10:05 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Allen-Saunders/>
{'author': 'Allen Saunders',
'birthday': 'April 24, 1899',
'bornplace': 'in The United States',
'quote': '“Life is what happens to us while we are making other plans.”'}
2022-11-10 21:10:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Mark-Twain/>
{'author': 'Mark Twain',
'birthday': 'November 30, 1835',
'bornplace': 'in Florida, Missouri, The United States',
'quote': '“Good friends, good books, and a sleepy conscience: this is the '
'ideal life.”'}
2022-11-10 21:10:36 [scrapy.extensions.logstats] INFO: Crawled 19 pages (at 8 pages/min), scraped 11 items (at 3 items/min)
2022-11-10 21:11:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Dr-Seuss/>
{'author': 'Dr. Seuss',
'birthday': 'March 02, 1904',
'bornplace': 'in Springfield, MA, The United States',
'quote': '“I like nonsense, it wakes up the brain cells. Fantasy is
a '
'necessary ingredient in living.”'}
2022-11-10 21:11:36 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Bob-Marley/>
{'author': 'Bob Marley',
'birthday': 'February 06, 1945',
'bornplace': 'in Nine Mile, Saint Ann, Jamaica',
'quote': '“You may not be her first, her last, or her only. She loved before '
'she may love again. But if she loves you now, what else matters? '
"She's not perfect—you aren't either, and the two of you may never "
'be perfect together but if she can make you laugh, cause you to '
'think twice, and admit to being human and making mistakes, hold '
'onto her and give her the most you can. She may not be thinking '
'about you every second of the day, but she will give you a part of '
"her that she knows you can break—her heart. So don't hurt
her, "
"don't change her, don't analyze and don't expect more than she can "
'give. Smile when she makes you happy, let her know when she makes '
"you mad, and miss her when she's not there.”"}
2022-11-10 21:12:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Elie-Wiesel/>
{'author': 'Elie Wiesel',
'birthday': 'September 30, 1928',
'bornplace': 'in Sighet, Romania',
'quote': "“The opposite of love is not hate, it's indifference. The
opposite "
"of art is not ugliness, it's indifference. The opposite of faith is "
"not heresy, it's indifference. And the opposite of life is not "
"death, it's indifference.”"}
2022-11-10 21:12:37 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/author/Douglas-Adams/>
{'author': 'Douglas Adams',
'birthday': 'March 11, 1952',
'bornplace': 'in Cambridge, England, The United Kingdom',
'quote': '“I may not have gone where I intended to go, but I think I have '
'ended up where I needed to be.”'}
2022-11-10 21:12:37 [scrapy.extensions.logstats] INFO: Crawled 19 pages (at 0 pages/min), scraped 15 items (at 4 items/min)
2022-11-10 21:12:37 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-10 21:12:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 3,
'downloader/request_bytes': 10240,
'downloader/request_count': 37,
'downloader/request_method_count/GET': 37,
'downloader/response_bytes': 93932,
'downloader/response_count': 34,
'downloader/response_status_count/200': 18,
'downloader/response_status_count/308': 15,
'downloader/response_status_count/404': 1,
'dupefilter/filtered': 16,
'elapsed_time_seconds': 458.005964,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 11, 10, 13, 12, 37, 535270),
'item_scraped_count': 15,
'log_count/DEBUG': 54,
'log_count/INFO': 15,
'log_count/WARNING': 1,
'request_depth_max': 3,
'response_received_count': 19,
'retry/count': 3,
'retry/reason_count/twisted.internet.error.TimeoutError': 3,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/404': 1,
'scheduler/dequeued': 36,
'scheduler/dequeued/memory': 36,
'scheduler/enqueued': 36,
'scheduler/enqueued/memory': 36,
'start_time': datetime.datetime(2022, 11, 10, 13, 4, 59, 529306)}
2022-11-10 21:12:37 [scrapy.core.engine] INFO: Spider closed (finished)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment