pip install scrapy-fake-useragent

DOWNLOADER_MIDDLEWARES = {
  'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
  'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
  'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
  'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
}
 

Logo

Agent 垂直技术社区,欢迎活跃、内容共建。

更多推荐