Scrapy: open source crawling and web scraping framework

Scrapy is a fast high-level web open source crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Requires Python 2.7 or Python 3.4+.