awesome-web-scraper

Highly extensible, highly scalable Web crawler. Pluggable parsing, protocols, storage and indexing.

open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes.

A full set of search functions. Build your own indexing strategy. Parsers extract full-text data. The crawlers can index everything.

Website-Specific Processors for HTML INformation eXtraction.

Navigation scripting & testing utility for PhantomJS and SlimerJS.

A JavaScript implementation of the WHATWG DOM and HTML standards, for use with node.js

Crawl a website and run it through Google lighthouse.

Nightmare is a high level wrapper for PhantomJS that lets you automate browser tasks

Web Crawler/Spider for NodeJS + server-side jQuery.

Flexible event driven crawler for node.

Built in C# 3.5 version. it contains a simple extention of web content categorizer, which can saparate between the web page depending on their content.

A library for Rapid Web Crawler and Scraper Development.

Simple and fast HTML parser.

Goutte, a simple PHP Web Scraper.

PHPCrawl is a framework for crawling/spidering websites written in PHP.

Just a Simple HTML DOM library fork.

Opensource Web Crawler built on top of a nosql database (apache couchdb, riak), AMQP database (rabbitmq), webmachine and mochiweb.

machine learning based content & metadata extraction framework for Python

gdom, DOM Traversing and Scraping using GraphQL.

An open source library for making scraping with the use of the AI

Scrapy, a fast high-level web crawling & scraping framework for Python.

Library and command-line tool to extract metadata, main text, and comments.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Polite, slim and concurrent web crawler.

Highly extensible, highly scalable Web crawler. Pluggable parsing, protocols, storage and indexing.