The fastest web crawler and indexer. Foundational building blocks for data curation workloads.
The simplest way to get started is to use the Spider Cloud for a pain free hosted service. View the spider or spider_cli directory for local installations.
See BENCHMARKS.
See EXAMPLES.
This project is licensed under the MIT license.
See CONTRIBUTING.