Welcome to Scrapy Do’s documentation!

Scrapy Do is a daemon that provides a convenient way to run Scrapy spiders. It can either do it once - immediately; or it can run them periodically, at specified time intervals. It’s been inspired by scrapyd but written from scratch. For the time being, it only comes with a REST API and a a command line client. Version 0.3.0 will have an interactive web interface.

• Homepage: https://jany.st/scrapy-do.html
• The code: https://github.com/ljanyst/scrapy-do

Contents

• Quick Start
• Basic Concepts
  • Projects
  • Jobs
  • Scheduling Specs
• Installation
  • The easy way
  • A systemd service
• Server Configuration
  • [scrapy-do] section
  • [web] section
  • Example configuration
• Command Line Client
  • Global parameters and the configuration file
  • Commands and their parameters
• REST API
  • status.json
  • push-project.json
  • list-projects.json
  • list-spiders.json
  • schedule-job.json
  • list-jobs.json
  • cancel-job.json
  • get-log
  • remove-project.json
• Source Documentation

Indices and tables

• Index
• Module Index
• Search Page