sep-011.rst

SEP	11
Title	Process models for Scrapy
Created	2009-11-16
Status	Partially implemented - see #168

SEP-011: Process models for Scrapy

There is an interest of supporting different process models for Scrapy, mainly to help prevent memory leaks which affect running all spiders in the same process.

By running each spider on a separate process (or pool of processes) we'll be able to "recycle" process when they exceed a maximum amount of memory.

Supported process models

The user could choose between different process models:

1. in process (only method supported so far)
2. pooled processes (a predefined pool of N processes, which could run more than one spider each)
3. separate processes (one process per spider)

Using different processes would increase reliability at the cost of performance.

Another ideas to consider

• configuring pipeline process models - so that we can have a process exclusive for running pipelines
• support writing spidersr in different languages when we don't use an in process model