Factor/To do/Spider

retries
redirects
page timeout, data timeout, overall timeout, stopping spiders if overall timeout is reached
https
cookies
make filters compile somehow
parallel version

This revision created on Thu, 2 Oct 2008 01:09:36 by erg

Contents