Factor/To do/Spider

retries
redirects
page timeout, data timeout, overall timeout, stopping spiders if overall timeout is reached
https
cookies
make filters compile somehow

This revision created on Thu, 2 Oct 2008 01:09:07 by erg

Contents