Crawler Frameworks

Frameworks providing deterministic crawling infrastructure — URL frontier, retry, rate limiting, session management.

Scoring method (open_source_ai_tools_v1): 40% GitHub activity (time-decay since last push) + 40% stars (log-normalized within category) + 20% license permissiveness, minus 5 points if archived. Last generated: 2026-09-13T07:03:13.193747+00:00