piusbird
/
snarfbot
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
A hipster web crawler
5 Commits 1 Branch 0 Tags 32 KiB
Python 100%
Go to file
Matt Arnold f0caf13470 * Set a proper user agent, and add a safety valve so infinate recursions don't happen 2021-01-01 00:27:17 -05:00
snarfbot * Set a proper user agent, and add a safety valve so infinate recursions don't happen 2021-01-01 00:27:17 -05:00
.gitignore Refactor in preperation building the crawler 2020-12-30 21:46:12 -05:00
README.md Inital Commit 2020-12-29 00:06:37 -05:00
requirements.txt Inital Commit 2020-12-29 00:06:37 -05:00
web2text.py Refactor in preperation building the crawler 2020-12-30 21:46:12 -05:00

README.md

Snarfbot

This will eventually be a web crawler that saves websites in plaintext files. For now please enjoy a few cli tools, written as POC. Comments, compliments, complaints, and pull requests accepted.