4 Commits

Author	SHA1	Message	Date
James Mills	13e60b7e5d	Add support for robots.txt user agent control from web crawlers	2018-03-24 17:15:26 -07:00
James Mills	74783dccd7	Minor refactor/cleanup	2017-10-15 13:16:35 -07:00
James Mills	76f337705e	Set sensible defaults	2017-06-26 01:54:33 -07:00
James Mills	c51b6d8887	Migrated to a Library structure with gopherproxy cmd for reuse by gopherclient	2016-09-30 19:40:24 -07:00