2020-07-12 06:43:11 +00:00
# getwtxt
[![builds.sr.ht status ](https://builds.sr.ht/~gbmor/getwtxt.svg )](https://builds.sr.ht/~gbmor/getwtxt?)
[![Build Status ](https://travis-ci.com/getwtxt/getwtxt.svg?branch=master )](https://travis-ci.com/getwtxt/getwtxt)
[![Go Report Card ](https://goreportcard.com/badge/github.com/getwtxt/getwtxt )](https://goreportcard.com/report/github.com/getwtxt/getwtxt)
[![Code Climate Maintainability ](https://api.codeclimate.com/v1/badges/0e48bd9002de0f84b24e/maintainability )](https://codeclimate.com/github/getwtxt/getwtxt/maintainability)
2019-05-10 07:26:46 +00:00
2019-08-28 21:36:23 +00:00
twtxt registry written in Go!
2019-05-20 08:40:35 +00:00
2019-08-28 21:36:23 +00:00
[twtxt ](https://github.com/buckket/twtxt ) is a decentralized microblogging platform
2020-03-16 06:42:35 +00:00
for hackers based on text files. The user is "followed" and "mentioned" by referencing
2019-06-06 05:50:56 +00:00
the URL to their `twtxt.txt` file and a nickname.
2020-07-12 06:43:11 +00:00
2019-05-28 07:10:02 +00:00
Registries are designed to aggregate several users' statuses into a single location,
facilitating the discovery of new users to follow and allowing the search of statuses
for tags and key words.
2020-06-20 07:07:30 +00:00
< table style = "width: 100%; text-align: center; margin: 0 auto; border: 0px;" >
2020-06-20 06:51:54 +00:00
< tr >
< td > [< a href = "#installation" > Installation< / a > ]< / td >
< td > [< a href = "#upgrading" > Upgrading< / a > ]< / td >
< td > [< a href = "#configuration" > Configuration< / a > ]< / td >
< td > [< a href = "#using-the-registry" > Using the Registry< / a > ]< / td >
< td > [< a href = "#benchmarks" > Benchmarks< / a > ]< / td >
< td > [< a href = "#other-documentation" > Other Documentation< / a > ]< / td >
< td > [< a href = "#notes" > Notes< / a > ]< / td >
< / tr >
< / table >
2019-05-28 07:10:02 +00:00
2019-08-28 21:36:23 +00:00
## Features
2019-05-28 07:10:02 +00:00
2020-03-16 06:42:35 +00:00
* Easy to set up
2019-06-04 22:39:51 +00:00
* Uses an in-memory cache to serve requests
2019-06-08 07:23:40 +00:00
* Pushes to a database at a configurable interval for persistent storage
2019-08-28 21:36:23 +00:00
* `leveldb (default)`
2019-06-08 07:23:40 +00:00
* `sqlite3`
2020-03-16 06:42:35 +00:00
* Easily run behind `nginx` , `Caddy` or another HTTP server.
2019-05-20 08:40:35 +00:00
2019-08-28 19:16:06 +00:00
## Public Instances
2019-05-27 04:15:39 +00:00
* [twtxt.tilde.institute ](https://twtxt.tilde.institute )
2019-08-28 19:16:06 +00:00
* [twtxt.envs.net ](https://twtxt.envs.net/ )
2020-06-20 06:51:54 +00:00
Would you like your instance listed? Send a message to the [mailing list ](https://lists.sr.ht/~gbmor/getwtxt )!
2019-05-27 04:15:39 +00:00
2019-08-28 21:36:23 +00:00
## Installation
2019-05-28 07:10:02 +00:00
2020-06-20 06:51:54 +00:00
I have tested getwtxt on the following:
* `Debian 9, 10`
2019-06-06 05:50:56 +00:00
* `Ubuntu Server 18.04LTS, 18.10, 19.04`
2020-06-20 06:51:54 +00:00
* `OpenBSD 6.6`
2019-06-05 04:32:27 +00:00
Build dependencies are minimal, and only include:
2020-06-20 06:51:54 +00:00
* `make`
2019-06-05 04:32:27 +00:00
* `go >= 1.11`
2019-06-12 03:29:46 +00:00
* `git`
2019-06-05 04:32:27 +00:00
2019-06-12 09:27:18 +00:00
First, fetch the sources using `git` and jump into the directory.
2019-05-28 07:10:02 +00:00
```
2020-06-20 06:51:54 +00:00
$ git clone https://git.sr.ht/~gbmor/getwtxt
2019-05-28 07:10:02 +00:00
...
$ cd getwtxt
2019-06-12 09:27:18 +00:00
```
2019-08-28 21:36:23 +00:00
Then, check out the latest release tag.
2019-06-12 09:27:18 +00:00
```
2019-06-11 23:12:14 +00:00
$ git checkout $(git describe --tags --abbrev=0)
2019-06-06 05:50:56 +00:00
```
2019-05-28 07:10:02 +00:00
2019-08-28 21:36:23 +00:00
Use `make` to initiate the build and install process.
2019-05-28 07:10:02 +00:00
```
2019-06-05 04:12:33 +00:00
$ make
...
$ sudo make install
2019-05-28 07:10:02 +00:00
```
2019-08-28 20:46:19 +00:00
## Upgrading
2020-07-12 06:43:11 +00:00
Upgrading is nearly a identical process. Pull the changes, check out the
latest tag, and rebuild.
2019-08-28 20:46:19 +00:00
2020-07-12 06:43:11 +00:00
systemd might yell at you about running `systemctl daemon-reload` when you
go to restart getwtxt.
2019-08-28 20:46:19 +00:00
2020-07-12 06:43:11 +00:00
While getwtxt is pre-`1.0`, any patch-level updates (`v0.4.x`) will not
change configuration values. If a minor version increase has happened, for
example `v0.4.x -> v0.5.x` , then check if you need to update the config
file before restarting getwtxt.
2019-08-28 20:46:19 +00:00
2019-05-28 07:10:02 +00:00
## Configuration
2019-06-06 05:50:56 +00:00
\[ [Proxying ](#proxying ) \] \[ [Starting getwtxt ](#starting-getwtxt ) \]
2019-05-28 07:10:02 +00:00
2019-08-28 21:36:23 +00:00
To configure getwtxt, you'll first need to open `/usr/local/getwtxt/getwtxt.yml`
in your favorite editor and modify any values necessary. There are comments in the
2019-06-05 04:12:33 +00:00
file explaining each option.
2019-05-28 07:10:02 +00:00
2019-08-28 21:36:23 +00:00
If you desire, you may additionally modify the template in
`/usr/local/getwtxt/assets/tmpl/index.html` to customize the page users will see
when they pull up your registry instance in a web browser. The values in the
configuration file under `Instance:` are used to replace text `{{.Like This}}` in
2019-06-05 04:12:33 +00:00
the template.
2019-05-28 07:10:02 +00:00
### Proxying
2019-06-06 05:50:56 +00:00
Though getwtxt will run perfectly fine facing the internet directly, it does not
2020-07-12 06:43:11 +00:00
understand virtual hosts, nor does it use TLS. You'll probably want to proxy it
behind
2019-08-28 21:36:23 +00:00
`Caddy` or `nginx` for this reason.
2019-05-28 07:10:02 +00:00
2020-07-12 06:43:11 +00:00
`Caddy` is ludicrously easy to set up, and automatically handles `TLS`
certificates. Here's the config:
2019-05-28 07:10:02 +00:00
```caddyfile
2019-08-28 21:36:23 +00:00
twtxt.example.com
2019-05-28 07:10:02 +00:00
proxy / example.com:9001
```
2020-07-12 06:43:11 +00:00
If you're using `nginx` , here's a skeleton config to get you started. Don't
forget to change the 5 instances of `twtxt.example.com` to the (sub)domain
you'll be using to access the registry, generate SSL/TLS certificates using
LetsEncrypt, and change the port in `proxy_pass` to whichever port you
specified when modifying the configuration file. Currently, it's set to the
default port `9001`
2019-05-28 07:10:02 +00:00
```nginx
server {
server_name twtxt.example.com;
listen [::]:443 ssl http2;
listen 0.0.0.0:443 ssl http2;
ssl_certificate /etc/letsencrypt/live/twtxt.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/twtxt.example.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
2019-06-05 05:27:09 +00:00
proxy_pass http://127.0.0.1:9001;
2019-05-28 07:10:02 +00:00
}
}
server {
if ($host = twtxt.example.com) {
return 301 https://$host$request_uri;
}
listen 80;
server_name twtxt.example.com;
return 404;
}
```
2019-06-06 05:50:56 +00:00
### Starting getwtxt
2019-05-28 07:10:02 +00:00
2019-06-06 05:50:56 +00:00
Once you have everything configured to your needs, use `systemctl` to enable it
to run on system boot, then start the service.
2019-05-20 08:40:35 +00:00
2019-05-28 07:10:02 +00:00
```
2019-06-05 04:12:33 +00:00
$ sudo systemctl enable getwtxt
...
$ sudo systemctl start getwtxt
2019-05-28 07:10:02 +00:00
```
2019-05-20 08:40:35 +00:00
2019-05-28 07:10:02 +00:00
## Using the Registry
2019-05-22 04:37:22 +00:00
2020-07-12 06:43:11 +00:00
The following examples will all apply to using `curl` from a `Linux` , `BSD` , or
`macOS` terminal. All timestamps are in `RFC3339` format, per the twtxt registry
specification. Additionally, all queries support the `?page=N` parameter, where
`N` is a positive integer, that will retrieve page `N` of results in groups of
twenty.
2019-05-20 08:40:35 +00:00
2020-07-12 06:43:11 +00:00
The example API calls can also be found on the landing page of any getwtxt
instance, assuming the admin has not customized the landing page.
2019-06-04 07:20:17 +00:00
2019-05-28 07:10:02 +00:00
### Adding a User
Both nickname and URL are required
2020-06-20 06:51:54 +00:00
2019-05-28 07:10:02 +00:00
```
2019-05-28 07:18:51 +00:00
$ curl -X POST 'https://twtxt.example.com/api/plain/users?url=https://mysite.ext/twtxt.txt& nickname=FooJr'
2019-05-20 08:40:35 +00:00
2019-05-28 07:10:02 +00:00
200 OK
```
2019-06-04 07:20:17 +00:00
### Get All Tweets
2020-06-20 06:51:54 +00:00
2019-05-28 07:10:02 +00:00
```
$ curl 'https://twtxt.example.com/api/plain/tweets'
2019-05-28 07:18:51 +00:00
foo_barrington https://foo.bar.ext/twtxt.txt 2019-03-01T09:31:02.000Z Hey! It's my first status!
2019-05-28 07:10:02 +00:00
...
...
```
2019-06-04 07:20:17 +00:00
### Query Tweets by Keyword
2020-06-20 06:51:54 +00:00
2019-06-04 07:20:17 +00:00
```
$ curl 'https://twtxt.example.com/api/plain/tweets?q=getwtxt'
foo_barrington https://example3.com/twtxt.txt 2019-04-30T06:00:09.000Z I just installed getwtxt!
```
### Get All Users
2019-05-28 07:14:53 +00:00
Timestamp reflects when the user was added to the registry.
2019-05-28 07:10:02 +00:00
```
$ curl 'https://twtxt.example.com/api/plain/users'
foo_barrington https://foo.barrington.ext/twtxt.txt 2017-01-01T09:17:02.000Z
foo_barrington_jr https://example.com/twtxt.txt 2019-03-01T09:31:02.000Z
...
...
```
### Query Users
Can use either keyword or URL.
```
$ curl 'https://twtxt.example.com/api/plain/users?url=https://example.com/twtxt.txt'
foo https://example.com/twtxt.txt 2019-05-09T08:42:23.000Z
$ curl 'https://twtxt.example.com/api/plain/users?q=foo'
foo https://example.com/twtxt.txt 2019-05-09T08:42:23.000Z
foobar https://example2.com/twtxt.txt 2019-03-14T19:23:00.000Z
foo_barrington https://example3.com/twtxt.txt 2019-05-01T15:59:39.000Z
```
2019-06-04 07:20:17 +00:00
### Get all tweets with mentions
Mentions are placed within a status using the format `@<nickname http://url/twtxt.txt>`
2020-06-20 06:51:54 +00:00
2019-05-28 07:10:02 +00:00
```
2019-06-04 07:20:17 +00:00
$ curl 'https://twtxt.tilde.institute/api/plain/mentions'
foo https://example.com/twtxt.txt 2019-02-28T11:06:44.000Z @< foo_barrington https: / / example3 . com / twtxt . txt > Hey!! Are you still working on that project?
bar https://mxmmplm.com/twtxt.txt 2019-02-27T11:06:44.000Z @< foobar https: / / example2 . com / twtxt . txt > How's your day going, bud?
foo_barrington https://example3.com/twtxt.txt 2019-02-26T11:06:44.000Z @< foo https: / / example . com / twtxt . txt > Did you eat my lunch?
```
### Query tweets by mention URL
2020-06-20 06:51:54 +00:00
2019-06-04 07:20:17 +00:00
```
$ curl 'https://twtxt.tilde.institute/api/plain/mentions?url=https://foobarrington.co.uk/twtxt.txt'
foo https://example.com/twtxt.txt 2019-02-26T11:06:44.000Z @< foo_barrington https: / / foobarrington . co . uk / twtxt . txt > Hey!! Are you still working on that project?e
```
### Get all Tags
2020-06-20 06:51:54 +00:00
2019-06-04 07:20:17 +00:00
```
$ curl 'https://twtxt.example.com/api/plain/tags'
2019-05-28 07:10:02 +00:00
2019-06-04 07:20:17 +00:00
foo https://example.com/twtxt.txt 2019-03-01T09:33:04.000Z No, seriously, I need #help
foo https://example.com/twtxt.txt 2019-03-01T09:32:12.000Z Seriously, I love #programming !
2019-05-28 07:10:02 +00:00
foo https://example.com/twtxt.txt 2019-03-01T09:31:02.000Z I love #programming !
```
2019-06-04 07:20:17 +00:00
### Query by Tag
2020-06-20 06:51:54 +00:00
2019-05-28 07:10:02 +00:00
```
2019-06-04 07:20:17 +00:00
$ curl 'https://twtxt.example.com/api/plain/tags/programming'
2019-05-28 07:14:53 +00:00
2019-06-04 07:20:17 +00:00
foo https://example.com/twtxt.txt 2019-03-01T09:31:02.000Z I love #programming !
2019-05-28 07:10:02 +00:00
```
2021-10-22 01:31:17 +00:00
### Delete a User
```
$ curl -X DELETE -H 'X-Auth: password_in_getwtxt.yml' 'https://twtxt.example.com/api/admin/users?url=https://example.com/twtxt.txt'
200 OK
```
2019-05-28 07:10:02 +00:00
## Benchmarks
* [bombardier ](https://github.com/codesenberg/bombardier )
```
$ bombardier -c 100 -n 200000 http://localhost:9001/api/plain/tweets
Bombarding http://localhost:9001/api/plain/tweets with 200000 request(s) using 100 connection(s)
2019-06-10 06:53:00 +00:00
200000 / 200000 [=============================================================] 100.00% 19961/s 10s
2019-05-28 07:10:02 +00:00
Done!
Statistics Avg Stdev Max
2019-06-10 06:53:00 +00:00
Reqs/sec 20006.58 2408.55 26054.73
Latency 5.00ms 3.58ms 62.99ms
2019-05-28 07:10:02 +00:00
HTTP codes:
1xx - 0, 2xx - 200000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
2019-08-28 21:36:23 +00:00
Throughput: 39.27MB/s
2019-05-28 07:10:02 +00:00
```
2019-06-04 07:20:17 +00:00
## Other Documentation
2020-07-12 06:43:11 +00:00
In addition to what is provided here, additional information, particularly
regarding the configuration file, may be found by running getwtxt with the `-m`
or `--manual` flags. You will likely want to pipe the output to `less` as it is
quite long.
2019-06-04 07:20:17 +00:00
```
$ ./getwtxt -m | less
$ ./getwtxt --manual | less
```
2019-06-06 05:50:56 +00:00
If you need to remove getwtxt from your system, navigate to the source directory
2019-06-05 04:12:33 +00:00
you acquired using `git` during the installation process and run the appropriate
`make` hook:
```
$ sudo make uninstall
```
2019-06-04 07:20:17 +00:00
## Notes
2019-05-28 07:10:02 +00:00
2019-06-10 06:27:32 +00:00
twtxt Information: [`twtxt.readthedocs.io` ](https://twtxt.readthedocs.io )
2019-05-28 07:10:02 +00:00
2019-12-20 02:58:37 +00:00
Interested in twtxt but don't have your own server? [`github.com/LuRsT/twtxt_on_heroku` ](https://github.com/LuRsT/twtxt_on_heroku )
2019-08-27 19:44:58 +00:00
2019-06-10 06:27:32 +00:00
twtxt Client Repo: [`github.com/buckket/twtxt` ](https://github.com/buckket/twtxt )
2019-06-04 07:24:22 +00:00
2019-06-10 06:27:32 +00:00
Registry Specification: [`twtxt.readthedocs.io/en/latest/user/registry.html` ](https://twtxt.readthedocs.io/en/latest/user/registry.html )
Special thanks to [`github.com/kognise/water.css` ](https://github.com/kognise/water.css ) for open-sourcing a pleasant, easy-to-use, importable stylesheet
2019-06-12 09:04:05 +00:00
2020-07-12 06:43:11 +00:00
## Contributing
2019-06-12 09:04:05 +00:00
2020-06-20 06:51:54 +00:00
All contributions are greatly appreciated!
2019-06-12 09:04:05 +00:00
2020-06-20 07:10:56 +00:00
* Mailing list for patches, discussion, etc:
2020-06-20 06:51:54 +00:00
* [lists.sr.ht/~gbmor/getwtxt ](https://lists.sr.ht/~gbmor/getwtxt )
* Ticket tracker:
* [todo.sr.ht/~gbmor/getwtxt ](https://todo.sr.ht/~gbmor/getwtxt )