Pigeon-Ruby/README.md

![](logo.png)

# Pigeon Ruby

This is a WIP [Pigeon Protocol](https://tildegit.org/PigeonProtocolConsortium/protocol_spec) client written in Ruby.

# Caveats

 * Probably does not work on Windows, but testers are welcome to try it out. Please let me know how it goes.
 * Blobs are not included in bundles yet.
 * Not published to RubyGems yet (see installation instructions below)
 * Not thread safe, never will be. This implementation is intended for a single user and makes many design tradeoffs around that use case.
 * Bundle works, but is inefficient. Will optimize after proof of concept.

# Installation

We are not yet on Rubygems. The gem will be released after we are fully compliant with the spec and have high test coverage stats.

In the meantime:

```
git clone https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby.git
cd pigeon_ruby
gem build pigeon.gemspec
gem install pigeon-0.0.7.gem
pigeon-cli identity new # Should work. Raise issue if not.
pigeon-cli status
pigeon-cli help
```

# Usage: CLI

See `pigeon-cli help` for documentation.
See `kitchen_sink.sh` examples.

# Usage: Ruby Lib

TODO

# Current Status

 - [X] pigeon identity new
 - [X] pigeon identity show
 - [X] pigeon status
 - [X] pigeon blob set
 - [X] pigeon blob get
 - [X] pigeon peer add
 - [X] pigeon peer remove
 - [X] pigeon peer block
 - [X] pigeon peer all
 - [X] 100% coverage
 - [X] Convert `".sig.ed25519"` literals to constants
 - [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."
 - [X] pigeon draft create
 - [X] pigeon draft append
 - [X] pigeon draft current
 - [X] pigeon draft save
 - [X] pigeon bundle create
 - [X] Use JSON.stringify() for string keys (instead of `inspect`)
 - [X] Move literals into `Pigeon` module as constants, again.
 - [X] pigeon message find
 - [X] pigeon message find-all for local feed.
 - [X] pigeon bundle consume (We are minimally feature complete at this point)
 - [X] Fix the diagram in the spec document
 - [X] Validate inputs for `Draft#[]=`.
 - [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
 - [X] Use raw SHA256 hashes for blob multihashes, not hex.
 - [X] Change all the `{40,90}` values in ::Lexer to real length values
 - [X] Don't double-ingest messages. It will screw up indexes.
 - [X] 100% test coverage
 - [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
 - [X] Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
 - [X] Fix `scratchpad.sh` to use Base32
 - [X] Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
 - [X] Fix diagram in spec doc
 - [X] refactor `Bundle.create` to use `message find-all`.
 - [X] Rename `message find` to `message read`, since other finders return a multihash.
 - [X] Message.ingest should be the only code path to message authoring.
 - [X] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
 - [X] Run Flog / Flay and friends to find duplications. Will aid in port to other languages.
 - [X] Make all methods private except those required for the CLI.
 - [X] Add Lipmaa links like the Bamboo folks do.
 - [X] Set a max message size.
 - [X] Clean up all singletons / .current hacks
 - [X] Reduce cross cutting where collaborating objects need access to `@db`
 - [X] Ensure all disks writes perform verification!
 - [X] Make CLI names consistent with API names. Eg: find vs. read.
 - [X] `find-all` should....find all. Currently finds your messages or maybe peers, but not all.
 - [X] Add log count to `pigeon-cli status`
 - [X] Delete `Draft#put` entirely.
 - [X] Check block list before ingesting bundles.
 - [X] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
 - [X] Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
 - [X] Rename `who_am_i` as `get_who_am_i` to follow VERB + NOUN convention.
 - [ ] Find that non-deterministic runaway loop in the test suite.
 - [ ] Update README.md / tutorial.rb (user manual for `Pigeon::Database`).
 - [ ] Update spec document CLI usage examples to reflect API changes in 2020.
 - [ ] Publish to RubyGems

# Optimizations
 - [ ] add parsers and validators for all CLI inputs
 - [ ] Make the switch to LevelDB, RocksDB, [UNQLite](https://unqlite.org/features.html) or similar (currently using Ruby PStore).
 - [ ] Convert literals to constants, remove unused locals, reduce duplication, run linter.
 - [ ] Reduce whole darn repo into single module to aide portability. `::Helpers` module is OK.
 - [ ] Update the bundles.md document once `bundle consume` works.
 - [ ] 100% documentation
 - [ ] Performance benchmarks (Do this second to last!)
 - [ ] Performance tuning (Do this last!)

# After v0.0.1

 - [ ] (later, not now) Support partial verification via `lipmaa` property.
 - [ ] Add mandatory `--since=` arg to `bundle create
 - [ ] Interest and Disinterest Signalling for document routing: Create a `$blob_status` message to express `have`, `want` signalling. This can steer bundle creation and an eventual `--for` flag at bundle creation time to customize a bundle to a particular user.
 - [ ] Add a schema for `$peer_status`. Eg: `block`, `unblock`, `follow`, `unfollow`.

# Idea Bin

 - [ ] Map/reduce plugin support for custom indices?
 - [ ] Ability to add a blob in one swoop using File objects and `Message#[]=`, maybe?
 - [ ] Bundling via [Optar](http://ronja.twibright.com/optar/) or [Colorsafe](https://github.com/colorsafe/colorsafe)

# New Bundle Format

We have a bundle format that works, but it only exports messages.

We need a bundle format that may optionally include blobs as well.

Here's how we will support that:

1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
2. In the root directory of `bundle_x/`, a single `messages.pgn` file contains all messages.
  * All messages are expected to be sorted by depth
  * Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the `depth` field.
3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
    * `blobs/bundle/7Z2CSZK/MB1RE5G/6SKXRZ6/3ZGCNP8/VVEM3K0/XFMYKET/RDQSM5W.BSG`

Additional notes:

 * It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.

# Unanswered Questions

 * PEER MESSAGES: I want to add a `--depth` option to bundle exports that would only return messages after the `nth` sequence number. It would not make sense to apply `--depth` to all peer messages in the bundle. It would not be practical to expect the user to provide a `--depth` for every peer every time a bundle is generated.
   * Create a new `received_on` index that records the local user's `depth` at the time of ingestion?
Add logo 2019-10-10 00:32:54 +00:00			`![](logo.png)`

Coo... coo... 2019-09-22 02:30:03 +00:00			`# Pigeon Ruby`

Store blobs in filesystem, not database 2020-04-20 13:37:08 +00:00			`This is a WIP [Pigeon Protocol](https://tildegit.org/PigeonProtocolConsortium/protocol_spec) client written in Ruby.`
Coo... coo... 2019-09-22 02:30:03 +00:00
Remove wonky draft API 2020-04-19 21:27:09 +00:00			`# Caveats`

Store blobs in filesystem, not database 2020-04-20 13:37:08 +00:00			`* Probably does not work on Windows, but testers are welcome to try it out. Please let me know how it goes.`
Storage layer obeys blocklist 2020-04-20 02:32:37 +00:00			`* Blobs are not included in bundles yet.`
Remove wonky draft API 2020-04-19 21:27:09 +00:00			`* Not published to RubyGems yet (see installation instructions below)`
Store blobs in filesystem, not database 2020-04-20 13:37:08 +00:00			`* Not thread safe, never will be. This implementation is intended for a single user and makes many design tradeoffs around that use case.`
Storage layer obeys blocklist 2020-04-20 02:32:37 +00:00			`* Bundle works, but is inefficient. Will optimize after proof of concept.`
Remove wonky draft API 2020-04-19 21:27:09 +00:00
:heavy_check_mark: Update CLI 2020-04-18 15:51:08 +00:00			`# Installation`
Coo... coo... 2019-09-22 02:30:03 +00:00
v0.0.4 doc updates 2020-04-18 18:18:00 +00:00			`We are not yet on Rubygems. The gem will be released after we are fully compliant with the spec and have high test coverage stats.`
Coo... coo... 2019-09-22 02:30:03 +00:00
Dead code removal 2020-04-18 16:05:19 +00:00			`In the meantime:`

			```
			`git clone https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby.git`
			`cd pigeon_ruby`
v0.0.4 doc updates 2020-04-18 18:18:00 +00:00			`gem build pigeon.gemspec`
v0.0.7 2020-04-19 22:37:29 +00:00			`gem install pigeon-0.0.7.gem`
v0.0.4 doc updates 2020-04-18 18:18:00 +00:00			`pigeon-cli identity new # Should work. Raise issue if not.`
			`pigeon-cli status`
			`pigeon-cli help`
Dead code removal 2020-04-18 16:05:19 +00:00			```

:heavy_check_mark: Update CLI 2020-04-18 15:51:08 +00:00			`# Usage: CLI`

v0.0.4 doc updates 2020-04-18 18:18:00 +00:00			See `pigeon-cli help` for documentation.
			See `kitchen_sink.sh` examples.
Coo... coo... 2019-09-22 02:30:03 +00:00
:heavy_check_mark: Update CLI 2020-04-18 15:51:08 +00:00			`# Usage: Ruby Lib`

Dead code removal 2020-04-18 16:05:19 +00:00			`TODO`

Coo... coo... 2019-09-22 02:30:03 +00:00			`# Current Status`

			`- [X] pigeon identity new`
			`- [X] pigeon identity show`
Cleanup and test scripts. TODO: Why does blocking not delete previous peers? 2019-09-25 01:19:25 +00:00			`- [X] pigeon status`
Update README 2019-09-22 02:31:01 +00:00			`- [X] pigeon blob set`
`blob get` 2019-09-22 11:00:19 +00:00			`- [X] pigeon blob get`
Finish most `peer` commands. NEXT: peer all 2019-09-23 01:07:14 +00:00			`- [X] pigeon peer add`
			`- [X] pigeon peer remove`
			`- [X] pigeon peer block`
Fix bug in block/remove 2019-09-25 01:31:32 +00:00			`- [X] pigeon peer all`
:tada: 100% coverage. NEXT: Verify message chain creation via tests. Currently, it is not working correctly. Take a look at scratchpad.sh. The `prev:` values are wrong. It is probably as simple as updating local indexes when Pigeon::Storage#save_message is called. Re run the indexes at that time. Or heck, add a map/reduce ability? 2020-03-07 15:38:34 +00:00			`- [X] 100% coverage`
Convert `".sig.ed25519"` literals to constants. Update README. 2020-03-14 03:02:45 +00:00			- [X] Convert `".sig.ed25519"` literals to constants
Convert "message..." commands to "draft..." commands 2020-03-14 03:17:56 +00:00			`- [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."`
TODO: pigeon bundle create 2020-03-14 03:41:31 +00:00			`- [X] pigeon draft create`
			`- [X] pigeon draft append`
			`- [X] pigeon draft current`
			`- [X] pigeon draft save`
We have BUNDLE CREATION :tada: :tada: :tada: 2020-03-15 16:51:13 +00:00			`- [X] pigeon bundle create`
Constantize literals for portability and DRYness 2020-03-15 17:50:57 +00:00			- [X] Use JSON.stringify() for string keys (instead of `inspect`)
			- [X] Move literals into `Pigeon` module as constants, again.
Add `message find` and `message find-all` (local user only - find-all for peers :soon:) 2020-03-15 18:50:00 +00:00			`- [X] pigeon message find`
			`- [X] pigeon message find-all for local feed.`
Test case: bundle ingestion 2020-03-31 12:30:44 +00:00			`- [X] pigeon bundle consume (We are minimally feature complete at this point)`
			`- [X] Fix the diagram in the spec document`
Add test for Storage#find_all 2020-04-02 12:32:21 +00:00			- [X] Validate inputs for `Draft#[]=`.
			`- [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers`
Add correct length numbers to Lexer regexes for multihashes 2020-04-02 12:58:57 +00:00			`- [X] Use raw SHA256 hashes for blob multihashes, not hex.`
			- [X] Change all the `{40,90}` values in ::Lexer to real length values
Works again! :tada: 2020-04-03 10:19:42 +00:00			`- [X] Don't double-ingest messages. It will screw up indexes.`
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [X] 100% test coverage`
			- [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
[STABLE] Convert URLSafeb64 to Crockford32. 2020-04-06 01:01:59 +00:00			`- [X] Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.`
Fix kitchen_sink.sh. Add tests for base32 encoder 2020-04-06 12:18:04 +00:00			- [X] Fix `scratchpad.sh` to use Base32
Rename .public_key attributes to .multihash for consistency 2020-04-06 12:22:24 +00:00			`- [X] Rename (RemoteIdentity\|LocalIdentity)#public_key to #multihash for consistency with other types.`
Use `find_all` when creating bundles 2020-04-06 12:32:19 +00:00			`- [X] Fix diagram in spec doc`
			- [X] refactor `Bundle.create` to use `message find-all`.
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [X] Rename `message find` to `message read`, since other finders return a multihash.
:heavy_check_mark: Complete TODO items 2020-04-09 13:14:50 +00:00			`- [X] Message.ingest should be the only code path to message authoring.`
:heavy_check_mark: Complete TODO items 2020-04-09 13:15:00 +00:00			- [X] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
Make most of Pigeon::Lexer methods and attrs private 2020-04-11 14:32:41 +00:00			`- [X] Run Flog / Flay and friends to find duplications. Will aid in port to other languages.`
Add Pigeon::Helpers.lipmaa in preparation for lipmaa backlinking 2020-04-11 14:46:40 +00:00			`- [X] Make all methods private except those required for the CLI.`
Done with `lipmaa` addtions 2020-04-14 13:21:51 +00:00			`- [X] Add Lipmaa links like the Bamboo folks do.`
Add message size limit of 64 keys. 2020-04-15 12:30:05 +00:00			`- [X] Set a max message size.`
:heavy_check_mark: Update CLI 2020-04-18 15:51:08 +00:00			`- [X] Clean up all singletons / .current hacks`
			- [X] Reduce cross cutting where collaborating objects need access to `@db`
v0.0.5 2020-04-18 19:51:10 +00:00			`- [X] Ensure all disks writes perform verification!`
v0.0.6- Fix `find-all` to actually find-all. Improve `status` output 2020-04-19 15:16:06 +00:00			`- [X] Make CLI names consistent with API names. Eg: find vs. read.`
			- [X] `find-all` should....find all. Currently finds your messages or maybe peers, but not all.
			- [X] Add log count to `pigeon-cli status`
Storage layer obeys blocklist 2020-04-20 02:32:37 +00:00			- [X] Delete `Draft#put` entirely.
			`- [X] Check block list before ingesting bundles.`
Add blobs + peer messages to bundle 2020-04-22 13:21:55 +00:00			`- [X] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)`
			`- [X] Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)`
Run rubocop 2020-04-25 15:11:25 +00:00			- [X] Rename `who_am_i` as `get_who_am_i` to follow VERB + NOUN convention.
			`- [ ] Find that non-deterministic runaway loop in the test suite.`
v0.0.6- Fix `find-all` to actually find-all. Improve `status` output 2020-04-19 15:16:06 +00:00			- [ ] Update README.md / tutorial.rb (user manual for `Pigeon::Database`).
Add blobs + peer messages to bundle 2020-04-22 13:21:55 +00:00			`- [ ] Update spec document CLI usage examples to reflect API changes in 2020.`
			`- [ ] Publish to RubyGems`

			`# Optimizations`
Formatting updates 2020-03-15 18:12:26 +00:00			`- [ ] add parsers and validators for all CLI inputs`
Add blobs + peer messages to bundle 2020-04-22 13:21:55 +00:00			`- [ ] Make the switch to LevelDB, RocksDB, [UNQLite](https://unqlite.org/features.html) or similar (currently using Ruby PStore).`
:tada: v0.1.0 - minimally feature complete 2020-04-22 13:44:12 +00:00			`- [ ] Convert literals to constants, remove unused locals, reduce duplication, run linter.`
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [ ] Reduce whole darn repo into single module to aide portability. `::Helpers` module is OK.
			- [ ] Update the bundles.md document once `bundle consume` works.
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [ ] 100% documentation`
			`- [ ] Performance benchmarks (Do this second to last!)`
			`- [ ] Performance tuning (Do this last!)`
Updates to README 2020-04-08 13:54:28 +00:00
README updates 2020-04-15 13:09:49 +00:00			`# After v0.0.1`
Updates to README 2020-04-08 13:54:28 +00:00
README updates 2020-04-15 13:09:49 +00:00			- [ ] (later, not now) Support partial verification via `lipmaa` property.
			- [ ] Add mandatory `--since=` arg to `bundle create
Storage layer obeys blocklist 2020-04-20 02:32:37 +00:00			- [ ] Interest and Disinterest Signalling for document routing: Create a `$blob_status` message to express `have`, `want` signalling. This can steer bundle creation and an eventual `--for` flag at bundle creation time to customize a bundle to a particular user.
			- [ ] Add a schema for `$peer_status`. Eg: `block`, `unblock`, `follow`, `unfollow`.
DRY up storage initialization / reset code 2020-03-08 16:18:05 +00:00
			`# Idea Bin`
:heavy_check_mark: Update CLI 2020-04-18 15:51:08 +00:00
DRY up storage initialization / reset code 2020-03-08 16:18:05 +00:00			`- [ ] Map/reduce plugin support for custom indices?`
			- [ ] Ability to add a blob in one swoop using File objects and `Message#[]=`, maybe?
Store blobs in filesystem, not database 2020-04-20 13:37:08 +00:00			`- [ ] Bundling via [Optar](http://ronja.twibright.com/optar/) or [Colorsafe](https://github.com/colorsafe/colorsafe)`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`# New Bundle Format`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`We have a bundle format that works, but it only exports messages.`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`We need a bundle format that may optionally include blobs as well.`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`Here's how we will support that:`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
			1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
			2. In the root directory of `bundle_x/`, a single `messages.pgn` file contains all messages.
			`* All messages are expected to be sorted by depth`
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			* Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the `depth` field.
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00			`3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:`
Add blobs + peer messages to bundle 2020-04-22 13:21:55 +00:00			* `blobs/bundle/7Z2CSZK/MB1RE5G/6SKXRZ6/3ZGCNP8/VVEM3K0/XFMYKET/RDQSM5W.BSG`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`Additional notes:`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
Fix kitchen_sink.sh. Add tests for base32 encoder 2020-04-06 12:18:04 +00:00			`* It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.`

			`# Unanswered Questions`

			* PEER MESSAGES: I want to add a `--depth` option to bundle exports that would only return messages after the `nth` sequence number. It would not make sense to apply `--depth` to all peer messages in the bundle. It would not be practical to expect the user to provide a `--depth` for every peer every time a bundle is generated.
Update gem version in doc 2020-04-19 16:16:04 +00:00			* Create a new `received_on` index that records the local user's `depth` at the time of ingestion?