Pigeon-Ruby/README.md

![](logo.png)

# Pigeon Ruby

A WIP pigeon protocol client.

# How to Use

This is a pre-release skeleton project. There is no gem yet. The gem will be released after we are fully compliant with the spec and have high test coverage stats.

To get started, clone this repo and run `./pigeon-cli` in place of `pigeon`.

Eg: `pigeon identity show` becomes `./pigeon-cli show`.

# Current Status

 - [X] pigeon identity new
 - [X] pigeon identity show
 - [X] pigeon status
 - [X] pigeon blob set
 - [X] pigeon blob get
 - [X] pigeon peer add
 - [X] pigeon peer remove
 - [X] pigeon peer block
 - [X] pigeon peer all
 - [X] 100% coverage
 - [X] Convert `".sig.ed25519"` literals to constants
 - [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."
 - [X] pigeon draft create
 - [X] pigeon draft append
 - [X] pigeon draft current
 - [X] pigeon draft save
 - [X] pigeon bundle create
 - [X] Use JSON.stringify() for string keys (instead of `inspect`)
 - [X] Move literals into `Pigeon` module as constants, again.
 - [X] pigeon message find
 - [X] pigeon message find-all for local feed.
 - [X] pigeon bundle consume (We are minimally feature complete at this point)
 - [X] Fix the diagram in the spec document
 - [X] Validate inputs for `Draft#[]=`.
 - [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
 - [X] Use raw SHA256 hashes for blob multihashes, not hex.
 - [X] Change all the `{40,90}` values in ::Lexer to real length values
 - [X] Don't double-ingest messages. It will screw up indexes.
 - [X] 100% test coverage
 - [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
 - [X] Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
 - [X] Fix `scratchpad.sh` to use Base32
 - [X] Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
 - [X] Fix diagram in spec doc
 - [X] refactor `Bundle.create` to use `message find-all`.
 - [X] Rename `message find` to `message read`, since other finders return a multihash.
 - [ ] Message.ingest should be the only code path to message authoring.
 - [ ] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
 - [ ] Add Lipmaa links like the Bamboo folks do.
 - [ ] Create regexes in ::Lexer using strings and Regexp.new() for cleaner regexes.
 - [ ] Make the switch to LevelDB, RocksDB, [UNQLite](https://unqlite.org/features.html) or similar (currently using Ruby PStore).
 - [ ] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
 - [ ] Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
 - [ ] Add mandatory `--since=` arg to `bundle create`
 - [ ] Check block list before ingesting bundles.
 - [ ] Handle the three outcomes of bundle ingestion: `ok`, `blocked`, `already_saved`.
 - [ ] add parsers and validators for all CLI inputs
 - [ ] Remove all `.current` "singletons" / hacks
 - [ ] Reduce whole darn repo into single module to aide portability. `::Helpers` module is OK.
 - [ ] Use URNs instead of multihash?
 - [ ] Ensure all disks writes perform verification!
 - [ ] Publish a RubyGem
 - [ ] Update the bundles.md document once `bundle consume` works.
 - [ ] 100% documentation
 - [ ] Update spec document CLI usage examples to reflect API changes in 2020.
 - [ ] Performance benchmarks (Do this second to last!)
 - [ ] Performance tuning (Do this last!)

# Idea Bin
 - [ ] Map/reduce plugin support for custom indices?
 - [ ] Ability to add a blob in one swoop using File objects and `Message#[]=`, maybe?

# New Bundle Format

We have a bundle format that works, but it only exports messages.

We need a bundle format that may optionally include blobs as well.

Here's how we will support that:

1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
2. In the root directory of `bundle_x/`, a single `messages.pgn` file contains all messages.
  * All messages are expected to be sorted by depth
  * Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the `depth` field.
3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
    * `blobs/sha256/AAAAAAAA/BBBBBBBB/CCCCCCCC/DDDDDDDD/EEEEEEEE/FFFFFFFF/G.HHH`

Additional notes:

 * It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.

# Unanswered Questions

 * PEER MESSAGES: I want to add a `--depth` option to bundle exports that would only return messages after the `nth` sequence number. It would not make sense to apply `--depth` to all peer messages in the bundle. It would not be practical to expect the user to provide a `--depth` for every peer every time a bundle is generated.
   * Create a new `received_on` index that records the local user's `depth` at the time of ingestion?
Add logo 2019-10-10 00:32:54 +00:00			`![](logo.png)`

Coo... coo... 2019-09-22 02:30:03 +00:00			`# Pigeon Ruby`

			`A WIP pigeon protocol client.`

			`# How to Use`

			`This is a pre-release skeleton project. There is no gem yet. The gem will be released after we are fully compliant with the spec and have high test coverage stats.`

			To get started, clone this repo and run `./pigeon-cli` in place of `pigeon`.

			Eg: `pigeon identity show` becomes `./pigeon-cli show`.

			`# Current Status`

			`- [X] pigeon identity new`
			`- [X] pigeon identity show`
Cleanup and test scripts. TODO: Why does blocking not delete previous peers? 2019-09-25 01:19:25 +00:00			`- [X] pigeon status`
Update README 2019-09-22 02:31:01 +00:00			`- [X] pigeon blob set`
`blob get` 2019-09-22 11:00:19 +00:00			`- [X] pigeon blob get`
Finish most `peer` commands. NEXT: peer all 2019-09-23 01:07:14 +00:00			`- [X] pigeon peer add`
			`- [X] pigeon peer remove`
			`- [X] pigeon peer block`
Fix bug in block/remove 2019-09-25 01:31:32 +00:00			`- [X] pigeon peer all`
:tada: 100% coverage. NEXT: Verify message chain creation via tests. Currently, it is not working correctly. Take a look at scratchpad.sh. The `prev:` values are wrong. It is probably as simple as updating local indexes when Pigeon::Storage#save_message is called. Re run the indexes at that time. Or heck, add a map/reduce ability? 2020-03-07 15:38:34 +00:00			`- [X] 100% coverage`
Convert `".sig.ed25519"` literals to constants. Update README. 2020-03-14 03:02:45 +00:00			- [X] Convert `".sig.ed25519"` literals to constants
Convert "message..." commands to "draft..." commands 2020-03-14 03:17:56 +00:00			`- [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."`
TODO: pigeon bundle create 2020-03-14 03:41:31 +00:00			`- [X] pigeon draft create`
			`- [X] pigeon draft append`
			`- [X] pigeon draft current`
			`- [X] pigeon draft save`
We have BUNDLE CREATION :tada: :tada: :tada: 2020-03-15 16:51:13 +00:00			`- [X] pigeon bundle create`
Constantize literals for portability and DRYness 2020-03-15 17:50:57 +00:00			- [X] Use JSON.stringify() for string keys (instead of `inspect`)
			- [X] Move literals into `Pigeon` module as constants, again.
Add `message find` and `message find-all` (local user only - find-all for peers :soon:) 2020-03-15 18:50:00 +00:00			`- [X] pigeon message find`
			`- [X] pigeon message find-all for local feed.`
Test case: bundle ingestion 2020-03-31 12:30:44 +00:00			`- [X] pigeon bundle consume (We are minimally feature complete at this point)`
			`- [X] Fix the diagram in the spec document`
Add test for Storage#find_all 2020-04-02 12:32:21 +00:00			- [X] Validate inputs for `Draft#[]=`.
			`- [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers`
Add correct length numbers to Lexer regexes for multihashes 2020-04-02 12:58:57 +00:00			`- [X] Use raw SHA256 hashes for blob multihashes, not hex.`
			- [X] Change all the `{40,90}` values in ::Lexer to real length values
Works again! :tada: 2020-04-03 10:19:42 +00:00			`- [X] Don't double-ingest messages. It will screw up indexes.`
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [X] 100% test coverage`
			- [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
[STABLE] Convert URLSafeb64 to Crockford32. 2020-04-06 01:01:59 +00:00			`- [X] Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.`
Fix kitchen_sink.sh. Add tests for base32 encoder 2020-04-06 12:18:04 +00:00			- [X] Fix `scratchpad.sh` to use Base32
Rename .public_key attributes to .multihash for consistency 2020-04-06 12:22:24 +00:00			`- [X] Rename (RemoteIdentity\|LocalIdentity)#public_key to #multihash for consistency with other types.`
Use `find_all` when creating bundles 2020-04-06 12:32:19 +00:00			`- [X] Fix diagram in spec doc`
			- [X] refactor `Bundle.create` to use `message find-all`.
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [X] Rename `message find` to `message read`, since other finders return a multihash.
WIP. 7 tests failing. Refactor Pigeon::Draft / ::Message Pigeon::Message was holding on to too many drafting responsibilities. It also was possible for the local identity to inadvertantly author malformed messages. This is because local messages were not passed through the ::Lexer / ::Parser and thereby did not receive the same scrutiny of remote messages. To avoid security problems later, and for additional security, I will only allow messages to be saved after passing through the lexer/parser. This means moving much of Pigeon::Message's logic into Pigeon::Draft. 2020-04-07 13:58:53 +00:00			`- [ ] Message.ingest should be the only code path to message authoring.`
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [ ] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
Use `find_all` when creating bundles 2020-04-06 12:32:19 +00:00			`- [ ] Add Lipmaa links like the Bamboo folks do.`
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			`- [ ] Create regexes in ::Lexer using strings and Regexp.new() for cleaner regexes.`
Test case: ingesting malformed messages (whitespace keys and kind) 2020-04-07 13:13:28 +00:00			`- [ ] Make the switch to LevelDB, RocksDB, [UNQLite](https://unqlite.org/features.html) or similar (currently using Ruby PStore).`
Fix kitchen_sink.sh. Add tests for base32 encoder 2020-04-06 12:18:04 +00:00			`- [ ] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)`
			`- [ ] Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)`
			- [ ] Add mandatory `--since=` arg to `bundle create`
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [ ] Check block list before ingesting bundles.`
			- [ ] Handle the three outcomes of bundle ingestion: `ok`, `blocked`, `already_saved`.
Formatting updates 2020-03-15 18:12:26 +00:00			`- [ ] add parsers and validators for all CLI inputs`
TODO: pigeon bundle create 2020-03-14 03:41:31 +00:00			- [ ] Remove all `.current` "singletons" / hacks
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [ ] Reduce whole darn repo into single module to aide portability. `::Helpers` module is OK.
Add test for Storage#find_all 2020-04-02 12:32:21 +00:00			`- [ ] Use URNs instead of multihash?`
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [ ] Ensure all disks writes perform verification!`
DRY up storage initialization / reset code 2020-03-08 16:18:05 +00:00			`- [ ] Publish a RubyGem`
WIP: Validate that `kind` and body keys dont contain whitespace 2020-04-06 12:52:32 +00:00			- [ ] Update the bundles.md document once `bundle consume` works.
Re-prioritize README TODO items 2020-04-03 10:35:48 +00:00			`- [ ] 100% documentation`
			`- [ ] Update spec document CLI usage examples to reflect API changes in 2020.`
			`- [ ] Performance benchmarks (Do this second to last!)`
			`- [ ] Performance tuning (Do this last!)`
DRY up storage initialization / reset code 2020-03-08 16:18:05 +00:00
			`# Idea Bin`
			`- [ ] Map/reduce plugin support for custom indices?`
			- [ ] Ability to add a blob in one swoop using File objects and `Message#[]=`, maybe?
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`# New Bundle Format`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`We have a bundle format that works, but it only exports messages.`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`We need a bundle format that may optionally include blobs as well.`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`Here's how we will support that:`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
			1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
			2. In the root directory of `bundle_x/`, a single `messages.pgn` file contains all messages.
			`* All messages are expected to be sorted by depth`
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			* Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the `depth` field.
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00			`3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:`
[STABLE] Convert URLSafeb64 to Crockford32. 2020-04-06 01:01:59 +00:00			* `blobs/sha256/AAAAAAAA/BBBBBBBB/CCCCCCCC/DDDDDDDD/EEEEEEEE/FFFFFFFF/G.HHH`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
=== Begin conversion to Base32 === 2020-04-05 20:31:44 +00:00			`Additional notes:`
WIP. Brainstorm base32 ideas. 2020-04-04 22:35:07 +00:00
Fix kitchen_sink.sh. Add tests for base32 encoder 2020-04-06 12:18:04 +00:00			`* It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.`

			`# Unanswered Questions`

			* PEER MESSAGES: I want to add a `--depth` option to bundle exports that would only return messages after the `nth` sequence number. It would not make sense to apply `--depth` to all peer messages in the bundle. It would not be practical to expect the user to provide a `--depth` for every peer every time a bundle is generated.
			* Create a new `received_on` index that records the local user's `depth` at the time of ingestion?