146 lines
6.8 KiB
Markdown
146 lines
6.8 KiB
Markdown
|
|
|
|
# Pigeon Ruby
|
|
|
|
This is a WIP [Pigeon Protocol](https://tildegit.org/PigeonProtocolConsortium/protocol_spec) client written in Ruby.
|
|
|
|
# Caveats
|
|
|
|
* Probably does not work on Windows, but testers are welcome to try it out. Please let me know how it goes.
|
|
* Blobs are not included in bundles yet.
|
|
* Not published to RubyGems yet (see installation instructions below)
|
|
* Not thread safe, never will be. This implementation is intended for a single user and makes many design tradeoffs around that use case.
|
|
* Bundle works, but is inefficient. Will optimize after proof of concept.
|
|
|
|
# Installation
|
|
|
|
We are not yet on Rubygems. The gem will be released after we are fully compliant with the spec and have high test coverage stats.
|
|
|
|
In the meantime:
|
|
|
|
```
|
|
git clone https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby.git
|
|
cd pigeon_ruby
|
|
gem build pigeon.gemspec
|
|
gem install pigeon-0.0.7.gem
|
|
pigeon-cli identity new # Should work. Raise issue if not.
|
|
pigeon-cli status
|
|
pigeon-cli help
|
|
```
|
|
|
|
# Usage: CLI
|
|
|
|
See `pigeon-cli help` for documentation.
|
|
See `kitchen_sink.sh` examples.
|
|
|
|
# Usage: Ruby Lib
|
|
|
|
TODO
|
|
|
|
# Current Status
|
|
|
|
- [X] pigeon identity new
|
|
- [X] pigeon identity show
|
|
- [X] pigeon status
|
|
- [X] pigeon blob set
|
|
- [X] pigeon blob get
|
|
- [X] pigeon peer add
|
|
- [X] pigeon peer remove
|
|
- [X] pigeon peer block
|
|
- [X] pigeon peer all
|
|
- [X] 100% coverage
|
|
- [X] Convert `".sig.ed25519"` literals to constants
|
|
- [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."
|
|
- [X] pigeon draft create
|
|
- [X] pigeon draft append
|
|
- [X] pigeon draft current
|
|
- [X] pigeon draft save
|
|
- [X] pigeon bundle create
|
|
- [X] Use JSON.stringify() for string keys (instead of `inspect`)
|
|
- [X] Move literals into `Pigeon` module as constants, again.
|
|
- [X] pigeon message find
|
|
- [X] pigeon message find-all for local feed.
|
|
- [X] pigeon bundle consume (We are minimally feature complete at this point)
|
|
- [X] Fix the diagram in the spec document
|
|
- [X] Validate inputs for `Draft#[]=`.
|
|
- [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
|
|
- [X] Use raw SHA256 hashes for blob multihashes, not hex.
|
|
- [X] Change all the `{40,90}` values in ::Lexer to real length values
|
|
- [X] Don't double-ingest messages. It will screw up indexes.
|
|
- [X] 100% test coverage
|
|
- [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
|
|
- [X] Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
|
|
- [X] Fix `scratchpad.sh` to use Base32
|
|
- [X] Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
|
|
- [X] Fix diagram in spec doc
|
|
- [X] refactor `Bundle.create` to use `message find-all`.
|
|
- [X] Rename `message find` to `message read`, since other finders return a multihash.
|
|
- [X] Message.ingest should be the only code path to message authoring.
|
|
- [X] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
|
|
- [X] Run Flog / Flay and friends to find duplications. Will aid in port to other languages.
|
|
- [X] Make all methods private except those required for the CLI.
|
|
- [X] Add Lipmaa links like the Bamboo folks do.
|
|
- [X] Set a max message size.
|
|
- [X] Clean up all singletons / .current hacks
|
|
- [X] Reduce cross cutting where collaborating objects need access to `@db`
|
|
- [X] Ensure all disks writes perform verification!
|
|
- [X] Make CLI names consistent with API names. Eg: find vs. read.
|
|
- [X] `find-all` should....find all. Currently finds your messages or maybe peers, but not all.
|
|
- [X] Add log count to `pigeon-cli status`
|
|
- [X] Delete `Draft#put` entirely.
|
|
- [X] Check block list before ingesting bundles.
|
|
- [X] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
|
|
- [X] Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
|
|
- [X] Rename `who_am_i` as `get_who_am_i` to follow VERB + NOUN convention.
|
|
- [ ] Find that non-deterministic runaway loop in the test suite.
|
|
- [ ] Update README.md / tutorial.rb (user manual for `Pigeon::Database`).
|
|
- [ ] Update spec document CLI usage examples to reflect API changes in 2020.
|
|
- [ ] Publish to RubyGems
|
|
|
|
# Optimizations
|
|
- [ ] add parsers and validators for all CLI inputs
|
|
- [ ] Make the switch to LevelDB, RocksDB, [UNQLite](https://unqlite.org/features.html) or similar (currently using Ruby PStore).
|
|
- [ ] Convert literals to constants, remove unused locals, reduce duplication, run linter.
|
|
- [ ] Reduce whole darn repo into single module to aide portability. `::Helpers` module is OK.
|
|
- [ ] Update the bundles.md document once `bundle consume` works.
|
|
- [ ] 100% documentation
|
|
- [ ] Performance benchmarks (Do this second to last!)
|
|
- [ ] Performance tuning (Do this last!)
|
|
|
|
# After v0.0.1
|
|
|
|
- [ ] (later, not now) Support partial verification via `lipmaa` property.
|
|
- [ ] Add mandatory `--since=` arg to `bundle create
|
|
- [ ] Interest and Disinterest Signalling for document routing: Create a `$blob_status` message to express `have`, `want` signalling. This can steer bundle creation and an eventual `--for` flag at bundle creation time to customize a bundle to a particular user.
|
|
- [ ] Add a schema for `$peer_status`. Eg: `block`, `unblock`, `follow`, `unfollow`.
|
|
|
|
# Idea Bin
|
|
|
|
- [ ] Map/reduce plugin support for custom indices?
|
|
- [ ] Ability to add a blob in one swoop using File objects and `Message#[]=`, maybe?
|
|
- [ ] Bundling via [Optar](http://ronja.twibright.com/optar/) or [Colorsafe](https://github.com/colorsafe/colorsafe)
|
|
|
|
# New Bundle Format
|
|
|
|
We have a bundle format that works, but it only exports messages.
|
|
|
|
We need a bundle format that may optionally include blobs as well.
|
|
|
|
Here's how we will support that:
|
|
|
|
1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
|
|
2. In the root directory of `bundle_x/`, a single `messages.pgn` file contains all messages.
|
|
* All messages are expected to be sorted by depth
|
|
* Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the `depth` field.
|
|
3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
|
|
* `blobs/bundle/7Z2CSZK/MB1RE5G/6SKXRZ6/3ZGCNP8/VVEM3K0/XFMYKET/RDQSM5W.BSG`
|
|
|
|
Additional notes:
|
|
|
|
* It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.
|
|
|
|
# Unanswered Questions
|
|
|
|
* PEER MESSAGES: I want to add a `--depth` option to bundle exports that would only return messages after the `nth` sequence number. It would not make sense to apply `--depth` to all peer messages in the bundle. It would not be practical to expect the user to provide a `--depth` for every peer every time a bundle is generated.
|
|
* Create a new `received_on` index that records the local user's `depth` at the time of ingestion?
|