Pigeon Ruby

This is a WIP Pigeon Protocol client written in Ruby.

Caveats

• Probably does not work on Windows, but testers are welcome to try it out. Please let me know how it goes.
• Blobs are not included in bundles yet.
• Not published to RubyGems yet (see installation instructions below)
• Not thread safe, never will be. This implementation is intended for a single user and makes many design tradeoffs around that use case.
• Bundle works, but is inefficient. Will optimize after proof of concept.

Installation

We are not yet on Rubygems. The gem will be released after we are fully compliant with the spec and have high test coverage stats.

In the meantime:

git clone https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby.git
cd pigeon_ruby
gem build pigeon.gemspec
gem install pigeon-0.0.7.gem
pigeon-cli identity new # Should work. Raise issue if not.
pigeon-cli status
pigeon-cli help

Usage: CLI

See pigeon-cli help for documentation. See kitchen_sink.sh examples.

Usage: Ruby Lib

TODO

Current Status

• pigeon identity new
• pigeon identity show
• pigeon status
• pigeon blob set
• pigeon blob get
• pigeon peer add
• pigeon peer remove
• pigeon peer block
• pigeon peer all
• 100% coverage
• Convert ".sig.ed25519" literals to constants
• Rename numerous "pigeon message ..." commands to "pigeon draft ..."
• pigeon draft create
• pigeon draft append
• pigeon draft current
• pigeon draft save
• pigeon bundle create
• Use JSON.stringify() for string keys (instead of inspect)
• Move literals into Pigeon module as constants, again.
• pigeon message find
• pigeon message find-all for local feed.
• pigeon bundle consume (We are minimally feature complete at this point)
• Fix the diagram in the spec document
• Validate inputs for Draft#[]=.
• Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
• Use raw SHA256 hashes for blob multihashes, not hex.
• Change all the {40,90} values in ::Lexer to real length values
• Don't double-ingest messages. It will screw up indexes.
• 100% test coverage
• Implement pigeon message find-all for peer feed. I will need to add index for author => message_count
• Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
• Fix scratchpad.sh to use Base32
• Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
• Fix diagram in spec doc
• refactor Bundle.create to use message find-all.
• Rename message find to message read, since other finders return a multihash.
• Message.ingest should be the only code path to message authoring.
• Don't allow any type of whitespace in kind or string keys. Write a test for this.
• Run Flog / Flay and friends to find duplications. Will aid in port to other languages.
• Make all methods private except those required for the CLI.
• Add Lipmaa links like the Bamboo folks do.
• Set a max message size.
• Clean up all singletons / .current hacks
• Reduce cross cutting where collaborating objects need access to @db
• Ensure all disks writes perform verification!
• Make CLI names consistent with API names. Eg: find vs. read.
• find-all should....find all. Currently finds your messages or maybe peers, but not all.
• Add log count to pigeon-cli status
• Delete Draft#put entirely.
• Check block list before ingesting bundles.
• Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
• Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
• Rename who_am_i as get_who_am_i to follow VERB + NOUN convention.
• Find that non-deterministic runaway loop in the test suite.
• Update README.md / tutorial.rb (user manual for Pigeon::Database).
• Update spec document CLI usage examples to reflect API changes in 2020.
• Publish to RubyGems

Optimizations

• add parsers and validators for all CLI inputs
• Make the switch to LevelDB, RocksDB, UNQLite or similar (currently using Ruby PStore).
• Convert literals to constants, remove unused locals, reduce duplication, run linter.
• Reduce whole darn repo into single module to aide portability. ::Helpers module is OK.
• Update the bundles.md document once bundle consume works.
• 100% documentation
• Performance benchmarks (Do this second to last!)
• Performance tuning (Do this last!)

After v0.0.1

• (later, not now) Support partial verification via lipmaa property.
• Add mandatory --since= arg to `bundle create
• Interest and Disinterest Signalling for document routing: Create a $blob_status message to express have, want signalling. This can steer bundle creation and an eventual --for flag at bundle creation time to customize a bundle to a particular user.
• Add a schema for $peer_status. Eg: block, unblock, follow, unfollow.

Idea Bin

• Map/reduce plugin support for custom indices?
• Ability to add a blob in one swoop using File objects and Message#[]=, maybe?
• Bundling via Optar or Colorsafe

New Bundle Format

We have a bundle format that works, but it only exports messages.

We need a bundle format that may optionally include blobs as well.

Here's how we will support that:

1. Create a bundle_X/ directory. The name is arbitrary and can be defined by the user.
2. In the root directory of bundle_x/, a single messages.pgn file contains all messages.

• All messages are expected to be sorted by depth
• Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the depth field.

3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
   • blobs/bundle/7Z2CSZK/MB1RE5G/6SKXRZ6/3ZGCNP8/VVEM3K0/XFMYKET/RDQSM5W.BSG

Additional notes:

• It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.

Unanswered Questions

• PEER MESSAGES: I want to add a --depth option to bundle exports that would only return messages after the nth sequence number. It would not make sense to apply --depth to all peer messages in the bundle. It would not be practical to expect the user to provide a --depth for every peer every time a bundle is generated.
  • Create a new received_on index that records the local user's depth at the time of ingestion?