Pigeon Ruby

This is a WIP [Pigeon Protocol] client written in Ruby.

Installation

This is a pre-release skeleton project. There is no gem yet. The gem will be released after we are fully compliant with the spec and have high test coverage stats.

In the meantime:

git clone https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby.git
cd pigeon_ruby
bundle install
./pigeon-cli # Should work. Raise issue if not.

Usage: CLI

We provide a CLI wrapper as pigeon-cli. Help is provided by running ./pigeon-cli.

See kitchen_sink.sh for an example of all commands.

Usage: Ruby Lib

TODO

Current Status

• pigeon identity new
• pigeon identity show
• pigeon status
• pigeon blob set
• pigeon blob get
• pigeon peer add
• pigeon peer remove
• pigeon peer block
• pigeon peer all
• 100% coverage
• Convert ".sig.ed25519" literals to constants
• Rename numerous "pigeon message ..." commands to "pigeon draft ..."
• pigeon draft create
• pigeon draft append
• pigeon draft current
• pigeon draft save
• pigeon bundle create
• Use JSON.stringify() for string keys (instead of inspect)
• Move literals into Pigeon module as constants, again.
• pigeon message find
• pigeon message find-all for local feed.
• pigeon bundle consume (We are minimally feature complete at this point)
• Fix the diagram in the spec document
• Validate inputs for Draft#[]=.
• Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
• Use raw SHA256 hashes for blob multihashes, not hex.
• Change all the {40,90} values in ::Lexer to real length values
• Don't double-ingest messages. It will screw up indexes.
• 100% test coverage
• Implement pigeon message find-all for peer feed. I will need to add index for author => message_count
• Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
• Fix scratchpad.sh to use Base32
• Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
• Fix diagram in spec doc
• refactor Bundle.create to use message find-all.
• Rename message find to message read, since other finders return a multihash.
• Message.ingest should be the only code path to message authoring.
• Don't allow any type of whitespace in kind or string keys. Write a test for this.
• Run Flog / Flay and friends to find duplications. Will aid in port to other languages.
• Make all methods private except those required for the CLI.
• Add Lipmaa links like the Bamboo folks do.
• Set a max message size.
• Clean up all singletons / .current hacks
• Reduce cross cutting where collaborating objects need access to @db
• Update README.md. Needs user manual for new Pigeon::Database class.
• Make the switch to LevelDB, RocksDB, UNQLite or similar (currently using Ruby PStore).
• Make CLI names consistent with API names. Eg: find vs. read.
• Create regexes in ::Lexer using strings and Regexp.new() for cleaner regexes.
• Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
• Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
• Check block list before ingesting bundles.
• Handle the three outcomes of bundle ingestion: ok, blocked, already_saved.
• add parsers and validators for all CLI inputs
• Reduce whole darn repo into single module to aide portability. ::Helpers module is OK.
• Use URNs instead of multihash?
• Ensure all disks writes perform verification!
• Publish a RubyGem
• Update the bundles.md document once bundle consume works.
• 100% documentation
• Update spec document CLI usage examples to reflect API changes in 2020.
• Performance benchmarks (Do this second to last!)
• Performance tuning (Do this last!)

After v0.0.1

• (later, not now) Support partial verification via lipmaa property.
• Add mandatory --since= arg to `bundle create
• Interest and Disinterest Signalling for document routing: Create a $gossip message to express blob.have, blob.want and to note last message received of a peer. This can steer bundle creation and an eventual --for flag at bundle creation time to customize a bundle to a particular user.

Idea Bin

• Map/reduce plugin support for custom indices?
• Ability to add a blob in one swoop using File objects and Message#[]=, maybe?

New Bundle Format

We have a bundle format that works, but it only exports messages.

We need a bundle format that may optionally include blobs as well.

Here's how we will support that:

1. Create a bundle_X/ directory. The name is arbitrary and can be defined by the user.
2. In the root directory of bundle_x/, a single messages.pgn file contains all messages.

• All messages are expected to be sorted by depth
• Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the depth field.

3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
   • blobs/sha256/AAAAAAAA/BBBBBBBB/CCCCCCCC/DDDDDDDD/EEEEEEEE/FFFFFFFF/G.HHH

Additional notes:

• It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.

Unanswered Questions

• PEER MESSAGES: I want to add a --depth option to bundle exports that would only return messages after the nth sequence number. It would not make sense to apply --depth to all peer messages in the bundle. It would not be practical to expect the user to provide a --depth for every peer every time a bundle is generated.
  • Create a new received_on index that records the local user's depth at the time of ingestion?