2019-10-10 00:32:54 +00:00
2019-09-22 02:30:03 +00:00
# Pigeon Ruby
A WIP pigeon protocol client.
# How to Use
This is a pre-release skeleton project. There is no gem yet. The gem will be released after we are fully compliant with the spec and have high test coverage stats.
To get started, clone this repo and run `./pigeon-cli` in place of `pigeon` .
Eg: `pigeon identity show` becomes `./pigeon-cli show` .
# Current Status
- [X] pigeon identity new
- [X] pigeon identity show
2019-09-25 01:19:25 +00:00
- [X] pigeon status
2019-09-22 02:31:01 +00:00
- [X] pigeon blob set
2019-09-22 11:00:19 +00:00
- [X] pigeon blob get
2019-09-23 01:07:14 +00:00
- [X] pigeon peer add
- [X] pigeon peer remove
- [X] pigeon peer block
2019-09-25 01:31:32 +00:00
- [X] pigeon peer all
2020-03-07 15:38:34 +00:00
- [X] 100% coverage
2020-03-14 03:02:45 +00:00
- [X] Convert `".sig.ed25519"` literals to constants
2020-03-14 03:17:56 +00:00
- [X] Rename numerous "pigeon message ..." commands to "pigeon draft ..."
2020-03-14 03:41:31 +00:00
- [X] pigeon draft create
- [X] pigeon draft append
- [X] pigeon draft current
- [X] pigeon draft save
2020-03-15 16:51:13 +00:00
- [X] pigeon bundle create
2020-03-15 17:50:57 +00:00
- [X] Use JSON.stringify() for string keys (instead of `inspect` )
- [X] Move literals into `Pigeon` module as constants, again.
2020-03-15 18:50:00 +00:00
- [X] pigeon message find
- [X] pigeon message find-all for local feed.
2020-03-31 12:30:44 +00:00
- [X] pigeon bundle consume (We are minimally feature complete at this point)
- [X] Fix the diagram in the spec document
2020-04-02 12:32:21 +00:00
- [X] Validate inputs for `Draft#[]=` .
- [X] Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
2020-04-02 12:58:57 +00:00
- [X] Use raw SHA256 hashes for blob multihashes, not hex.
- [X] Change all the `{40,90}` values in ::Lexer to real length values
2020-04-03 10:19:42 +00:00
- [X] Don't double-ingest messages. It will screw up indexes.
2020-04-03 10:35:48 +00:00
- [X] 100% test coverage
- [X] Implement pigeon message find-all for peer feed. I will need to add index for `author => message_count`
2020-04-04 22:35:07 +00:00
- [ ] Stop using base64 in favor of base32 with no padding? Simplifies support for legacy systems. Easy to implement.
- [ ] Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
- [ ] Need a way of adding a peers messages / blobs to bundles. (see "Bundle Brainstorming" below)
2020-04-03 10:35:48 +00:00
- [ ] refactor `Bundle.create` to use `message find-all` .
2020-04-04 22:35:07 +00:00
- [ ] Add mandatory `--from=` arg to `bundle create`
- [ ] Make the switch to LevelDB, RocksDB or similar (currently using Ruby PStore).
- [ ] Change all multihashes to Base32 to support case-insensitive file systems?
2020-04-02 12:58:57 +00:00
- [ ] Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
2020-03-18 12:32:40 +00:00
- [ ] Rename `message find` to `message read` , since other finders return a multihash.
2020-04-03 10:35:48 +00:00
- [ ] Don't allow any type of whitespace in `kind` or `string` keys. Write a test for this.
- [ ] Check block list before ingesting bundles.
2020-03-17 13:40:09 +00:00
- [ ] Create regexes in ::Lexer using strings and Regexp.new() for cleaner regexes.
2020-04-03 10:35:48 +00:00
- [ ] Handle the three outcomes of bundle ingestion: `ok` , `blocked` , `already_saved` .
2020-03-15 18:12:26 +00:00
- [ ] add parsers and validators for all CLI inputs
2020-03-14 03:41:31 +00:00
- [ ] Remove all `.current` "singletons" / hacks
2020-04-02 12:32:21 +00:00
- [ ] Reduce whole darn repo into single module to aide portability. Maybe a second `::Support` module is OK.
- [ ] Update the bundles.md document once `bundle consume` works.
- [ ] Use URNs instead of multihash?
- [ ] Add `.pigeon` file extensions
2020-03-14 03:02:45 +00:00
- [ ] Add Lipmaa links like the Bamboo folks do.
2020-04-03 10:35:48 +00:00
- [ ] Ensure all disks writes perform verification!
2020-03-08 16:18:05 +00:00
- [ ] Publish a RubyGem
2020-04-03 10:35:48 +00:00
- [ ] 100% documentation
- [ ] Update spec document CLI usage examples to reflect API changes in 2020.
- [ ] Performance benchmarks (Do this second to last!)
- [ ] Performance tuning (Do this last!)
2020-03-08 16:18:05 +00:00
# Idea Bin
- [ ] Map/reduce plugin support for custom indices?
- [ ] Ability to add a blob in one swoop using File objects and `Message#[]=` , maybe?
2020-04-04 22:35:07 +00:00
# Bundle Brainstorming
Bundles export need to export two things:
* Messages
* Blobs
Currently, we're only exporting messages.
We need to export both, though.
## Idea: Output a directory (Zip it Yourself)
1. Create a `bundle_X/` directory. The name is arbitrary and can be defined by the user.
2. In the root directory of `bundle_x/` , a single `messages.pgn` file contains all messages.
* All messages are expected to be sorted by depth
* Messages from multiple authors may be included in a single bundle, but the messages must apear in the correct order with regards to the `depth` field.
3. Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
* Blob multihashes are decoded from URL safe b64
* Option I:
* Blobs are re-encoded into base32 without padding (to support legacy filesystems)
* Option II (not safe for FAT16?):
* Blobs are encoded as Base16:
* `xxxxxxxxxxxxxxxx/blobs/sha256/HEZB3/IFXHL4Y3/H4MWGKZV/42VVMO4S.4VQ`
* `N` represents a single character in a blob's multihash
# Why Base32?
I want to support things like:
* Hosting bundles on file servers (SFTP, HTTP, etc..)
* Ingesting blobs on constraints devices (Eg: FAT16 filesystems, retro machines)
To do that:
* Base 16 (hex) creates blob filenames that are too long for FAT16 and visually awkward.
* Base64 can't be hosted on a web server
* URLsafeBase64 can't be used on case insensitive file systems
