Netscape Navigator 866bd9d2d0 | ||
---|---|---|
dist | ||
spec | ||
views | ||
.gitignore | ||
.rspec | ||
Gemfile | ||
Gemfile.lock | ||
README.md | ||
TODO.md | ||
kitchen_sink.sh | ||
logo.png | ||
pigeon-cli | ||
scratchpad.jpg |
README.md
Pigeon Ruby
A WIP pigeon protocol client.
How to Use
This is a pre-release skeleton project. There is no gem yet. The gem will be released after we are fully compliant with the spec and have high test coverage stats.
To get started, clone this repo and run ./pigeon-cli
in place of pigeon
.
Eg: pigeon identity show
becomes ./pigeon-cli show
.
Current Status
- pigeon identity new
- pigeon identity show
- pigeon status
- pigeon blob set
- pigeon blob get
- pigeon peer add
- pigeon peer remove
- pigeon peer block
- pigeon peer all
- 100% coverage
- Convert
".sig.ed25519"
literals to constants - Rename numerous "pigeon message ..." commands to "pigeon draft ..."
- pigeon draft create
- pigeon draft append
- pigeon draft current
- pigeon draft save
- pigeon bundle create
- Use JSON.stringify() for string keys (instead of
inspect
) - Move literals into
Pigeon
module as constants, again. - pigeon message find
- pigeon message find-all for local feed.
- pigeon bundle consume (We are minimally feature complete at this point)
- Fix the diagram in the spec document
- Validate inputs for
Draft#[]=
. - Put all the [HEADER, string, FOOTER].join("") nonsense into Pigeon::Helpers
- Use raw SHA256 hashes for blob multihashes, not hex.
- Change all the
{40,90}
values in ::Lexer to real length values - Don't double-ingest messages. It will screw up indexes.
- 100% test coverage
- Implement pigeon message find-all for peer feed. I will need to add index for
author => message_count
- Switch to Crockford base32- Simplifies support for legacy systems. Easy to implement.
- Fix
scratchpad.sh
to use Base32 - Rename (RemoteIdentity|LocalIdentity)#public_key to #multihash for consistency with other types.
- Fix diagram in spec doc
- refactor
Bundle.create
to usemessage find-all
. - Rename
message find
tomessage read
, since other finders return a multihash. - Message.ingest should be the only code path to message authoring.
- Don't allow any type of whitespace in
kind
orstring
keys. Write a test for this. - Make all methods private except those required for the CLI.
- Run Flog / Flay and friends to find duplications. Will aid in port to other languages.
- Make CLI names consistent with API names. Eg: find vs. read.
- Add Lipmaa links like the Bamboo folks do.
- Create regexes in ::Lexer using strings and Regexp.new() for cleaner regexes.
- Make the switch to LevelDB, RocksDB, UNQLite or similar (currently using Ruby PStore).
- Need a way of importing / exporting a feeds blobs. (see "Bundle Brainstorming" below)
- Need a way of adding peers messages / gossip to bundles. (see "Bundle Brainstorming" below)
- Add mandatory
--since=
arg tobundle create
- Check block list before ingesting bundles.
- Handle the three outcomes of bundle ingestion:
ok
,blocked
,already_saved
. - add parsers and validators for all CLI inputs
- Remove all
.current
"singletons" / hacks - Reduce whole darn repo into single module to aide portability.
::Helpers
module is OK. - Use URNs instead of multihash?
- Ensure all disks writes perform verification!
- Publish a RubyGem
- Update the bundles.md document once
bundle consume
works. - 100% documentation
- Update spec document CLI usage examples to reflect API changes in 2020.
- Performance benchmarks (Do this second to last!)
- Performance tuning (Do this last!)
- Set a max message size of 4k
Do this later
- Interest and Disinterest Signalling for document routing: Create a
$gossip
message to expressblob.have
,blob.want
and to note last message received of a peer. This can steer bundle creation and an eventual--for
flag at bundle creation time.
Idea Bin
- Map/reduce plugin support for custom indices?
- Ability to add a blob in one swoop using File objects and
Message#[]=
, maybe?
New Bundle Format
We have a bundle format that works, but it only exports messages.
We need a bundle format that may optionally include blobs as well.
Here's how we will support that:
- Create a
bundle_X/
directory. The name is arbitrary and can be defined by the user. - In the root directory of
bundle_x/
, a singlemessages.pgn
file contains all messages.
- All messages are expected to be sorted by depth
- Messages from multiple authors may be included in a single bundle, but the messages must appear in the correct order with regards to the
depth
field.
- Blobs are stored in a very specific hierarchy to maintain FAT compatibility:
blobs/sha256/AAAAAAAA/BBBBBBBB/CCCCCCCC/DDDDDDDD/EEEEEEEE/FFFFFFFF/G.HHH
Additional notes:
- It is recommended to compress bundles (ex: *.zip files) but these concerns are not handled by the protocol currently.
Unanswered Questions
- PEER MESSAGES: I want to add a
--depth
option to bundle exports that would only return messages after thenth
sequence number. It would not make sense to apply--depth
to all peer messages in the bundle. It would not be practical to expect the user to provide a--depth
for every peer every time a bundle is generated.- Create a new
received_on
index that records the local user'sdepth
at the time of ingestion?
- Create a new