18 KiB

Raw Permalink Blame History

Ruby API Follow Along Tutorial

Introduction and Intended Audience

Pigeon is a peer-to-peer log database that serves the needs of off grid and delay-tolerant systems. Pigeon Ruby is a Ruby-based database client that is interoperable with other compliant Pigeon Protocol clients. It allows users to manage replicated, distributed log databases. Pigeon makes this possible even on systems with no internet access via sneakernet, thanks to a bundle file specification and extreme delay tolerance properties.

This document will teach you how to:

Create and manage a database.
Build messages using drafts.
Manage and query existing messages.
Replicate a database among peers.
Attach binary files to messages.
Communicate with remote databases using "bundles".

This guide assumes you are familiar with Ruby and the Pigeon Protocol. For an introduction to the protocol, see our protocol specification here.

Below is a list of all methods needed to run a Pigeon node. Pigeon strives to have a natural API rather than a simple one, which means you may not need to know every single method to operate a node successfully.

BLOB METHODS: #add_blob,#get_blob

BUNDLE METHODS: #export_bundle, #import_bundle

DRAFT METHODS: #publish_draft, #delete_current_draft, #get_draft, #new_draft, #update_draft

HELPER METHODS: #who_am_i, #get_message_by_depth, #get_message_count_for, #reset_database

MESSAGE METHODS: #add_message, #all_messages, #read_message, #have_message

PEER METHODS: #all_peers, #add_peer, #remove_peer, #all_blocks, #block_peer, #peer_blocked

Note to application developers: Please let us know what you build! Send an email to contact at vaporsoft.xyz with your progress.

Installation

Installation steps change over time. Please see README.md for the most up-to-date information.

Creating a Database Object

When building Pigeon-based applications, a Pigeon::Database object controls all interactions with the database. For the rest of the tutorial we will use the variable name db to refer to the current database.

You can create your own database with the following steps:

require "pigeon"
db = Pigeon::Database.new(path: "pigeon.db")
# => #<Pigeon::Database:0x000055a1ecca45e8>

An optional path: argument can be passed to Pigeon::Database.new. This arg will default to pigeon.db within the local directory if not provided. We recommend this default as it will allow you to use the command line interface more effectively.

If at any point you wish to start the tutorial over, you can reset your local database with the following command:

db.reset_database

One note about the pigeon.db file before moving to the next section: Do not share the pigeon.db file with anyone. Doing so will compromise the tamper-resistant properties of Pigeon and allow bad actors to forge messages using your name. Use bundles to safely share data with remote peers (covered later).

Working with Drafts

A message is the basic building block of a Pigeon database. As mentioned in the protocol spec, there are three parts to a message:

A header containing a kind field (similar to an email subject line) plus some additional meta data used by protocol clients.
A body containing user definable key / value pairs.
A footer containing a Crockford Base32 encoded ED25119 signature to prevent forgery.

As a convenience, the Pigeon Ruby client allows developers to keep zero or one "draft" messages. A draft message is a log message that has not been signed and has not been committed to the database. Think of it as your own personal scratchpad.

A draft is not part of the protocol spec. It is a convenience provided to users of this library. You could absolutely write messages by hand, calculate their signatures, convert everything to Base32 and manually add them to the database. This would be extremely tedious, however, so the draft functionality was added for convenience.

The examples below will center around the creation of a fictitious gardening journal app, a use case that Pigeon is well suited for.

Let's see if we have a draft to work with:

db.get_draft
RuntimeError: NO DRAFT. CREATE ONE FIRST. Call db.new_draft(kind:, body:)
from lib/pigeon/database.rb:104:in `get_draft'

We do not have a draft yet. We need to create one:

db.new_draft(kind: "garden_diary", body: {"message_text"=>"Tomato plant looking healthy."})
=> #<Pigeon::Draft:0x00005603ed399b48 @author="NONE",
#    @body={"greeting"=>"\"Hello, world!\""}, @depth=-1, @kind="example123",
#    @prev="NONE", @signature="NONE">

The command above creates a new draft entry of kind garden_entry with one key/value pair in the body. We can view the draft at any time via #get_draft:

db.get_draft
# => #<Pigeon::Draft:0x00005603ed81e830 @author="NONE",
#      @body={"greeting"=>"\"Hello, world!\""}, @depth=-1,
#      @kind="example123", @prev="NONE",
#      @signature="NONE">

Since the draft has not been published to the feed, its contents are mutable. We can add a new key/value pair to the message body with the following command:

db.update_draft("current_mood", "Feeling great")
# => "\"Feeling great\""

A few notes about this draft message:

"garden_diary is the message kind. This is definable by application developers and helps determine the type of message we are dealing with. A fictitious diary app might have other entries such as "status_update" or "photo_entry". It depends on the application you are building.
I used string keys for the "message_text" body entry rather than symbols or numbers. This is because you can only use strings for key / value pairs (no :symbols or numbers). Later on we will learn how to attach files to messages.
The body: part is optional. I could have called db.new_draft(kind: "garden_diary") and added key / value pairs to the body later.

Let's take a final look at our draft message. To get a more human readable version, I can use the render_as_draft method on a Draft object:

human_readable_string = db.get_draft.render_as_draft
# => "author DRAFT\nkind garden_dia...."
puts human_readable_string
# => author DRAFT
#    depth DRAFT
#    kind garden_diary
#    prev DRAFT
#
#    greeting:"Hello, world!"
#    current_mood:"Feeling great"

Some interesting things about the draft we just rendered:

Unlike a message, a draft has no signature (yet).
The author, kind, prev, depth properties are all set to "DRAFT". Real values will be populated when we finally publish the draft.

If we want to start over, we can delete a draft via delete_current_draft:

db.delete_current_draft
# => nil

Let's not do this, though. Instead, we will publish this draft in the next section.

Turning Drafts Into Messages

Now that I am happy with my draft, I can publish it. Once published, my message cannot be modified, so it is very important to visually inspect a draft with db.get_draft before proceeding. Since we did that in the last step, I will go ahead and publish the message:

my_message = db.publish_draft
=> #<Pigeon::Message:0x000055a751032c28
 @author=#<Pigeon::RemoteIdentity:0x000055a751032cf0 @multihash="USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG">,
 @body={"greeting"=>"\"Hello, world!\"", "current_mood"=>"\"Feeling great\""},
 @depth=0,
 @kind="garden_diary",
 @prev="NONE",
 @signature="QNY...208">

Let's look at our new message in a human-readable way:

puts my_message.render
# author USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG
# depth 0
# kind garden_diary
# prev NONE
#
# greeting:"Hello, world!"
# current_mood:"Feeling great"
#
# signature QNY...208

Unlike our draft, the message has a signature. The header fields are also populated.

In the next section, we will learn more about messages.

Working With Messages

Drafts can be helpful when you are building a message incrementally and need a place to temporarily store things between application restarts. What about when you have all the information you need and want to publish immediately?

In those cases, you can call db.add_message and your message will be published to your database immediately:

message = db.add_message("garden_entry", {"message_text" => "The basil is just OK", "current_mood" => "content"})
# => #<Pigeon::Message:0x00005653352af998
#  @author=#<Pigeon::RemoteIdentity:0x00005653352afa38 @multihash="USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG">,
#  @body={"message_text"=>"\"The basil is just OK\"", "current_mood"=>"\"content\""},
#  @depth=1,
#  @kind="garden_entry",
#  @prev="TEXT.NPNQZAP9CB79GP8J0SN52F38EBJ9WV370HX6MVZD3XB804TVQQB0",
#  @signature="95E...J3G">

puts message.render
# author USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG
# depth 1
# kind garden_entry
# prev TEXT.NPNQZAP9CB79GP8J0SN52F38EBJ9WV370HX6MVZD3XB804TVQQB0
#
# message_text:"The basil is just OK"
# current_mood:"content"
#
# signature 95E...J3G

We should now have 2 messages in the local database. Let's take a look using the db.all_messages method:

db.all_messages
# => ["TEXT.NPN...QB0", "TEXT.444...92G"]

The #all_messages method returns an array containing every message multihash in the database. We can then pass the multihash to the db.read_message method to retrieve the corresponding Pigeon::Message object.

Let's look at the old log message we created from a draft previously:

old_message = db.read_message("TEXT.444CC4NFHGQDQEZ6B6HSEPNZAZ80RSQF8TCAX8QR9NBR5T0XX92G")
# => #<Pigeon::Message:0x0000565335384f08
#  @author=#<Pigeon::RemoteIdentity:0x0000565335384da0 @multihash="USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG">,
#  @body={"message_text"=>"\"The basil is just OK\"", "current_mood"=>"\"content\""},
#  @depth=1,
#  @kind="garden_entry",
#  @prev="TEXT.NPNQZAP9CB79GP8J0SN52F38EBJ9WV370HX6MVZD3XB804TVQQB0",
#  @signature="95E...J3G">

puts old_message.render
# author USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG
# depth 1
# kind garden_entry
# prev TEXT.NPNQZAP9CB79GP8J0SN52F38EBJ9WV370HX6MVZD3XB804TVQQB0

# message_text:"The basil is just OK"
# current_mood:"content"

# signature 95E...J3G

Additionally, there is a have_message? helper that let's us know if we have a message in the local DB. It will return a Pigeon::Message (if found) or false:

db.have_message?("TEXT.QPNQGRBREXN4CB49RFZ8SQGXD98Z46FS08QH5ZATT6NE2HACC40X")
# => false

Working with Peers

STOPPED HERE.

Building a gardening diary is not very fun unless there is a way of sharing your work. Pigeon supports data transfer through the use of peers. When a peer adds you to their local machine, they begin replicating your database on their machine, thereby giving them access to your diary entries and also creating a redundant backup.

As we learned in the last section, Pigeon messages have a unique ID ("multihash"). Every message starts with the word "TEXT." and is followed by a base32 string of digits and letters. Much like messages, every Pigeon database has a unique identifier to identify itself. A database's multihash starts with the word "USER.". Like a message multihash, it is a long string of base32 characters:

USER.58844MCNB7ZF7HVKYFRBR7R7E75T8YXP4JBR267AS09RNMHEG3EG

We can call db.who_am_i to determine our local database multihash:

me = db.who_am_i
# => #<Pigeon::LocalIdentity:0x0000558496ad4bf0 @seed="___", @signing_key=#<Ed25519::SigningKey:0x0000558496ad4bc8>>

db.who_am_i returns a Pigeon::LocalIdentity rather than a string (which is what we want). To get results in a more copy/pastable format, call #multihash on the LocalIdentity object:

puts me.multihash
# USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG

You can send this string to all your friends so they can add you as a peer to their respective databases. Let's add a friend to our database now. My friend has informed me her Pigeon identity is "USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G":

db.add_peer("USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G")
# => "USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G"

My client will now keep a local copy of my friend's DB on disk at all times, assuming I have received a "bundle" for the user. Since Pigeon is an offline-only protocol, she will need to mail me an SD Card with her files. We will cover this later in the "Bundles" section.

If you ever lose track of who your peers are, you can call db.all_peers to get a list:

db.all_peers
# => ["USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G"]

You can also remove peers if you no longer need to replicate their messages:

db.remove_peer("USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G")
# => "USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G"
db.all_peers
# => []

It is also possible to block peers as needed via db.block_peer. block_peer is not the same as remove_peer. Blocking a peer will prevent gossip from flowing through your database. All of their messages will be ignored and none of your peers will be able to retrieve their messages through you via gossip. When you block a peer, you are essentially imposing an embargo against them. Bundles from blocked peers will neither be imported nor exported under any circumstance.

db.block_peer("USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G")
# => "USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G"

db.all_blocks
# => ["USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G"]

db.peer_blocked?("USER.753FT97S1FD3SRYPTVPQQ64F7HCEAZMWVBKG0C2MYMS5MJ3SBT6G")
# => true

Querying the Database

The client offers minimal query capabilities and indexes. More will be added at a later date. Please email contact at vaporsoft.xyz if you are interested in helping. Feature requests are welcome.

Fetch a Message by Feed Identity + Message Depth

my_peer = "USER.MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0"
db.get_message_by_depth(my_peer, 1)
# => "%6JD96QB2EQ30EN3DMHH50NXMR0RZ2GMH43P2DZB3HN6PE6NFE9A0.sha256"

Fetch Total Number of Messages in a Feed

db.get_message_count_for(my_peer)
# => 23

Attaching Files to Messages

There are limits to the usefulness (and size) of text content for a Pigeon message. When building a gardening diary app, a user will eventually want to attach garden photos to their entries.

Pigeon supports message file attachments in the form of blobs.

Once you have added a blob to your local database, it can be attached to messages using the special blob multihash string. Passing arbitrary binary data as a string via db.add_blob(binary_data) will do two things:

Store the file in your local database (unless we already have a copy)
Return a 57 character blob multihash, which can be used to reference the file inside of a message.

binary_data = File.read("kitty_cat.gif")
db.add_blob(binary_data)
# => "FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG"

Creating a blob returns a blob multihash (&FV0...MRG.sha256) which can be attached to a message in the form of keys or values:

the_blob_from_before = "FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG"
msg = db.add_message("photo", {"my_cat_picture" => "FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG"})
puts msg.render
# author USER.6DQ4RRNBKJ2T4EY5E1GZYYX6X6SZXV1W0GNH1HA4KGKA5KZ2Y2DG
# depth 2
# kind photo
# prev TEXT.444CC4NFHGQDQEZ6B6HSEPNZAZ80RSQF8TCAX8QR9NBR5T0XX92G
#
# my_cat_picture:FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG
#
# signature WT96XNJ6T006YS51ZDVPT1A7DJW2E4BTBZF66WHHWMKEP35MZ0YD30C32M8WQ85VK19SQFK47MXPEDMWW1GC0RV5XPYHT6WNDMGZM1R

If you want to retrieve a blob later, you can pass the blob multihash to db#get_blob. The client will return it as binary data.

db.get_blob("FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG")
# => "GIF89aX\u0000\u001F\u0000\xD58\u0000\u0000\u0000\u0000...

Communication with Peers via "Bundles"

Eventually, you will want to share your log messages with a peer, either as a form of communication or for the sake of creating redundant backups.

All data transfer operations in Pigeon are file based. To export data from your local database, one must create a "bundle", which is a file directory with a very specific layout. You can take this directory structure and burn it to a DVD-R, Zip and email, host on an HTTP server, seed it as a torrent, etc..

Think of bundles as a specialized archive format that a Pigeon-compliant database can easily ingest. The bundle mechanism will package all blobs and messages into a single exportable directory structure automatically. As long as your peer's client is compliant with the Pigeon spec, they will be replicated onto the peer's machine upon import.

Pigeon does not specify transport or compression concerns, but any reliable file transfer method is possible. For example, applying GZip compression to a bundle is a great idea, but the spec itself does not dictate how to do this. This is particularly true for network-related concerns. The Pigeon spec will never reference network transport in the spec.

In the example below, I will create a bundle called "bundle_for_my_peer".

db.export_bundle("bundle_for_my_peer")

After running this command, a directory with the name bundle_for_my_peer will appear in the current directory. I can send this directory to my peer using any reliable file transfer method.

Examples of possible file transfer mechanisms:

Host the directory on an HTTP / FTP server.
Apply ZIP compression and put it onto optical media (such as a CD-R)
Move the bundle onto a USB thumb drive with or without compression.

If you wish to ingest a peer's message, you can perform the operation in reverse:

db.import_bundle("a_bundle_my_peer_gave_me")

Wrapping Up

That's all there is to the protocol spec. In summary:

Messages are the core building block of Pigeon databases.
Messages can have file attachments via "blobs"
Messages can reference other users via "USER.****" multihashes
Messages are replicated onto peer machines via "following"
Messages can be avoided via "blocking"
Messages are shared between machines by passing around a disk directory structure known as a "bundle".

18 KiB Raw Permalink Blame History