Simple and interoperable CI/CD system

forgehook is a collection of scripts following a simple interface to build a full Continuous Integration/Delivery platform. forgehook provides first-class multi-user experience for tilde/pubnix servers, and easy integration with your custom tooling.

These scripts are not intended to be used for generic webhooks, but for those produced indicating changes on a software repository. If you are looking for a more generic solution, please take a look at webhook or webhookd instead.

Note: Some specific parts of forgehook do not yet correspond to what you will find here. That's because I'm taking some time to step back and think about what's implemented so far and how it should end up, before i deepdive into the code again. The inconsistencies are marked with TODO notes.

Current status

As of April 28, the reference implementation for forgehook-db (databases/unix) works fine, so does forgehook-notify and the default forgehook-trigger (triggers/git-build). No endpoint is implemented yet.

Introduction to webhooks

Webhooks are simple web pings performed by a client to inform a server that something happened (push model). In the context of a forge (such as Gitea), a webhook additionally contains information as JSON payload about what changed on the repository, as well as an HTTP signature in order to authenticate the client who sent the webhook via a shared secret.

Examples of forge webhooks can be found for: Gitea Github Gitlab Gogs. Currently, no endpoint is implemented, however Gitea support will come soon (for tildegit integration).

Note: If you are not running a web-based forge such as those mentioned above, but run your own git server, you do not need webhooks to build a CI/CD system. All you need are server-side git hooks as explained in the docs.

Getting started

In this section, you will find:

• an overview of the core concepts of forgehook
• a short intro to the forgehook command-line interface (CLI)
• a setup guide
• a configuration guide

General overview

forgehook is a collection of scripts to let users manage their subscriptions to remote repositories (currently git only) and trigger something when a legitimate updated notification is received. Typically, forgehook is intended to be run alongside git-build.sh to automatically trigger build tasks when an update is perfomed on the repository.

Simplicity and extensibility are core concerns of forgehook, focusing on defining standard interfaces between different components so that you can reimplement each part of it to better suit your needs. Sharing of your tricks is highly encouraged! And remember, if the tool gets in your way, it's a bug so please report it.

The three components of forgehook are:

• endpoints which receive and validate webhooks of dubious authenticity (through a shared secret) (TODO: no endpoint is implemented yet!)
• databases which retrieve/store subscriptions and secrets
• triggers which does stuff as a subscribed user in case of update

Although there can be many endpoints, there can only ever be one database and one trigger configured for the system. These components are interchangeable as they follow simple interfaces described in the Endpoints, Triggers and Databases sections.

forgehook aims to be secure through simplicity and auditability. Security concerns are addressed in a dedicated section.

Setup

In most cases, you can just do ./setup.sh from the repository folder and it will setup everything just fine as long as you are a sudoer

TODO: introduce the setup.sh script and describe manual steps for installing

Configuration

Configurating the forgehook user can only be done at setup time. Please refer to the setup docs.

TODO: Configuration for endpoint (none at the moment), databases (only unix support at the moment) and triggers (only git-build at the moment)

CLI intro

So you want to subscribe to updates on a remote repository? This is a quick introduction to the forgehook CLI. The complete reference can be found in the docs/cli.md file.

Subscribing to a repository

To subscribe to a remote repository, use the forgehook add REPO command, where REPO is the URL of this repository.

If there is already a secret shared with the repository, the command will simply subscribe you to updates. However, if no secret has been configured yet, it will either extract it from your first argument (forgehook add REPO SECRET) or generate one for you.

Note: If you just registered a secret, don't forget to give it to your forge.

$ webhook add "https://tildegit.org/tilde-fr/infra"
[webhook] Your secret for https://tildegit.org/tilde-fr/infra is now:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # <--- Hex-encoded /dev/urandom

TODO: Currently, no ownership check is performed on the repo (see issue #6)

Unsubscribing

To unsubscribe from a repository, simply use the forgehook remove command:

$ webhook unsubscribe "https://tildegit.org/tilde-fr/infra"
[webhook] Unsubscribed from https://tildegit.org/tilde-fr/infra
[webhook] Users can still subscribe to this remote. To remove it entirely, run:
webhook remove "https://tildegit.org/tilde-fr/infra"

Listing subscriptions

The list command lists your current subscriptions. If a URL is passed as argument, the command returns the list of users currently subscribed to the repository.

$ webhook list
https://tildegit.org/tilde-fr/infra\tXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Note: For repositories you do not own, the list command only returns one repository URLs per line. However, additional information separated by tabulations (\t) are output for repositories you own, as explained in the CLI reference.

TODO: currently, only listing one's subscriptions is supported

Viewing/changing a secret

For a repository you own, you can always view your secret with the forgehook secret URL command. If you give it an additional SECRET, it will replace the current secret with this value:

$ webhook secret https://tildegit.org/tilde-fr/infra
[webhook] Your secret for https://tildegit.org/tilde-fr/infra is now:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ 
$ webhook secret "https://tildegit.org/tilde-fr/infra" YYYYYYYYYYYYYYYY
[webhook] Your secret for https://tildegit.org/tilde-fr/infra is now:
YYYYYYYYYYYYYYYY

Architecture

There are two entry points for the forgehook system:

• a user running the forgehook script to manage their subscriptions/secrets

• an endpoint running forgehook-notify to announce a legitimate update was received for a remote

forgehook-notify takes the URL of the updated repository as argument, checks for current user subscription, and runs the trigger (/usr/local/bin/forgehook-trigger) as each user currently subscribed. The forgehook CLI interface is further described here.

In the following sections, we'll explore how endpoints, triggers and databases are.

Endpoints

Endpoints are simple services vouching for repository updates, usually by verifying a shared secret. Shortly, they are trusted 3rd party tools querying the local database in order to validate update notifications submitted by a remote forge (usually through a webhook). When a request is validated, an endpoint notifies the current trigger through the forgehook-notify command.

The steps performed by an endpoint are:

1. Extract repository URL and secret from the remote notification (eg. webhook)
2. Query the local database for a secret matching this URL
3. If nothing is found, the program stops
4. Compare the remote and local secrets
5. If they don't match, the program stops
6. Run forgehook-notify with the remote URL as argument

There can be as many endpoints as you like to suit your needs. This allows to receive updates from multiple sources, such as different web forges, or even from a local post-receive git hook. However, remember your system is only as secure as your endpoints, as explained in the Security section.

Triggers

Triggers are simple scripts running as a user who subscribed to a task, thanks to sudo magic performed by forgehook-notify. Trigger configuration is system-wide and there is only one trigger at any given moment (/usr/local/bin/forgehook-trigger).

A trigger receives a repository URL as argument. This repository is assumed to have legitimately received an update, as bad-faith requests have been filtered by the endpoint who received the original request.

A trigger may:

• send a mail or some sort of notification for the update
• trigger a test/deployment using a 3rd party tool such as git-build.sh

Currently, only git-build is supported as trigger. More may come in the future, and contributions are welcome! Notably, it should be possible to integrate Github/Gitlab CI scripts as a trigger.

Note: Triggers are by nature unprivileged and only runs code the user should consent to (having subscribed to a repository). Running tests or building websites should not require complex infrastructure, as long as you have an account on a machine somewhere.

Databases

Databases are simple programs storing information about repository ownership, secrets and subscriptions. They may operate over an SQL or LDAP database, as long as they respect the forgehook CLI interface described here. Database configuration is systemwide and there is only one database at any given moment (/usr/local/bin/forgehook-db).

The provided reference implementation for a forgehook database is a flat-file database managed by bash scripts, located in databases/unix.sh, and documented here.

Security

TODO: Explain sudo tricks and suggest everyone should read the code in its entirely because it's brief

FAQ

See docs/faq.md