Webhook endpoints

This repository contains documentation and tests about running webhook endpoints as part of a forge suite setup. There are tests for CLI webhook validation programs (such as whck), as well as for web validation programs (HTTP endpoints such as endpoints.php). A certain number of conventions (environment variables and folder structure) should make it "just work" for different kinds of setups including shared hosts with unprivileged accounts. For setup information, please head over to the Setup section.

Below, you will find specifications for the CLI and Web programs. For the reference of supported webhook sources and their respective property, please read the Sources section

Command-line interface

A CLI validator is a program that parses an identifier, a webhook body as well as a claimed secret provided by the caller, and attempts to validate the claim against the actual secret matching the provided identifier (if it was found). The following verification mechanisms are supported so far:

hmac-sha256: used by Github and Gitea to sign the JSON payload
token: used by Gitlab without signatures, so the body (payload) is ignored

The secrets are usually stored in your personal folder, but CLI validators support storing the actual secrets on a separate user account. This allows other users (such as www-data) to validate data against your secret without enabling them to read them in plaintext (zero-knowledge validation), as explained in the setup section.

The webhook validator uses the following syntax:

whck KIND IDENTIFIER CLAIM (+ BODY as STDIN), where:
KIND is a supported verification mechanism (hmac-sha256 or token)
IDENTIFIER is the repository unique ID, usually a base16 or base64 encoding of the repo URL
CLAIM is the claimed signature or secret token
BODY is the actual HTTP body (usually a JSON payload), passed via STDIN

Conventions

In order to facilitate deployment in varied setups, each command-line validator respects the following conventions:

returns 0 when validation succeeded, or any other exit code if it fails. In the future, error codes could be standardized
the secrets directory is generated from the following possibilities, in order of precedence:
- from the WHCK_DIR environment variable (automated tests)
- from the XDG_CONFIG_HOME environment variable, with "whck" appended (user operating CLI, or WWW running as user)
- from the HOME environment variable, with ".config/whck" appended (user operating CLI, or WWW running as user)
- from the current path to the executable, checking if it's inside a /home/ folder, and if so appending ".config/whck" (WWW running as system account, calling a symlink to a user-provided executable) -- TODO: maybe try ./secrets folder too from pwd?
- if all those discovery methods failed, the validator program fails to start and returns an error code
a bin/cli script builds the command-line validator, returning 0 upon success (or if the code doesn't require building) and outputting a complete path to the executable starting the program ; any other exit code indicates failure
a spec submodule contains this very repository (forge/endpoints)
a bin/test.sh script starts the tests: ensures the spec submodule is cloned, and starts the spec/test_cli.sh script, passing the full path to the executable (generated by bin/cli) as first argument

Testing

If you want to run tests for an existing implementation, please run the bin/test.sh script from the implementation's repository.

NOTE: Running tests requires the bats testing framework. You can usually find it in your distro repositories. If not, head over to the bats repository.

The test_cli.sh script is used for testing CLI webhook validators (such as whck). It takes as first positional argument a path to the (compiled) program to check. If no program is provided as argument, the implementation's bin/cli script will be executed to find a path to a (freshly-compiled) executable. The following paths will be attempted, in order of precedence:

bin/cli, relative to the directory from which you called the tests (eg. if you're running specs/test_cli.sh from whck repository)
../bin/cli, relative to the directory containing test_cli.sh (eg. if you're running ./test_web.sh from specs submodule in whck repository)
if no path was matched, the test suite fails to start

To run tests, simply call bin/test.sh from your implementation repository. If you call tests using specs/test_cli.sh directly, it will assume you either want to test a specific implementation (passed by argument), or that you want to test the implementation in the parent repository.

If you're making an implementation from scratch, you need to write your custom bin/cli script. Then, you can copy the test.sh script from whck and it should work.

Note: to test that the test suites can be called from several folders using different kind of relative/absolute paths, use the test_tests_cli.sh script.:

Sources

Name	Payload	Secret type	Secret location	Repo location
Gitea	JSON body	hmac-sha256 sig	header:X_GITEA_SIGNATURE	repository>html_url
Github	JSON body	hmac-sha256 sig	header:X_HUB_SIGNATURE	repository>html_url
Gitlab	JSON body	token	header:X_GITLAB_TOKEN	project->git_http_url

Setup

Now it gets tricky... In all cases you need some kind of shell account! Talk about sudo, suid, please etc.

Unprivileged user, web server running as user

Unprivileged user, web server running as system account

Privileged user (requires root)

TODO

reimplement calling forgehook-notify from endpoints.php
add test to check it's actually called
split READMEs separately for CLI and HTTP endpoint
maybe have all specs/tests/READMEs (with forgebuild) in a single repo?
make higher-level helper functions that can be shared across test files and implementation, for better consistency

6.4 KiB Raw Permalink Blame History