Lightweight, KISS system to rebuild some git projects when they change.
Go to file
southerntofu a98e43a5c4 hg: Don't forget to update working dir to reflect update 2022-01-07 14:55:16 +01:00
.git-build Removed old examples 2020-04-29 21:48:03 +02:00
spec@dc256b15d4 Update spec submodule 2021-01-04 16:25:06 +01:00
.gitignore Initial commit 2020-04-17 21:51:03 -04:00
.gitmodules Move translation files to submodule 2020-09-13 20:23:35 +02:00
LICENSE Initial commit 2020-04-17 21:51:03 -04:00
README.md Allowlist for task can take multiple hosts 2020-09-16 12:07:03 +02:00
forgebuild.sh hg: Don't forget to update working dir to reflect update 2022-01-07 14:55:16 +01:00
setup.sh Ensure translations are updated 2020-12-01 18:18:06 +01:00

README.md

forgebuild.sh

Lightweight, KISS system to start tasks that depend on a remote git project. Depending on your needs, forgebuild can:

  • check updates for one or more repositories, and start corresponding tasks if there was any update (forgebuild [tasks])
  • run one or more local tasks, whether there were updates or not (forgebuild -f [tasks])
  • use task settings, or for host-specific settings

Setup

To automatically setup forgebuild.sh, you can run the setup.sh script, like so:

$ git clone https://tildegit.org/forge/build.sh
$ build.sh/setup.sh

The setup script is very short and simple so you may take the time to inspect it. But if you'd rather do things your way, you should:

  • copy forgebuild.sh as forgebuild somewhere on your $PATH (sudo cp forgebuild.sh /usr/local/bin/forgebuild)
  • clone the git submodule with the translation files (git submodule init && git submodule update)
  • copy the translations to $HOME/.local/share/forgebuild (mkdir $HOME/.local/share/forgebuild/ && cp -R spec/i18n $HOME/.local/share/forgebuild/)

That's it! You can now start using forgebuild to build all your amazing things.

TODO: should also check /usr/local/share

If you want to do this your way, you need to:

Usage

forgebuild looks up your ~/.forgebuild/ folder to find tasks to run. A task t is defined by a t.source file containing a git remote, and a t executable file that will be run on updates to the corresponding repository.

A task may also have the following optional files:

  • t.branch: checkout the specified branch on the repository
  • t.hosts: only run the task on a newline-separated list of hosts, see multihost setup

TLDR: a task t needs t (executable) and t.source, and accepts t.branch and t.hosts

Running tasks

Simply running forgebuild.sh will update task repositories, and run tasks which received updates. When positional arguments are passed to forgebuild, only the corresponding tasks are triggered. For example, forgebuild foo bar will only check updates for tasks foo and bar, and run them if updates were found.

Additionally, forgebuild.sh can take a -f or --force flag which triggers task runs even when no updates were found. This flag is useful when you want to trigger a task which failed because of side-effects outside of the reach of forgebuild, such as a missing package on the system. Similarly to without the -f flag, you can give forgebuild a specific list of tasks to trigger.

Note: Please remember to make your task scripts executable! chmod +x is your friend.

Logging

Task output is stored in t.log for STDOUT, and t.err for STDERR. These files are removed on a task run, so that they always contain information about the last build.

If your task respects STDOUT/STDERR convension, you can simply check the size of t.err to figure out whether a task failed. If some people would want that, we could further track task progress/errors and/or introduce configurable loglevels.

Configuration

Some tasks may need additional settings in order to run. A website building task may wish to know about a base URL and a destination folder, for instance. Such information may be placed in the ~/.forgebuild/config/ folder.

Any task receives the $GITBUILDCONF environment variable pointing to this repository, unless specific host configuration is found, as will be explained in the next section.

Multihost setup

The distinctive feature of forgebuild.sh is the host-based configuration system. If a folder matching $HOSTNAME is found in your ~/.forgebuild/, this folder will be passed to the task as $GITBUILDCONF, instead of the default configuration folder. If no host-specific configuration is found, the default configuration is used as explained previously.

Additionally, tasks can be configured to always/never run on a specific host. To run a task t on a defined list of hosts, place a t.hosts file containing a hostname per line in ~/.forgebuild/. To ignore a task on a specific host, place an empty t.ignore file in ~/.forgebuild/$HOSTNAME/.

For an example of a multi-host setup, you can check out my own ~/.forgebuild.

Building your own task

To build your own task, you simply need an executable program. If your task executable is a script, don't forget to add the shebang (eg #! /bin/bash) at the top of your script or it will fail. Please ensure your task is executable (chmod +x t) or it will fail.

A task can be any program in any language, that takes exactly one positional argument containing the task name, and the following environment variables:

  • $GITBUILDCONF containing the current configuration directory, which is either ~/.forgebuild/$HOSTNAME or ~/.forgebuild/config (in this order)
  • $GITBUILDDIR which is always ~/.forgebuild/

Here's an example simple task that writes the current time to ~/.forgebuild/t.log and appends random characters to another file:

#! /bin/bash

date # STDOUT is automatically appended to task log

hexdump -n 16 -e ' /1 "%02x"' /dev/urandom >> ~/random.list

Note: The task name is passed to the task because the task may be a symlink to a program located somewhere else on the system. This is further described in the Shared build scripts section.

Advanced usage

Shared build scripts

If you need to use the same task script for different tasks, you can use symbolic links:

$ ln -s ~/.forgebuild/task1 ~/.forgebuild/task2

This is possible because the task receives the current task name as first (and only) positional argument ($1). In the above example, the following commands are run when running forgebuild.sh:

  • ~/.forgebuild/task1 task1 # when task1.source was updated
  • ~/.forgebuild/task1 task2 # when task2.source was updated

This allows for instance to build different websites to different folders, from the same task script. A complex example for that is available in the zola script in my ~/.forgebuild.

Ordering tasks

forgebuild.sh supports naive ordering of tasks because it uses a glob pattern (~/.forgebuild/*.source) to iterate over files with alphanumeric ordering. This means task abcd runs before dcba, but 10-dcba runs before 20-abcd.

However, tasks can take some time and are therefore run in the background (in parallel). So if you need to run a task precisely after another one, the first task should create a lockfile, which it removes once done. Then, the second task waits for the first to finish its job.

For example, this is how the autoupdater waits for forgebuild.sh to stop running:

# Wait for forgebuild.sh to stop running
SECONDS=0
while [[ $SECONDS < 60 ]]; do
	if [ ! -f $GITBUILDDIR/.LOCK ]; then
		# We're done waiting, the road is free!
	fi
	sleep 1
	SECONDS=$(( SECONDS + 1 ))
done

exit 1 # TIMEOUT

Versioning

Of course, you can version your ~/.forgebuild/ in git. This allows you to easily share the same folder across serves and keep them in sync. You can use the following task script to auto-update your folder:

#! /bin/bash

cd ..
git pull

Note: You should probably not do something so simple, as other tasks may be running which would make the pull fail, as those files are busy and git cannot write to them. In the future, a more integrated approach to this problem may be taken.

When keeping your ~/.forgebuild/ in git, you need need to tell git to ignore local artifacts, by appending the following to your .gitignore:

*.log
*.err
.*/

You can check out my own ~/.forgebuild for an example of how to version your build scripts.

Auto-run tasks

Auto-triggering updates/builds is beyond the scope of this project! You should consider using standard cronjobs for that.

I'm also interested in running this script through a webhook endpoint for Github/Gitlab/Gitea webhooks, in order to build a simple and reliable CI/CD system. This is not implemented yet and will not be part of this repository, but i will update this README when a such solution is available. If you want to contribute this feature in the meantime, you are more than welcome!

Auto-update

Yes, you read that correctly! The .forgebuild/forgebuild{,.source} task provided in this repository is a proof-of-concept autoupdater that will download and install forgebuild updates on every run.

It's a terrible idea for security to autoupdate executables in your $PATH and you probably do not want to do that. But i thought it was a nice demo and it can be useful for people new to the command-line, although this project should not receive too many updates in the future as it is already doing what it is supposed to.

Contribute

Right now the state of this project is "works for me". It could be useful for other people which is why i just spent a full hour writing this README. However, there could be ways in which forgebuild.sh fails you!

If a feature is missing for you, or you encounter a bug, please report it on tildegit (requires an account on a tildeverse server) or send a mail to southerntofu (@) thunix.net. If you can contribute the feature or fix the bug yourself, your patches are welcome!

This project abides by the ~fr operating principles.