From fd85af8e3704487083ec48b7cb2ba85abec03320 Mon Sep 17 00:00:00 2001 From: Ben Harris Date: Mon, 23 Sep 2019 21:46:22 -0400 Subject: [PATCH] add make and pandoc to issue 2 --- issues/2/make.md | 136 +++++++++++++++++++++++++++++++++++++++++++++ issues/2/pandoc.md | 135 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 issues/2/make.md create mode 100644 issues/2/pandoc.md diff --git a/issues/2/make.md b/issues/2/make.md new file mode 100644 index 0000000..6c267c5 --- /dev/null +++ b/issues/2/make.md @@ -0,0 +1,136 @@ +# makefile magic + +author: [ben](https://tilde.team/~ben/) + +i've used makefiles plenty of times for a variety of projects, but my +understanding has not really improved beyond the most basic level. + +working with pandoc for the [\~club wiki](https://tilde.club/wiki/) and +the [zine](https://zine.tildeverse.org/) inspired me to replace a [janky +build +script](https://github.com/tildeclub/site/blob/25e3ed375791d48d085b434beabac10f584f5540/wiki/build-wiki.sh) +with some makefile goodness. + +feeling a loss for where to start, i reached out to +[tomasino](gopher://tilde.black) for some guidance. he responded that i +should have a look at a makefile he built for a javascript project +template which he linked on [his +github](https://github.com/jamestomasino/ts-boilerplate/blob/master/Makefile). + +let's go through some of my biggest takeaways from inspecting tomasino's +example and creating several of my own makefiles from it. + +## anatomy of a makefile + +this was something that i thought i understood, but realized i didn't +fully grok. + +i *think* that i understand it now, or at least know now what to search +for when i get stuck the next time. + +let's start with the basics: a makefile should be named Makefile (note +the capital M). the basic syntax consists of a target followed by a +colon (`:`) and an optional list of requirements, then a list of +commands to be run (indented by literal tabs). + + myproc: source.c + cc -o myproc source.c + +this example defines how to build myproc, which depends on source.c. + +### variables + +makefiles use variables in two main formats. + +- recursively-expanded: evaluated every time they're referenced, even + in other variables + - defined with a single `=` +- simply-expanded: evaluated once at the time of definition + - defined with `:=` + +simply-expanded variables are generally more predictable and will likely +be what you should use in most cases. + + x := foo + y := $(x) bar + x := later + +will become + + y := foo bar + x := later + +variable names are case-sensitive. you'll likely see all-caps variable +names more often than not. + +### guided example + +this was the major revelation for me coming from my brittle buildscript +that deleted and recreated everything on each run. + +make wants to know what files you want built, as well as how to build +them. + +let's take the \~club wiki as an example: we want to build one html for +each markdown file in the source directory. how do we tell make which +files to build and how to build each one? + +using a standard static target name, you'd need to add a target for each +markdown file. + + ssh.html: source/ssh.md + pandoc -so ssh.html source/ssh.md + + git.html: source/git.md + pandoc -so git.html source/git.md + +this already feels like a lot of duplication. let's stay +[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself). here's an +example from tomasino's makefile that i adapted for the wiki: [github +source](https://github.com/jamestomasino/ts-boilerplate/blob/master/Makefile#L46) + + $(DST_DIR)/js/%.js: $(SRC_DIR)/ts/%.ts + $(mkdir) + $(tsc) $< --outFile $@ + +in this example, we're telling make that files under `DST_DIR` are built +from `SRC_DIR` with a command defined in the `tsc` variable. + +let's adapt this to our wiki example: + + %.html: source/%.md + pandoc -so $@ $< + +here we're telling make that it can build html files from the +corresponding md file in source by running pandoc. but how do we tell +make which html files we want to build? let's go back to tomasino's +[example](https://github.com/jamestomasino/ts-boilerplate/blob/master/Makefile#L18-L19) + + SRC_TS_FILES != find $(SRC_DIR)/ts -name '*.ts' + DST_JS_FILES := $(SRC_TS_FILES:$(SRC_DIR)/ts/%.ts=$(DST_DIR)/js/%.js) + +in this case, we're using the special assignment form with `!=` that +uses the output of a shell executable. we now have a list all the `.ts` +files that need to be compiled in `SRC_TS_FILES`. next, we assign +`DST_JS_FILES` with the special replacement syntax, which changes +`ts/%.ts` into `js/%.js`. so, these variables now contain a list of +source and desired files, along with a definition of how to build those +files. + +here's how i adapted this step for the wiki: + + SRC_MD_FILES != find source -name '*.md' + DST_HTML_FILES := $(SRC_MD_FILES:source/%.md=%.html) + +this means that make now has a list of html files that are required +based on the list of all markdown files in the source directory. + +now all that's left is to tell make that we want to build all of the +`DST_HTML_FILES`. let's add that list to the `all` target: + + all: $(DST_HTML_FILES) + +now we can generate all html files by simply calling `make` + +here's the completed +[Makefile](https://github.com/tildeclub/site/blob/master/wiki/Makefile). diff --git a/issues/2/pandoc.md b/issues/2/pandoc.md new file mode 100644 index 0000000..c5f80bf --- /dev/null +++ b/issues/2/pandoc.md @@ -0,0 +1,135 @@ +# pandoc magic + +author: [ben](https://tilde.team/~ben/) + +[pandoc](https://pandoc.org) is an incredibly powerful tool for creating +and converting documents. + +i recently started using it for all kinds of different things, +including: + +- the [tilde.club wiki](https://tilde.club/wiki/) +- my personal [page on \~club](https://tilde.club/~benharri/) (source + [here](https://tildegit.org/ben/club-site)) +- this zine itself (see [zine + makefile](https://tildegit.org/tildeverse/zine/src/branch/master/Makefile)) + +let's look at some of the tricks and tips that i've learned from these. + +## format markdown files + +this seems like it might not work, but `pandoc` will format your +markdown files for you if you convert the source file from markdown *to* +markdown again. for example, if you wanted to tidy up your `README.md`, +you could run: + +> `pandoc -f markdown -t markdown -o README.md README.md` + +note that if you want to preserve any yaml frontmatter blocks, you'll +need to change the from and to types to `markdown+yaml_metadata_block`. +another thing to note is that you can preserve github-formatted markdown +by using `gfm` instead of `markdown`. additionally, if you prefer to use +atx-style headers, don't forget to add the `--atx-headers` switch. + +a complete example: + + pandoc \ + -f markdown+yaml_metadata_block \ + -t markdown+yaml_metadata_block \ + --atx-headers \ + -o README.md \ + README.md + +## templates + +pandoc ships with a full set of templates that are used to control how +things are displayed when writing to certain formats. + +let's look at the templates for html and latex. to get the default +template for a given format, use `-D`. the template is written to +stdout, so let's save it to a file. + +> `pandoc -D html > html.template` + +open up this template in your editor and you'll see what's used when you +convert things to html. + +important note: these templates are only used when generating standalone +documents (the `-s` or `--standalone` flag). + +note that the template uses lots of variable substitutions. we can set +values for these in yaml frontmatter or in the command invocation. + +to make some basic customizations, we can fill in the metadata values +that will be automatically replaced in the document when we build. + +however, some circumstances require a custom layout or additional +changes to meet the requirements. + +have a look at the +[wiki.tmpl](https://github.com/tildeclub/site/blob/master/wiki/wiki.tmpl) +used on the [\~club wiki](https://tilde.club/wiki/). some of the notable +changes include: + +- table of contents title +- hardcoded stylesheet +- link to author's user page + +the stylesheet change would be possible on the command line by using the +`-c` or `--css` option, but the other changes require a custom template. + +try it out on your own project! + +## lua filters + +sometimes the rendered page still isn't what you're looking for. maybe +you need some additional tweaks (css classes, extra html items, etc). + +pandoc has supported filters for a long time in the form of json passed +around on pipes that allows you to modify the internal data structures +before they're written to the output format. + +the main downside of using these filters is that it introduces another +layer of dependencies (namely the language that the filter's written in +and also the library for that language to interact with pandoc's json). + +as of pandoc 2.0, a lua interpreter with a class library for creating +filters is built in to the pandoc executable. let's go over some basic +examples. + +### permalinks for headers + +a common feature on most html created via markdown is a small link +displayed next to the header so that you can deep-link directly to that +section of the page. this is frequently seen on github and other +documentation sites. + +here's a very simple way to create the link: + + function Header(elem) + table.insert(elem.content, pandoc.Space()) + table.insert(elem.content, pandoc.Link("ยง", "#" .. elem.identifier)) + return elem + end + +save this to a file and call it in your pandoc conversion with the +`--lua-filter` option. + +### change header levels + +this is an example from this zine. i wanted to decrease the size of +headers below h1 level without changing the external css. here's how i +did it. + + function Header(elem) + if elem.level > 1 then + elem.level = elem.level + 1 + end + return elem + end + +this changes all header elements to be one level smaller. + +there are additional examples on +[pandoc.org](https://pandoc.org/lua-filters.html) if you'd like to see +more.