This commit started out investigating a `concurrent map read write` issue, ending by replacing the map with a struct.
This is easier to reason about, and it's more effective:
```
name old time/op new time/op delta
SiteNew/Regular_Deep_content_tree-16 71.5ms ± 3% 69.4ms ± 5% ~ (p=0.200 n=4+4)
name old alloc/op new alloc/op delta
SiteNew/Regular_Deep_content_tree-16 29.7MB ± 0% 27.9MB ± 0% -5.82% (p=0.029 n=4+4)
name old allocs/op new allocs/op delta
SiteNew/Regular_Deep_content_tree-16 313k ± 0% 303k ± 0% -3.35% (p=0.029 n=4+4)
```
See #8749
This change is mostly motivated to get a more stable CI build (we're building the Hugo site there, with Instagram and Twitter shortcodes sometimes failing).
Fixes#7866
Before a robots.txt is created in every Site. So in public/robots.txt if there are no languages (was correct). But if there are multiple languages in every language directory, too (was wrong). If defaultContentLanguageInSubdir is true, no language is created into the root directory, so no robots.txt is in the root directory (was wrong). If multihosts are configured for each language, that is the only case where one robots.txt must be created in each language directory (was correct).
I've changed the behaviour, that only in the multihost case the robots.txt is created in the language directories. In any other case it is created in public/robots.txt. I've also added tests that files are not created in the wrong directories.
Fixes#5160
See also #4193
The main use case for this is to use with resources.PostProcess and resources.PostCSS with purgecss.
You would normally set it up to extract keywords from your templates, doing it from the full /public takes forever for bigger sites.
Doing the template thing misses dynamically created class names etc., and it's hard/impossible to set up in when using themes.
You can enable this in your site config:
```toml
[build]
writeStats = true
```
It will then write a `hugo_stats.json` file to the project root as part of the build.
If you're only using this for the production build, you should consider putting it below `config/production`.
You can then set it up with PostCSS like this:
```js
const purgecss = require('@fullhuman/postcss-purgecss')({
content: [ './hugo_stats.json' ],
defaultExtractor: (content) => {
let els = JSON.parse(content).htmlElements;
return els.tags.concat(els.classes, els.ids);
}
});
module.exports = {
plugins: [
require('tailwindcss'),
require('autoprefixer'),
...(process.env.HUGO_ENVIRONMENT === 'production' ? [ purgecss ] : [])
]
};
```
Fixes#6999
This more or less completes the simplification of the template handling code in Hugo started in v0.62.
The main motivation was to fix a long lasting issue about a crash in HTML content files without front matter.
But this commit also comes with a big functional improvement.
As we now have moved the base template evaluation to the build stage we now use the same lookup rules for `baseof` as for `list` etc. type of templates.
This means that in this simple example you can have a `baseof` template for the `blog` section without having to duplicate the others:
```
layouts
├── _default
│ ├── baseof.html
│ ├── list.html
│ └── single.html
└── blog
└── baseof.html
```
Also, when simplifying code, you often get rid of some double work, as shown in the "site building" benchmarks below.
These benchmarks looks suspiciously good, but I have repeated the below with ca. the same result. Compared to master:
```
name old time/op new time/op delta
SiteNew/Bundle_with_image-16 13.1ms ± 1% 10.5ms ± 1% -19.34% (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16 13.0ms ± 0% 10.7ms ± 1% -18.05% (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16 46.4ms ± 2% 43.1ms ± 1% -7.15% (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16 52.2ms ± 2% 47.8ms ± 1% -8.30% (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16 77.9ms ± 1% 70.9ms ± 1% -9.01% (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16 43.0ms ± 0% 37.2ms ± 1% -13.54% (p=0.029 n=4+4)
SiteNew/Page_collections-16 58.2ms ± 1% 52.4ms ± 1% -9.95% (p=0.029 n=4+4)
name old alloc/op new alloc/op delta
SiteNew/Bundle_with_image-16 3.81MB ± 0% 2.22MB ± 0% -41.70% (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16 3.60MB ± 0% 2.01MB ± 0% -44.20% (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16 19.3MB ± 1% 14.1MB ± 0% -26.91% (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16 70.7MB ± 0% 69.0MB ± 0% -2.40% (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16 37.1MB ± 0% 31.2MB ± 0% -15.94% (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16 17.6MB ± 0% 10.6MB ± 0% -39.92% (p=0.029 n=4+4)
SiteNew/Page_collections-16 25.9MB ± 0% 21.2MB ± 0% -17.99% (p=0.029 n=4+4)
name old allocs/op new allocs/op delta
SiteNew/Bundle_with_image-16 52.3k ± 0% 26.1k ± 0% -50.18% (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16 52.3k ± 0% 26.1k ± 0% -50.16% (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16 336k ± 1% 269k ± 0% -19.90% (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16 422k ± 0% 395k ± 0% -6.43% (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16 401k ± 0% 313k ± 0% -21.79% (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16 247k ± 0% 143k ± 0% -42.17% (p=0.029 n=4+4)
SiteNew/Page_collections-16 282k ± 0% 207k ± 0% -26.55% (p=0.029 n=4+4)
```
Fixes#6716Fixes#6760Fixes#6768Fixes#6778
This commit also
* revises the change detection for templates used by content files in server mode.
* Adds a Page.RenderString method
Fixes#6545Fixes#4663Closes#6043
This is preparation for #6041.
For historic reasons, the code for bulding the section tree and the taxonomies were very much separate.
This works, but makes it hard to extend, maintain, and possibly not so fast as it could be.
This simplification also introduces 3 slightly breaking changes, which I suspect most people will be pleased about. See referenced issues:
This commit also switches the radix tree dependency to a mutable implementation: github.com/armon/go-radix.
Fixes#6154Fixes#6153Fixes#6152
This commit implements Hugo Modules.
This is a broad subject, but some keywords include:
* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`, `hugo mod get`, `hugo mod graph`, `hugo mod tidy`, and `hugo mod vendor`.
All of the above is backed by Go Modules.
Fixes#5973Fixes#5996Fixes#6010Fixes#5911Fixes#5940Fixes#6074Fixes#6082Fixes#6092
There have been reports about infrequent paginator crashes when running the Hugo server since 0.55.0.
The reason have been narrowed down to that of parallel rebuilds.
This isn't a new thing, but the changes in 0.55.0 made it extra important to serialize the page initialization.
This commit fixes that by protecting the `Build` method with a lock when running in server mode.
Fixes#5885Fixes#5968
The main motivation of this commit is to add a `page.Page` interface to replace the very file-oriented `hugolib.Page` struct.
This is all a preparation step for issue #5074, "pages from other data sources".
But this also fixes a set of annoying limitations, especially related to custom output formats, and shortcodes.
Most notable changes:
* The inner content of shortcodes using the `{{%` as the outer-most delimiter will now be sent to the content renderer, e.g. Blackfriday.
This means that any markdown will partake in the global ToC and footnote context etc.
* The Custom Output formats are now "fully virtualized". This removes many of the current limitations.
* The taxonomy list type now has a reference to the `Page` object.
This improves the taxonomy template `.Title` situation and make common template constructs much simpler.
See #5074Fixes#5763Fixes#5758Fixes#5090Fixes#5204Fixes#4695Fixes#5607Fixes#5707Fixes#5719Fixes#3113Fixes#5706Fixes#5767Fixes#5723Fixes#5769Fixes#5770Fixes#5771Fixes#5759Fixes#5776Fixes#5777Fixes#5778
This commit adds support for a configuration directory (default `config`). The different pieces in this puzzle are:
* A new `--environment` (or `-e`) flag. This can also be set with the `HUGO_ENVIRONMENT` OS environment variable. The value for `environment` defaults to `production` when running `hugo` and `development` when running `hugo server`. You can set it to any value you want (e.g. `hugo server -e "Sensible Environment"`), but as it is used to load configuration from the file system, the letter case may be important. You can get this value in your templates with `{{ hugo.Environment }}`.
* A new `--configDir` flag (defaults to `config` below your project). This can also be set with `HUGO_CONFIGDIR` OS environment variable.
If the `configDir` exists, the configuration files will be read and merged on top of each other from left to right; the right-most value will win on duplicates.
Given the example tree below:
If `environment` is `production`, the left-most `config.toml` would be the one directly below the project (this can now be omitted if you want), and then `_default/config.toml` and finally `production/config.toml`. And since these will be merged, you can just provide the environment specific configuration setting in you production config, e.g. `enableGitInfo = true`. The order within the directories will be lexical (`config.toml` and then `params.toml`).
```bash
config
├── _default
│ ├── config.toml
│ ├── languages.toml
│ ├── menus
│ │ ├── menus.en.toml
│ │ └── menus.zh.toml
│ └── params.toml
├── development
│ └── params.toml
└── production
├── config.toml
└── params.toml
```
Some configuration maps support the language code in the filename (e.g. `menus.en.toml`): `menus` (`menu` also works) and `params`.
Also note that the only folders with "a meaning" in the above listing is the top level directories below `config`. The `menus` sub folder is just added for better organization.
We use `TOML` in the example above, but Hugo also supports `JSON` and `YAML` as configuration formats. These can be mixed.
Fixes#5422
We do that by re-render visited pages that is not already in the stack. This may potentially do some double work, but that small penalty should be well worth it.
Fixes#5281
The main item in this commit is showing of errors with a file context when running `hugo server`.
This can be turned off: `hugo server --disableBrowserError` (can also be set in `config.toml`).
But to get there, the error handling in Hugo needed a revision. There are some items left TODO for commits soon to follow, most notable errors in content and config files.
Fixes#5284Fixes#5290
See #5325
See #5324
There have been one report of a site with truncated `.Content` after the Hugo `0.40.1` release.
This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats.
Fixes#4706
This is a follow-up to #4632. There were some assumptions in that implementation that did not hold water in all situations.
This commit simplifies the content lazy initalization making it more robust.
Fixes#4664
This resolves some surprising behaviour when reading other pages' content from shortcodes. Before this commit, that behaviour was undefined. Note that this has never been an issue from regular templates.
It will still not be possible to get **the current shortcode's page's rendered content**. That would have impressed Einstein.
The new and well defined rules are:
* `.Page.Content` from a shortcode will be empty. The related `.Page.Truncated` `.Page.Summary`, `.Page.WordCount`, `.Page.ReadingTime`, `.Page.Plain` and `.Page.PlainWords` will also have empty values.
* For _other pages_ (retrieved via `.Page.Site.GetPage`, `.Site.Pages` etc.) the `.Content` is there to use as you please as long as you don't have infinite content recursion in your shortcode/content setup. See below.
* `.Page.TableOfContents` is good to go (but does not support shortcodes in headlines; this is unchanged)
If you get into a situation of infinite recursion, the `.Content` will be empty. Run `hugo -v` for more information.
Fixes#4632Fixes#4653Fixes#4655
For the content from other pages in shortcodes there are some chicken and
egg dependencies that is hard to get around. But we can improve on this by preparing the pages in a certain order:
1. The headless bundles goes first. These are page typically page and image collections..
2. Leaf bundles
3. Regular single pages
4. Branch bundles
Fixes#4632
As an example:
```html
{{ $pages := .Site.RegularPages | lang.Merge $frSite.RegularPages | lang.Merge $enSite.RegularPages }}
```
Will "fill in the gaps" in the current site with, from left to right, content from the French site, and lastly the English.
Fixes#4463
This commit adds support for `headless bundles` for the `index` bundle type.
So:
```toml
headless = true
```
In front matter means that
* It will have no `Permalink` and no rendered HTML in /public
* It will not be part of `.Site.RegularPages` etc.
But you can get it by:
* `.Site.GetPage ...`
The use cases are many:
* Shared media galleries
* Reusable page content "snippets"
* ...
Fixes#4311
This commit is not the smallest in Hugo's history.
Some hightlights include:
* Page bundles (for complete articles, keeping images and content together etc.).
* Bundled images can be processed in as many versions/sizes as you need with the three methods `Resize`, `Fill` and `Fit`.
* Processed images are cached inside `resources/_gen/images` (default) in your project.
* Symbolic links (both files and dirs) are now allowed anywhere inside /content
* A new table based build summary
* The "Total in nn ms" now reports the total including the handling of the files inside /static. So if it now reports more than you're used to, it is just **more real** and probably faster than before (see below).
A site building benchmark run compared to `v0.31.1` shows that this should be slightly faster and use less memory:
```bash
▶ ./benchSite.sh "TOML,num_langs=.*,num_root_sections=5,num_pages=(500|1000),tags_per_page=5,shortcodes,render"
benchmark old ns/op new ns/op delta
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 101785785 78067944 -23.30%
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 185481057 149159919 -19.58%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 103149918 85679409 -16.94%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 203515478 169208775 -16.86%
benchmark old allocs new allocs delta
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 532464 391539 -26.47%
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 1056549 772702 -26.87%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 555974 406630 -26.86%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 1086545 789922 -27.30%
benchmark old bytes new bytes delta
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 53243246 43598155 -18.12%
BenchmarkSiteBuilding/TOML,num_langs=1,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 105811617 86087116 -18.64%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=500,tags_per_page=5,shortcodes,render-4 54558852 44545097 -18.35%
BenchmarkSiteBuilding/TOML,num_langs=3,num_root_sections=5,num_pages=1000,tags_per_page=5,shortcodes,render-4 106903858 86978413 -18.64%
```
Fixes#3651Closes#3158Fixes#1014Closes#2021Fixes#1240
Updates #3757
Hugo already, in its server mode, support partial rebuilds. To put it simply: If you change `about.md`, only that content page is read and processed, then Hugo does some processing (taxonomies etc.) and the full site is rendered.
This commit covers the rendering part: We now only re-render the pages you work on, i.e. the last n pages you watched in the browser (which obviously also includes the page in the example above).
To be more specific: When you are running the hugo server in watch (aka. livereload) mode, and change a template or a content file, then we do a partial re-rendering of the following:
* The current content page (if it is a content change)
* The home page
* Up to the last 10 pages you visited on the site.
This should in most cases be enough, but if you navigate to something completely different, you may see stale content. Doing an edit will then refresh that page.
Note that this feature is enabled by default. To turn it off, run `hugo server --disableFastRender`.
Fixes#3962
See #1643
This commit allows shortcode per output format, a typical use case would be the special AMP media tags.
Note that this will only re-render the "overridden" shortcodes and only in pages where these are used, so performance in the normal case should not suffer.
Closes#3220
This is a pretty fundamental change in Hugo, but absolutely needed if we should have any hope of getting "multiple outputs" done.
This commit's goal is to say:
* Every file target path is created by `createTargetPath`, i.e. one function for all.
* That function takes every page and site parameter into account, to avoid fragile string parsing to uglify etc. later on.
* The path creation logic has full test coverage.
* All permalinks, paginator URLs etc. are then built on top of that same logic.
Fixes#1252Fixes#2110Closes#2374Fixes#1885Fixes#3102Fixes#3179Fixes#1641Fixes#1989
Bjørn Erik Pedersen
Nico
Cameron Moore