Confgen: Solving Config Files

If you've known me for any amount of time, you'll probably have witnessed me bragging about this, only to be disappointed when I ultimately had to answer that it's kind of a steep learning curve. This is what I'm trying to solve here by giving you an introduction as gentle as I'm able to make it to Confgen, the last template engine you will ever need.

Despite its namesake, Confgen is good at far more than generating config files. It ultimately evolved into a universal template engine for generating... anything you could think of.

What even is this?

Confgen is a template engine. If you've never heard that term before, it's basically a tool that takes in a file with placeholders in it, and then inserts some data in place of them. This might sound unspectacular at first, but that's also a gross oversimplification. Confgen, for one, doesn't only let you have placeholders (expression blocks), but also lets you have control flow (statement blocks). fors, ifs and functions are all fair game. What makes Confgen even more special is that, unlike most template engines, it doesn't implement all these daunting concepts itself, but instead harnesses the full power of the Lua programming language (basic knowledge of which will be considered a prerequisite for the rest of this article, but it's an easy language to learn!).

If you're still not convinced, the page you are reading right now is, in fact, generated by Confgen, from Markdown. Syntax highlighting and all.

Markdown, for those uninitiated, is a very easy to read markup language, which is fun to write in! Think of it as HTML in easy mode.

This is roughly what the source of this page looks:

## Header here!
This paragraph contains a **bold** statement.

> I'm a quote!

..and so on.

All that is turned into HTML by Confgen, so your browser can show it while I can still keep the last remaining bit of my sanity while writing. So now that you're convinced, let's get you started!

Note: Bring some form of brain coolant :D

Installation

Please refer to the Confgen repository README for instructions. If you're on NixOS, a simple nix shell git+https://git.mzte.de/LordMZTE/confgen will suffice. You can also find Linux binaries built on a Debian CI on the releases tab, but those may be outdated. On other platforms (except Windows, which it wasn't worth having backwards-compatibility for), you'll need to build from source using a Zig compiler.

Your first Confgenfile

Now that you've got Confgen ready to roll, let's experiment with it! You can do this in a blank directory, your dotfiles, or, if you're feeling adventurous, your website's source code.

Confgen is centered around the so-called Confgenfile, commonly named confgen.lua. It describes the layout of the project, declares common values, functions and tells confgen what files to process. Assuming that you want to generate your configuration files, this is what it might look like.

Note: This file would be placed in your dotfiles source, not your home directory. The .config here is not ~/.config!

-- confgen.lua

cg.addPath(".config")

cg.opt = {
    font = "MyFavoriteFont",
    font_size = "11",
}

Alright, now what does this actually mean? Let's start with the basic order of operations Confgen works in.

Confgen starts off by evaluating the Confgenfile, like a normal Lua program. Here, you may declare functions, options by adding to the cg.opt table, or execute any arbitrary Lua code, for example to load machine-local options.
During this evaluation, Confgen builds a list of input files. As the user, it's your job to add to this list. Here, cg.addPath will recursively add all files inside the given directory to the list. There are more functions available, such as cg.addFile. For a full documentation of the API, please refer to man 3 confgen.
Then, Confgen will iterate through the file list built during the evaluation phase. Each file ending in .cgt (Confgen Template) will be evaluated as a template (more on that in a second), while other files will be copied to the output directory.

The first template (of many)

If you now run confgen confgen.lua output (specifying your Confgenfile and an output directory), you will end up with a disappointingly empty directory. Let's fix that by creating a configuration file template, in this case for a hypothetical terminal emulator called coolterm.

# .config/coolterm/config.ini.cgt

font = "<% opt.font %>:<% opt.font_size %>"

[keybinds]
zoom-in = "ctrl+plus"
zoom-out = "ctrl+minus"

First, note the .cgt we added to that filename (since coolterm expects a file called ~/.config/coolterm/config.ini). This is how Confgen knows that it should not just copy the file. Confgen will strip this second extension during evaluation.

If we now run confgen confgen.lua output again, we'll find a fully generated configuration file:

# output/.config/coolterm/config.ini

font = "MyFavoriteFont:11"

[keybinds]
zoom-in = "ctrl+plus"
zoom-out = "ctrl+minus"

Now, this might seem like a massively overcomplicated way of declaring a font in a terminal configuration, but hear me out. Let's say we also want to configure GTK (or anything else that wants to know a font). We can simply add this to our files:

# .config/gtk-3.0/settings.ini.cgt

[Settings]
gtk-font-name=<% opt.font %> <% opt.font_size %>

We've just deduplicated our font name and size! Whereas previously, if you changed your mind about your favorite font, you'd had to have to change two files, you now only need to adjust it in your confgen.lua! Sure, sounds unspectacular at first, after all, why go to all of this trouble for a slight gain in convenience in a very rare situation, but just look at my (admittedly mind-bogglingly gigantic) config files:

$ rg --hidden 'opt.font' | wc -l
30

Not so insignificant anymore, is it? And this is also far from all Confgen has to offer!

But wait, there's more (way more)!

Say that you'd like to use Waybar, but you're also not sure if you want to use Hyprland or River as your wayland compositor of choice, but you want workspaces and the title of the focused window in your bar no matter, which one you're using. Let's create a config!

// .config/waybar/config.jsonc.cgt

{
    "modules-left": [
        <! if opt.compositor == "river" then !>
        "river/tags",
        <! elseif opt.compositor == "hyprland" then !>
        "hyprland/workspaces",
        <! end !>

        "<% opt.compositor %>/window"
    ],
    // ...
}

Now, it's just a matter of adding a cg.opt.compositor = "X" to your Confgenfile and rebuilding! And of course, this is not limited to just ifs! fors as well as anything else Lua allows is possible! Sky's the limit!

Note: Seems inconvenient to rebuild your config every time you switch compositors, doesn't it? This problem is solved through ConfgenFS, where the config file is able to automagically change each time either compositor starts. You can even have non-deterministic config files (sorry for destroying your hopes and dreams, dear NixOS user reading). Article coming soon!

This is still not optimal, though. After all, it's JSON and that's inconvenient to work with. Wouldn't it be nice if this were Lua too? Well, this is where post-processors come into play!

-- .config/waybar/config.jsonc.cgt
<! tmpl:setPostProcessor(function(prev)
    local value = loadstring(prev)() -- Load and evaluate the Lua code this template generates.
    return cg.fmt.json.serialize(value) -- Serialize it into JSON using Confgen's builtin serializer.
end) !>

return {
    ["modules-left"] = {
        <! if opt.compositor == "river" then !>
        "river/tags",
        <! elseif opt.compositor == "hyprland" then !>
        "hyprland/workspaces",
        <! end !>

        "<% opt.compositor %>/window"
    },
    -- ...
}

While this snippet will probably prevent any experienced Lua programmer from sleeping well for a week, it's still kinda cool, don't you agree? Since this might be a little hard to digest, I'll break it down.

Templates (referenced through the tmpl global variable inside the template's code) may have a post-processor. A post-processor is nothing more than a function that will be called by Confgen once the template itself is already done generating. The function will be passed the result of the template and return a modified version. In this case, the content before the post-processor is the Lua code under the "header", and the modified version is the JSON value Waybar will see. This post-processor is set by calling the setPostProcessor function on the template. In this case, it has a rather simple implementation.

And this is precisely how the source code of this article has been converted from Markdown to HTML. It's a post-processor, albeit one with a far more complicated implementation.

How is this sorcery possible?!

As you write more complex templates, it's always important to remember the gist of how they work under the hood. While a Lua runtime certainly is happy evaluating single statements as those you might know from <% ... %> blocks, it definetely won't swallow partial statements and unclosed delimiters such as those you'll often have in <! ... !> blocks. The trick Confgen uses to get around this is to compile the template to Lua, which is then evaluated in one go per template. For debugging your templates, Confgen provides a command to compile a template to this intermediate Lua, confgen --compile. Let's try it on our waybar config by invoking confgen -c .config/waybar/config.jsonc.cgt (cleaned up a bit for ease of reading):

tmpl:pushLitIdx(tmplcode, 0)

tmpl:setPostProcessor(function(prev)
    local value = loadstring(prev)() -- Load and evaluate the Lua code this template generates.
    return cg.fmt.json.serialize(value) -- Serialize it into JSON using Confgen's builtin serializer.
end)

tmpl:pushLitIdx(tmplcode, 1)

if opt.compositor == "river" then
    tmpl:pushLitIdx(tmplcode, 2)
elseif opt.compositor == "hyprland" then
    tmpl:pushLitIdx(tmplcode, 3)
end

tmpl:pushLitIdx(tmplcode, 4)

tmpl:pushValue(opt.compositor)

tmpl:pushLitIdx(tmplcode, 5)

This should look familiar. All our <! ... !> blocks have been inserted here verbatim, like our post-processor! This is what makes arbitrary control flow possible. Contrarily, our <% ... %> blocks have been replaced with tmpl:pushValue(...). pushValue is simply a mostly invisible function from Confgen that turns the given value to a string and appends it to the output. Something else that should be jumping out here are all those calls to pushLitIdx (push literal (at) index). In short, Confgen uses this number to look up a span from your template's source code, stored in the opaque tmplcode object to append to the output.

tmplcode is a separate object from tmpl, which is required when templates either don't have their own source or use the code from another template file. You'll (possibly) understand in a second.

Subtemplates

That's right, as if this wasn't complicated enough already, you can nest them.

"We were so preoccupied with asking if we could, that we didn't stop to think if we should."

The reason for the existance of subtemplates is only hard, and not impossible to justify. Let me try.

Say you had an HTML file you want to generate. It contains some boring enclosing tags on the outside, and a huge chunk of text in the middle that also contained some template expressions, that you'd love to write in Markdown because HTML is just a pain to work with after all. The Markdown to HTML Lua function is also already written (or pulled in from an external Lua library because that does in fact work) and ready to rock, but if you just use that as a post-processor, all your surrounding HTML breaks. Whatever shall you do? Subtemplates to the rescue! With subtemplates, you can run a region of your template file through a separate post-processor, or even get the resulting, templated string value returned to your outer template's Lua code. Here's an example:

<! -- my_page.html.cgt
   -- This is a comment in Confgen, by the way. You can probably guess how it works. !>
<html>
<head>
    ...
</head>

<body>
<! tmpl:pushSubtmpl(function(tmpl) tmpl:setPostProcessor(opt.myMarkdownRenderer) !>
# Welcome to my awesome website!

May I introduce you to *Confgen*, this absolutely amazing template engine I learnt about recently!
Also, two times two is <% 2 * 2 %>.
<! end) !>
</body>
</html>

"Alright, now he's gone completely nuts" I hear you say. First, you are probably completely correct, and second, if you don't understand this on a syntactical level, please re-read the previous section (repeat this procedure until you have internalized how expression blocks and statement blocks work :P).

A subtemplate is nothing but another instance of a template (tmpl) that is passed to the function we wrote here. This changes all the generated statements inside our function to refer to that new, inner template in place of the outer one. This inner template is then fully evaluated by Confgen, and its output is pushed to the outer template's output.

Note: If you want the result of the inner template returned as a string rather than have it pushed to the outer template, use tmpl:subtmpl instead of tmpl:pushSubtmpl.

Library Templates

What if you wanted to share a snippet between templates, and didn't want to write it all in Lua? Well, Confgen's got you covered, as always! The opt (or cg.opt table as it's called in the Confgenfile) is always writable. You just shouldn't modify it in regular template file, as the order they're evaluated in is unspecified. However, you can use Confgen's powerful API to evaluate a template during the Confgenfile evaluation phase. Let's adjust our Confgenfile accordingly:

-- confgen.lua

cg.addPath(".config")

-- Evaluate the template `lib.cgt` and return it's output as a string,
-- which we don't care about here.
cg.doTemplateFile("lib.cgt")

cg.opt = {
    font = "MyFavoriteFont",
    font_size = "11",
}

Unlike the cg.add* family of functions, the cg.doTemplate* family immediately evaluates templates rather than adding them to the file list. They also return the resulting code, which is also handy. Let's get to the interesting file, though.

<! -- lib.cgt !>

<! opt.greet = function(tmpl, name) !>
Hello <% name %>!
<! end !>

Note the tmpl argument this function takes. This works similarly to subtemplates, except that we now append to another file's (the caller's) template rather than a subtemplate. We could now use this in another template:

<! -- message.txt.cgt !>

<! opt.greet(tmpl, "my Friend") !>

Note: We use a statement block here rather than an expression block, because the greet function pushes to our template, not returning any value.

Alright, I think that's probably enough insanity for one article, but there are more to come! This also isn't all that Confgen is packing, far from it. There are many API functions (such as onDone callbacks, file iterators and more), a whole assortment of handy CLI flags, and most importantly the previously hinted at ConfgenFS that didn't fit in here. Thank you for reading, and feel free to reach out if you have any questions, remarks or concerns.

Found a bug?

Sorry about that! Confgen is still a little rough around the edges sometimes. Please report it on the issue tracker.