Allow to disable directory listing

Signed-off-by: Solderpunk <solderpunk@posteo.net>

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2019 Solderpunk <solderpunk@sdf.org> . All rights reserved.
+Copyright (c) 2019-23 Solderpunk <solderpunk@posteo.net>. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification,
 are permitted provided that the following conditions are met:

README.md

@@ -59,6 +59,12 @@ Molly Brown only has a single dependency beyond the Go standard
 library, which is [this TOML parsing
 library](https://github.com/BurntSushi/toml).
 
+The OpenBSD implementation also uses the [golang.org/x/sys/unix
+package](https://godoc.org/golang.org/x/sys/unix) to provide the
+[pledge(2)](https://man.openbsd.org/pledge.2) and
+[unveil(2)](https://man.openbsd.org/unveil.2) system calls to provide
+additional security features.
+
 ## Installation
 
 The easiest way for now to install Molly Brown is to use the standard
@@ -101,6 +107,16 @@ command line option to tell Molly Brown where to find it.
 
 ### Running
 
+The `molly-brown` executable recognises the following command line
+switches:
+
+* `-c`: Used to specify a config file.
+* `-C`: Used to specify a directory to chroot to (unix only).
+* `-u`: Used to specify the name of an unprivileged user which
+        Molly Brown should switch to running as if started as
+        root or run as a setuid executable (unix only).
+* `-v`: Print version number and exit.
+
 Molly Brown does not handle details like daemonising itself, changing
 the user it runs as, etc.  You will need to take care of these tasks
 by, e.g. integrating Molly Brown with your operating system's init
@@ -165,6 +181,23 @@ You can start your `mollybrownd` daemon with `rcctl`:
 rcctl start mollybrownd
 ```
 
+#### FreeBSD
+
+An example FreeBSD rc script is in
+`contrib/init/molly-brown.freebsd.example`.
+
+Copy rc script to `/etc/rc.d/molly`, and add `molly_enable="YES"`
+to `/etc/rc.conf` to enable the service.
+
+Make sure the `daemon` user has access to config locations in
+`molly.conf` like `CertPath`, `KeyPath`, `DocBase`, etc.
+
+Start `molly` with,
+
+```
+service molly start
+```
+
 ## Configuration Options
 
 The following sections detail all the options which can be set in
@@ -199,11 +232,13 @@ examples of the appropriate syntax.
 * `AccessLog`: Path to access log file (default value `access.log`,
   i.e. in the current wrorking directory).  Note that all intermediate
   directories must exist, Molly Brown won't create them for you.  Set
-  to `-` for logging to `stdout`.
-* `ErrorLog`: Path to error log file (default value `error.log`, i.e.
-  in the current wrorking directory).  Note that all intermediate
-  directories must exist, Molly Brown won't create them for you.  Set
-  to `-` for logging to `stdout`.
+  to `-` for logging to `stdout`, or to an empty string to disable
+  access logging.
+* `ErrorLog`: Path to error log file.  If set to an empty string (the
+  default), Molly Brown will log errors to stderr (where they are
+  easily captured by systemd or similar init systems).  If set to a
+  file, note that all intermediate directories must exist, Molly Brown
+  won't create them for you.
 * `GeminiExt`: Files with this extension will be served with a MIME
   type of `text/gemini` (default value `gmi`).
 * `MimeOverrides`: In this section of the config file, keys are path
@@ -212,6 +247,8 @@ examples of the appropriate syntax.
   will be used instead of one inferred from the filename extension.
 * `DefaultLang`: If this option is set, it will be served as the
   `lang` parameter of the MIME type for all `text/gemini` content.
+* `DefaultEncoding`: If this option is set, it will be served as the
+  `charset` parameter of the MIME type for all `text/gemini` content.
 
 ### Directory listings
 
@@ -225,9 +262,15 @@ instead of the default "Directory listing" title.
 The following options allow users to configure various aspects of the
 directory listing:
 
+* `DirectoryListing` (boolean): if true, enable directory listing; if false,
+  return 51 Not found (default value true)
 * `DirectorySort`: A string specifying how to sort files in
   automatically generated directory listings.  Must be one of "Name",
   "Size" or "Time" (default value "Name").
+* `DirectorySubdirsFirst` (boolean): if true, list subdirectories of
+  the directory being listed before files.  Subdirs and files will be
+  sorted within their respective categories according to
+  `DirectorySort` (default value false).
 * `DirectoryReverse` (boolean): if true, automatically generated
   directory listings will list files in descending order of whatever
   `DirectorySort` is set to (default value false).
@@ -268,17 +311,51 @@ a request URL includes components after the path to an executable
 executable (e.g. `/var/gemini/cgi-bin/scripty.py`) while the variable
 `PATH_INFO` will contain the remainder (e.g. `foo/bar/baz`).
 
-It is very important to be aware that programs written in Go are
-*unable* to reliably change their UID once started, due to how
-goroutines are implemented on unix systems.  As an unavoidable
-consequence of this, CGI processes started by Molly Brown are run as
-the same user as the server process.  This means CGI processes
-necessarily have read and write access to the server logs and to the
-TLS private key.  There is no way to work around this.  As such you
-must be extremely careful about only running trustworthy CGI
-applications, ideally only applications you have carefully written
-yourself.  Allowing untrusted users to upload arbitrary executable
-files into a CGI path is a serious security vulnerability.
+Molly Brown itself tries very hard to avoid being tricked into serving
+content that isn't supposed to be served, but it is completely unable
+to impose any control over what CGI processes can or can't go after
+they are started!  Where possible, Molly Brown will use the operating
+system's security features to reduce risk, but it is your
+responsibility to understand what it can and cannot do and weigh the
+risks accordingly:
+
+When compiled on GNU/Linux with Go version 1.16 or later, or on any
+other unix operating system with any version of Go, Molly Brown will
+use the setuid() system call as follows.  When the compiled
+`molly-brown` executable has its SETUID bit set, so that it starts
+with the privileges of the user who owns the binary, it will change
+the effective UID back to the real UID before it begins accepting
+network connections.  This way, config files, log files and TLS keys
+can be set readable by the user who owns the binary, but not readable
+by the user who runs the binary.  CGI processes will then be unable to
+read any of those sensitive files.  If the binary is not SETUID but is
+run by the superuser/root, then Molly will change its UID to that of
+the `nobody` user (or any other user specified with the `-u` option)
+before accepting network connections, so CGI processes will again not
+be able to read sensitive files.  Note that while these measures can
+protect Molly's own sensitive files from CGI processes, CGI processes
+may still be able to read other sensitive files anywhere else on the
+system.  Consider chroot()-ing Molly Brown into a small corner of the
+filesystem (see discussion of the `-C` option at the start of the
+Running section) to reduce this risk.
+
+When compiled on GNU/Linux with Go versions 1.15 or earlier, Molly
+Brown is completley unable to reliably change its UID due to the way
+early implementations of goroutines interacted with the setuid()
+system call.  In this situation, Molly Brown will refuse to run as
+superuser/root.  It will run as any other user, but CGI processes will
+necessary run as the same user as the server and so unavoidably will
+have access to sensitive files.  You should proceed with extreme
+caution and only use carefully vetted CGI programs.  Consider using
+systemd's ability to chroot a non-privileged process at the moment of
+startup to at least confine the risk to Molly Brown's sensitive files
+and not the entire system's.
+
+Molly Brown will compile on non-unix operating systems and is known to
+run on Plan9, for example, but no special security measures are taken
+on these non-unix platforms.  It is your responsibility to understand
+the risks.  If you are aware of security measures for these systems
+which can be implemented in Go, patches are extremely welcome.
 
 SCGI applications must be started separately (i.e. Molly Brown expects
 them to already be running and will not attempt to start them itself),
@@ -296,7 +373,8 @@ startup, database connection etc. on each request).
   single non-separator character and `*` matches a sequence of them -
   if wildcards are used, the path should *not* end in a trailing slash
   - this appears to be a peculiarity of the Go standard library's
-  `filepath.Glob` function.
+  `filepath.Glob` function.  Any non-absolute paths will be resolved
+  relative to `DocBase`.
 * `SCGIPaths`: In this section of the config file, keys are URL path
   prefixes and values are filesystem paths to unix domain sockets.
   Any request for a URL whose path begins with one of the specified
@@ -306,7 +384,16 @@ startup, database connection etc. on each request).
   SCGI applications are responsible for generating their own response
   headers.
 
-### Certificate zones
+### TLS options
+
+* `AllowTLS12` (boolean): if true, Molly Brown will accept connections
+  from clients using TLS version 1.2 or later (1.2 is the bare minimum
+  allowed by the Gemini spec).  If set to false, Molly Brown will
+  instead require TLS version 1.3 or later - 1.2 to 1.3 was a big
+  change and drastic simplification of the TLS spec which discarded a
+  wide range of old and insecure configurations.  (default value `true`)
+
+#### Certificate zones
 
 Molly Brown allows you to use client certificates to restrict access
 to certain resources (which may be static or dynamic).  The overall
@@ -352,7 +439,9 @@ other settings in `.molly` files will be ignored:
 
 * `CertificateZones`
 * `DefaultLang`
+* `DefaultEncoding`
 * `DirectorySort`
+* `DirectorySubdirsFirst`
 * `DirectoryReverse`
 * `DirectoryTitles`
 * `GeminiExt`

certificate.go

@@ -10,24 +10,24 @@ import (
 	"time"
 )
 
-func enforceCertificateValidity(clientCerts []*x509.Certificate, conn net.Conn, log *LogEntry) {
+func enforceCertificateValidity(clientCerts []*x509.Certificate, conn net.Conn, logEntry *LogEntry) {
 	// This will fail if any of multiple certs are invalid
 	// Maybe we should just require one valid?
 	now := time.Now()
 	for _, cert := range clientCerts {
 		if now.Before(cert.NotBefore) {
 			conn.Write([]byte("64 Client certificate not yet valid!\r\n"))
-			log.Status = 64
+			logEntry.Status = 64
 			return
 		} else if now.After(cert.NotAfter) {
 			conn.Write([]byte("65 Client certificate has expired!\r\n"))
-			log.Status = 65
+			logEntry.Status = 65
 			return
 		}
 	}
 }
 
-func handleCertificateZones(URL *url.URL, clientCerts []*x509.Certificate, config Config, conn net.Conn, log *LogEntry) {
+func handleCertificateZones(URL *url.URL, clientCerts []*x509.Certificate, config UserConfig, conn net.Conn, logEntry *LogEntry) {
 	authorised := true
 	for zone, allowedFingerprints := range config.CertificateZones {
 		matched, err := regexp.Match(zone, []byte(URL.Path))
@@ -47,10 +47,10 @@ func handleCertificateZones(URL *url.URL, clientCerts []*x509.Certificate, confi
 	if !authorised {
 		if len(clientCerts) > 0 {
 			conn.Write([]byte("61 Provided certificate not authorised for this resource\r\n"))
-			log.Status = 61
+			logEntry.Status = 61
 		} else {
 			conn.Write([]byte("60 A pre-authorised certificate is required to access this resource\r\n"))
-			log.Status = 60
+			logEntry.Status = 60
 		}
 		return
 	}

config.go

@@ -2,84 +2,137 @@ package main
 
 import (
 	"errors"
+	"github.com/BurntSushi/toml"
 	"log"
 	"os"
 	"path/filepath"
-	"github.com/BurntSushi/toml"
+	"strings"
 )
 
-type Config struct {
-	Port             int
-	Hostname         string
-	CertPath         string
-	KeyPath          string
-	DocBase          string
-	HomeDocBase      string
-	GeminiExt        string
-	DefaultLang      string
-	AccessLog        string
-	ErrorLog         string
-	ReadMollyFiles   bool
-	TempRedirects    map[string]string
-	PermRedirects    map[string]string
-	MimeOverrides    map[string]string
-	CGIPaths         []string
-	SCGIPaths        map[string]string
-	CertificateZones map[string][]string
-	DirectorySort    string
-	DirectoryReverse bool
-	DirectoryTitles  bool
+type SysConfig struct {
+	Port                  int
+	Hostname              string
+	CertPath              string
+	KeyPath               string
+	AccessLog             string
+	ErrorLog              string
+	DocBase               string
+	HomeDocBase           string
+	CGIPaths              []string
+	SCGIPaths             map[string]string
+	ReadMollyFiles        bool
+	AllowTLS12            bool
+	RateLimitEnable       bool
+	RateLimitAverage      int
+	RateLimitSoft         int
+	RateLimitHard         int
 }
 
-type MollyFile struct {
-	GeminiExt        string
-	TempRedirects    map[string]string
-	PermRedirects    map[string]string
-	MimeOverrides    map[string]string
-	CertificateZones map[string][]string
-	DefaultLang      string
-	DirectorySort    string
-	DirectoryReverse bool
-	DirectoryTitles  bool
+type UserConfig struct {
+	GeminiExt             string
+	DefaultLang           string
+	DefaultEncoding       string
+	TempRedirects         map[string]string
+	PermRedirects         map[string]string
+	MimeOverrides         map[string]string
+	CertificateZones      map[string][]string
+	DirectoryListing      bool
+	DirectorySort         string
+	DirectorySubdirsFirst bool
+	DirectoryReverse      bool
+	DirectoryTitles       bool
 }
 
-func getConfig(filename string) (Config, error) {
+func getConfig(filename string) (SysConfig, UserConfig, error) {
 
-	var config Config
+	var sysConfig SysConfig
+	var userConfig UserConfig
 
 	// Defaults
-	config.Port = 1965
-	config.Hostname = "localhost"
-	config.CertPath = "cert.pem"
-	config.KeyPath = "key.pem"
-	config.DocBase = "/var/gemini/"
-	config.HomeDocBase = "users"
-	config.GeminiExt = "gmi"
-	config.DefaultLang = ""
-	config.AccessLog = "access.log"
-	config.ErrorLog = "error.log"
-	config.TempRedirects = make(map[string]string)
-	config.PermRedirects = make(map[string]string)
-	config.CGIPaths = make([]string, 0)
-	config.SCGIPaths = make(map[string]string)
-	config.DirectorySort = "Name"
+	sysConfig.Port = 1965
+	sysConfig.Hostname = "localhost"
+	sysConfig.CertPath = "cert.pem"
+	sysConfig.KeyPath = "key.pem"
+	sysConfig.AccessLog = "access.log"
+	sysConfig.ErrorLog = ""
+	sysConfig.DocBase = "/var/gemini/"
+	sysConfig.HomeDocBase = "users"
+	sysConfig.CGIPaths = make([]string, 0)
+	sysConfig.SCGIPaths = make(map[string]string)
+	sysConfig.ReadMollyFiles = false
+	sysConfig.AllowTLS12 = true
+	sysConfig.RateLimitEnable = false
+	sysConfig.RateLimitAverage = 1
+	sysConfig.RateLimitSoft = 10
+	sysConfig.RateLimitHard = 50
+
+	userConfig.GeminiExt = "gmi"
+	userConfig.DefaultLang = ""
+	userConfig.DefaultEncoding = ""
+	userConfig.TempRedirects = make(map[string]string)
+	userConfig.PermRedirects = make(map[string]string)
+	userConfig.DirectoryListing = true
+	userConfig.DirectorySort = "Name"
+	userConfig.DirectorySubdirsFirst = false
 
 	// Return defaults if no filename given
 	if filename == "" {
-		return config, nil
+		return sysConfig, userConfig, nil
 	}
 
 	// Attempt to overwrite defaults from file
+	sysConfig, err := readSysConfig(filename, sysConfig)
+	if err != nil {
+		return sysConfig, userConfig, err
+	}
+	userConfig, err = readUserConfig(filename, userConfig, true)
+	if err != nil {
+		return sysConfig, userConfig, err
+	}
+	return sysConfig, userConfig, nil
+}
+
+func readSysConfig(filename string, config SysConfig) (SysConfig, error) {
+
 	_, err := toml.DecodeFile(filename, &config)
 	if err != nil {
 		return config, err
 	}
 
-	// Validate pseudo-enums
-	switch config.DirectorySort {
-	case "Name", "Size", "Time":
-	default:
-		return config, errors.New("Invalid DirectorySort value.")
+	// Force hostname to lowercase
+	config.Hostname = strings.ToLower(config.Hostname)
+
+	// Absolutise paths
+	config.DocBase, err = filepath.Abs(config.DocBase)
+	if err != nil {
+		return config, err
+	}
+	config.CertPath, err = filepath.Abs(config.CertPath)
+	if err != nil {
+		return config, err
+	}
+	config.KeyPath, err = filepath.Abs(config.KeyPath)
+	if err != nil {
+		return config, err
+	}
+	if config.AccessLog != "" && config.AccessLog != "-" {
+		config.AccessLog, err = filepath.Abs(config.AccessLog)
+		if err != nil {
+			return config, err
+		}
+	}
+	if config.ErrorLog != "" {
+		config.ErrorLog, err = filepath.Abs(config.ErrorLog)
+		if err != nil {
+			return config, err
+		}
+	}
+
+	// Absolutise CGI paths
+	for index, cgiPath := range config.CGIPaths {
+		if !filepath.IsAbs(cgiPath) {
+			config.CGIPaths[index] = filepath.Join(config.DocBase, cgiPath)
+		}
 	}
 
 	// Expand CGI paths
@@ -93,44 +146,71 @@ func getConfig(filename string) (Config, error) {
 	}
 	config.CGIPaths = cgiPaths
 
+	// Absolutise SCGI paths
+	for index, scgiPath := range config.SCGIPaths {
+		config.SCGIPaths[index], err = filepath.Abs( scgiPath)
+		if err != nil {
+			return config, err
+		}
+	}
+
 	return config, nil
 }
 
-func parseMollyFiles(path string, config *Config, errorLog *log.Logger) {
+func readUserConfig(filename string, config UserConfig, requireValid bool) (UserConfig, error) {
+
+	_, err := toml.DecodeFile(filename, &config)
+	if err != nil {
+		return config, err
+	}
+
+	// Validate pseudo-enums
+	if requireValid {
+		switch config.DirectorySort {
+			case "Name", "Size", "Time":
+			default:
+				return config, errors.New("Invalid DirectorySort value.")
+		}
+	}
+
+	// Validate redirects
+	for key, value := range config.TempRedirects {
+		if strings.Contains(value, "://") && !strings.HasPrefix(value, "gemini://") {
+			if requireValid {
+				return config, errors.New("Invalid cross-protocol redirect to " + value)
+			} else {
+				log.Println("Ignoring cross-protocol redirect to " + value + " in .molly file " + filename)
+				delete(config.TempRedirects, key)
+			}
+		}
+	}
+	for key, value := range config.PermRedirects {
+		if strings.Contains(value, "://") && !strings.HasPrefix(value, "gemini://") {
+			if requireValid {
+				return config, errors.New("Invalid cross-protocol redirect to " + value)
+			} else {
+				log.Println("Ignoring cross-protocol redirect to " + value + " in .molly file " + filename)
+				delete(config.PermRedirects, key)
+			}
+		}
+	}
+
+	return config, nil
+}
+
+func parseMollyFiles(path string, docBase string, config UserConfig) UserConfig {
 	// Replace config variables which use pointers with new ones,
 	// so that changes made here aren't reflected everywhere.
-	newTempRedirects := make(map[string]string)
-	for key, value := range config.TempRedirects {
-		newTempRedirects[key] = value
-	}
-	config.TempRedirects = newTempRedirects
-	newPermRedirects := make(map[string]string)
-	for key, value := range config.PermRedirects {
-		newPermRedirects[key] = value
-	}
-	config.PermRedirects = newPermRedirects
-	newMimeOverrides := make(map[string]string)
-	for key, value := range config.MimeOverrides {
-		newMimeOverrides[key] = value
-	}
-	config.MimeOverrides = newMimeOverrides
-	newCertificateZones := make(map[string][]string)
-	for key, value := range config.CertificateZones {
-		newCertificateZones[key] = value
-	}
-	config.CertificateZones = newCertificateZones
-	// Initialise MollyFile using main Config
-	var mollyFile MollyFile
-	mollyFile.GeminiExt = config.GeminiExt
-	mollyFile.DefaultLang = config.DefaultLang
-	mollyFile.DirectorySort = config.DirectorySort
-	mollyFile.DirectoryReverse = config.DirectoryReverse
-	mollyFile.DirectoryTitles = config.DirectoryTitles
+	config.TempRedirects = make(map[string]string)
+	config.PermRedirects = make(map[string]string)
+	config.MimeOverrides = make(map[string]string)
+	config.CertificateZones = make(map[string][]string)
+
 	// Build list of directories to check
 	var dirs []string
 	dirs = append(dirs, path)
 	for {
-		if path == filepath.Clean(config.DocBase) {
+		if path == filepath.Clean(docBase) {
 			break
 		}
 		subpath := filepath.Dir(path)
@@ -152,28 +232,12 @@ func parseMollyFiles(path string, config *Config, errorLog *log.Logger) {
 			continue
 		}
 		// If the file exists and we can read it, try to parse it
-		_, err = toml.DecodeFile(mollyPath, &mollyFile)
+		config, err = readUserConfig(mollyPath, config, false)
 		if err != nil {
-			errorLog.Println("Error parsing .molly file " + mollyPath + ": " + err.Error())
+			log.Println("Error parsing .molly file " + mollyPath + ": " + err.Error())
 			continue
 		}
-		// Overwrite main Config using MollyFile
-		config.GeminiExt = mollyFile.GeminiExt
-		config.DefaultLang = mollyFile.DefaultLang
-		config.DirectorySort = mollyFile.DirectorySort
-		config.DirectoryReverse = mollyFile.DirectoryReverse
-		config.DirectoryTitles = mollyFile.DirectoryTitles
-		for key, value := range mollyFile.TempRedirects {
-			config.TempRedirects[key] = value
-		}
-		for key, value := range mollyFile.PermRedirects {
-			config.PermRedirects[key] = value
-		}
-		for key, value := range mollyFile.MimeOverrides {
-			config.MimeOverrides[key] = value
-		}
-		for key, value := range mollyFile.CertificateZones {
-			config.CertificateZones[key] = value
-		}
 	}
+
+	return config
 }

contrib/init/molly-brown.freebsd.example

@@ -0,0 +1,47 @@
+#!/bin/sh
+#
+# $FreeBSD$
+#
+
+# PROVIDE: molly
+# REQUIRE: networking
+# KEYWORD: shutdown
+
+. /etc/rc.subr
+
+name="molly"
+desc="Gemini Protocol daemon"
+rcvar="molly_enable"
+command="/usr/local/sbin/molly-brown"
+command_args="-c /etc/molly.conf"
+molly_brown_user="daemon"
+pidfile="/var/run/${name}.pid"
+required_files="/etc/molly.conf"
+
+start_cmd="molly_start"
+stop_cmd="molly_stop"
+status_cmd="molly_status"
+
+molly_start() {
+        /usr/sbin/daemon -P ${pidfile} -r -f -u $molly_brown_user $command
+}
+
+molly_stop() {
+        if [ -e "${pidfile}" ]; then
+                kill -s TERM `cat ${pidfile}`
+        else
+                echo "${name} is not running"
+        fi
+
+}
+
+molly_status() {
+        if [ -e "${pidfile}" ]; then
+                echo "${name} is running as pid `cat ${pidfile}`"
+        else
+                echo "${name} is not running"
+        fi
+}
+
+load_rc_config $name
+run_rc_command "$1"

dirlist.go

@@ -11,7 +11,7 @@ import (
 	"strings"
 )
 
-func generateDirectoryListing(URL *url.URL, path string, config Config) (string, error) {
+func generateDirectoryListing(URL *url.URL, path string, config UserConfig) (string, error) {
 	var listing string
 	files, err := ioutil.ReadDir(path)
 	if err != nil {
@@ -36,7 +36,7 @@ func generateDirectoryListing(URL *url.URL, path string, config Config) (string,
 		up := filepath.Dir(URL.Path)
 		listing += fmt.Sprintf("=> %s %s\n", up, "..")
 	}
-	// Sort files
+	// Sort files by criteria first
 	sort.SliceStable(files, func(i, j int) bool {
 		if config.DirectoryReverse {
 			i, j = j, i
@@ -50,6 +50,17 @@ func generateDirectoryListing(URL *url.URL, path string, config Config) (string,
 		}
 		return false // Should not happen
 	})
+	// Sort directories before file
+	if config.DirectorySubdirsFirst {
+		sort.SliceStable(files, func(i, j int) bool {
+			// If i is a dir and j is a file, i < j
+			if files[i].IsDir() && !files[j].IsDir() {
+				return true
+			} else {
+				return false
+			}
+		})
+	}
 	// Format lines
 	for _, file := range files {
 		// Skip dotfiles
@@ -71,7 +82,7 @@ func generateDirectoryListing(URL *url.URL, path string, config Config) (string,
 	return listing, nil
 }
 
-func generatePrettyFileLabel(info os.FileInfo, path string, config Config) string {
+func generatePrettyFileLabel(info os.FileInfo, path string, config UserConfig) string {
 	var size string
 	if info.IsDir() {
 		size = "        "

dynamic.go

@@ -15,7 +15,7 @@ import (
 	"time"
 )
 
-func handleCGI(config Config, path string, cgiPath string, URL *url.URL, log *LogEntry, errorLog *log.Logger, conn net.Conn) {
+func handleCGI(config SysConfig, path string, cgiPath string, URL *url.URL, logEntry *LogEntry, conn net.Conn) {
 	// Find the shortest leading part of path which maps to an executable file.
 	// Call this part scriptPath, and everything after it pathInfo.
 	components := strings.Split(path, "/")
@@ -58,42 +58,50 @@ func handleCGI(config Config, path string, cgiPath string, URL *url.URL, log *Lo
 	response, err := cmd.Output()
 
 	if ctx.Err() == context.DeadlineExceeded {
-		errorLog.Println("Terminating CGI process " + path + " due to exceeding 10 second runtime limit.")
+		log.Println("Terminating CGI process " + path + " due to exceeding 10 second runtime limit.")
 		conn.Write([]byte("42 CGI process timed out!\r\n"))
-		log.Status = 42
+		logEntry.Status = 42
 		return
 	}
 	if err != nil {
-		errorLog.Println("Error running CGI program " + path + ": " + err.Error())
+		log.Println("Error running CGI program " + path + ": " + err.Error())
 		if err, ok := err.(*exec.ExitError); ok {
-			errorLog.Println("↳ stderr output: " + string(err.Stderr))
+			log.Println("↳ stderr output: " + string(err.Stderr))
 		}
 		conn.Write([]byte("42 CGI error!\r\n"))
-		log.Status = 42
+		logEntry.Status = 42
 		return
 	}
 	// Extract response header
-	header, _, err := bufio.NewReader(strings.NewReader(string(response))).ReadLine()
-	status, err2 := strconv.Atoi(strings.Fields(string(header))[0])
-	if err != nil || err2 != nil {
-		errorLog.Println("Unable to parse first line of output from CGI process " + path + " as valid Gemini response header.  Line was: " + string(header))
+	responseString := string(response)
+	if len(responseString) == 0 {
+		log.Println("Received no response from CGI process " + path)
 		conn.Write([]byte("42 CGI error!\r\n"))
-		log.Status = 42
+		logEntry.Status = 42
 		return
 	}
-	log.Status = status
+	header, _, _ := bufio.NewReader(strings.NewReader(string(response))).ReadLine()
+	status, err := strconv.Atoi(strings.Fields(string(header))[0])
+	if err != nil {
+		log.Println("Unable to parse first line of output from CGI process " + path + " as valid Gemini response header.  Line was: " + string(header))
+		conn.Write([]byte("42 CGI error!\r\n"))
+		logEntry.Status = 42
+		return
+	}
+	logEntry.Status = status
+
 	// Write response
 	conn.Write(response)
 }
 
-func handleSCGI(URL *url.URL, scgiPath string, scgiSocket string, config Config, log *LogEntry, errorLog *log.Logger, conn net.Conn) {
+func handleSCGI(URL *url.URL, scgiPath string, scgiSocket string, config SysConfig, logEntry *LogEntry, conn net.Conn) {
 
 	// Connect to socket
 	socket, err := net.Dial("unix", scgiSocket)
 	if err != nil {
-		errorLog.Println("Error connecting to SCGI socket " + scgiSocket + ": " + err.Error())
+		log.Println("Error connecting to SCGI socket " + scgiSocket + ": " + err.Error())
 		conn.Write([]byte("42 Error connecting to SCGI service!\r\n"))
-		log.Status = 42
+		logEntry.Status = 42
 		return
 	}
 	defer socket.Close()
@@ -123,9 +131,9 @@ func handleSCGI(URL *url.URL, scgiPath string, scgiSocket string, config Config,
 				break
 			} else if !first {
 				// Err
-				errorLog.Println("Error reading from SCGI socket " + scgiSocket + ": " + err.Error())
+				log.Println("Error reading from SCGI socket " + scgiSocket + ": " + err.Error())
 				conn.Write([]byte("42 Error reading from SCGI service!\r\n"))
-				log.Status = 42
+				logEntry.Status = 42
 				return
 			} else {
 				break
@@ -138,17 +146,17 @@ func handleSCGI(URL *url.URL, scgiPath string, scgiSocket string, config Config,
 			status, err := strconv.Atoi(strings.Fields(lines[0])[0])
 			if err != nil {
 				conn.Write([]byte("42 CGI error!\r\n"))
-				log.Status = 42
+				logEntry.Status = 42
 				return
 			}
-			log.Status = status
+			logEntry.Status = status
 		}
 		// Send to client
 		conn.Write(buffer[:n])
 	}
 }
 
-func prepareCGIVariables(config Config, URL *url.URL, conn net.Conn, script_path string, path_info string) map[string]string {
+func prepareCGIVariables(config SysConfig, URL *url.URL, conn net.Conn, script_path string, path_info string) map[string]string {
 	vars := prepareGatewayVariables(config, URL, conn)
 	vars["GATEWAY_INTERFACE"] = "CGI/1.1"
 	vars["SCRIPT_PATH"] = script_path
@@ -156,7 +164,7 @@ func prepareCGIVariables(config Config, URL *url.URL, conn net.Conn, script_path
 	return vars
 }
 
-func prepareSCGIVariables(config Config, URL *url.URL, scgiPath string, conn net.Conn) map[string]string {
+func prepareSCGIVariables(config SysConfig, URL *url.URL, scgiPath string, conn net.Conn) map[string]string {
 	vars := prepareGatewayVariables(config, URL, conn)
 	vars["SCGI"] = "1"
 	vars["CONTENT_LENGTH"] = "0"
@@ -165,7 +173,7 @@ func prepareSCGIVariables(config Config, URL *url.URL, scgiPath string, conn net
 	return vars
 }
 
-func prepareGatewayVariables(config Config, URL *url.URL, conn net.Conn) map[string]string {
+func prepareGatewayVariables(config SysConfig, URL *url.URL, conn net.Conn) map[string]string {
 	vars := make(map[string]string)
 	vars["QUERY_STRING"] = URL.RawQuery
 	vars["REQUEST_METHOD"] = ""

example.conf

@@ -14,7 +14,9 @@
 #
 ## Directory listing
 #
+#DirectoryListing = true
 #DirectorySort = "Time"
+#DirectorySubdirsFirst = false
 #DirectoryReverse = true
 #DirectoryTitles = true
 #

go.mod

@@ -0,0 +1,8 @@
+module tildegit.org/solderpunk/molly-brown
+
+go 1.15
+
+require (
+	github.com/BurntSushi/toml v1.2.1 // indirect
+	golang.org/x/sys v0.5.0 // indirect
+)

go.sum

@@ -0,0 +1,4 @@
+github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
+github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
+golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

handler.go

@@ -9,153 +9,251 @@ import (
 	"log"
 	"mime"
 	"net"
+	"net/http"
 	"net/url"
 	"os"
 	"path/filepath"
 	"regexp"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 )
 
-func handleGeminiRequest(conn net.Conn, config Config, accessLogEntries chan LogEntry, errorLog *log.Logger) {
+// Utility function below borrowed from
+// https://stackoverflow.com/questions/28024731/check-if-given-path-is-a-subdirectory-of-another-in-golang
+func isSubdir(subdir, superdir string) (bool, error) {
+    up := ".." + string(os.PathSeparator)
+
+    // path-comparisons using filepath.Abs don't work reliably according to docs (no unique representation).
+    rel, err := filepath.Rel(superdir, subdir)
+    if err != nil {
+        return false, err
+    }
+    if !strings.HasPrefix(rel, up) && rel != ".." {
+        return true, nil
+    }
+    return false, nil
+}
+
+func handleGeminiRequest(conn net.Conn, sysConfig SysConfig, config UserConfig, accessLogEntries chan LogEntry, rl *RateLimiter, wg *sync.WaitGroup) {
 	defer conn.Close()
+	defer wg.Done()
 	var tlsConn (*tls.Conn) = conn.(*tls.Conn)
-	var log LogEntry
-	log.Time = time.Now()
-	log.RemoteAddr = conn.RemoteAddr()
-	log.RequestURL = "-"
-	log.Status = 0
-	defer func() { accessLogEntries <- log }()
+	var logEntry LogEntry
+	logEntry.Time = time.Now()
+	logEntry.RemoteAddr = conn.RemoteAddr()
+	logEntry.RequestURL = "-"
+	logEntry.Status = 0
+	if accessLogEntries != nil {
+		defer func() { accessLogEntries <- logEntry }()
+	}
+
+	// Enforce rate limiting
+	if sysConfig.RateLimitEnable {
+		noPort := logEntry.RemoteAddr.String()
+		noPort = noPort[0:strings.LastIndex(noPort, ":")]
+		limited := rl.hardLimited(noPort)
+		if limited {
+			conn.Close()
+			return
+		}
+		delay, limited := rl.softLimited(noPort)
+		if limited {
+			conn.Write([]byte("44 " + strconv.Itoa(delay) + " second cool down, please!\r\n"))
+			logEntry.Status = 44
+			return
+		}
+	}
 
 	// Read request
-	URL, err := readRequest(conn, &log, errorLog)
+	URL, err := readRequest(conn, &logEntry)
 	if err != nil {
 		return
 	}
 
 	// Enforce client certificate validity
 	clientCerts := tlsConn.ConnectionState().PeerCertificates
-	enforceCertificateValidity(clientCerts, conn, &log)
-	if log.Status != 0 {
+	enforceCertificateValidity(clientCerts, conn, &logEntry)
+	if logEntry.Status != 0 {
 		return
 	}
 
 	// Reject non-gemini schemes
 	if URL.Scheme != "gemini" {
 		conn.Write([]byte("53 No proxying to non-Gemini content!\r\n"))
-		log.Status = 53
+		logEntry.Status = 53
 		return
 	}
 
 	// Reject requests for content from other servers
-	if URL.Hostname() != config.Hostname || (URL.Port() != "" && URL.Port() != strconv.Itoa(config.Port)) {
+	requestedHost := strings.ToLower(URL.Hostname())
+	// Trim trailing . from FQDNs
+	if strings.HasSuffix(requestedHost, ".") {
+		requestedHost = requestedHost[:len(requestedHost)-1]
+	}
+	if requestedHost != sysConfig.Hostname || (URL.Port() != "" && URL.Port() != strconv.Itoa(sysConfig.Port)) {
 		conn.Write([]byte("53 No proxying to other hosts or ports!\r\n"))
-		log.Status = 53
+		logEntry.Status = 53
 		return
 	}
 
 	// Fail if there are dots in the path
 	if strings.Contains(URL.Path, "..") {
 		conn.Write([]byte("50 Your directory traversal technique has been defeated!\r\n"))
-		log.Status = 50
+		logEntry.Status = 50
 		return
 	}
 
-	// Resolve URI path to actual filesystem path
-	path := resolvePath(URL.Path, config)
-
-	// Paranoid security measures:
-	// Fail ASAP if the URL has mapped to a sensitive file
-	if path == config.CertPath || path == config.KeyPath || path == config.AccessLog || path == config.ErrorLog || filepath.Base(path) == ".molly" {
-		conn.Write([]byte("51 Not found!\r\n"))
-		log.Status = 51
-		return
-	}
-
-	// Read Molly files
-	if config.ReadMollyFiles {
-		parseMollyFiles(path, &config, errorLog)
-	}
-
 	// Check whether this URL is in a certificate zone
-	handleCertificateZones(URL, clientCerts, config, conn, &log)
-	if log.Status != 0 {
+	handleCertificateZones(URL, clientCerts, config, conn, &logEntry)
+	if logEntry.Status != 0 {
 		return
 	}
 
 	// Check for redirects
-	handleRedirects(URL, config, conn, &log, errorLog)
-	if log.Status != 0 {
+	handleRedirects(URL, config, conn, &logEntry)
+	if logEntry.Status != 0 {
 		return
 	}
 
-	// Check whether this URL is mapped to an SCGI app
-	for scgiPath, scgiSocket := range config.SCGIPaths {
-		if strings.HasPrefix(URL.Path, scgiPath) {
-			handleSCGI(URL, scgiPath, scgiSocket, config, &log, errorLog, conn)
+	// Resolve URI path to actual filesystem path
+	path := resolvePath(URL.Path, sysConfig)
+
+	// Read Molly files.  Yes, even before checking if `path` exists!
+	// /foo/bar/baz.gmi may not exist on the disk but /foo/.molly may and it
+	// may inform us that /foo/bar/baz.gmi ought to redirect to somewhere which
+	// *does* exist on disk!
+	if sysConfig.ReadMollyFiles {
+		config = parseMollyFiles(path, sysConfig.DocBase, config)
+		// We may have picked up new cert zones and/or redirects above, so:
+		handleCertificateZones(URL, clientCerts, config, conn, &logEntry)
+		if logEntry.Status != 0 {
+			return
+		}
+		handleRedirects(URL, config, conn, &logEntry)
+		if logEntry.Status != 0 {
 			return
 		}
 	}
 
 	// Check whether this URL is in a configured CGI path
-	for _, cgiPath := range config.CGIPaths {
+	for _, cgiPath := range sysConfig.CGIPaths {
 		if strings.HasPrefix(path, cgiPath) {
-			handleCGI(config, path, cgiPath, URL, &log, errorLog, conn)
-			if log.Status != 0 {
+			handleCGI(sysConfig, path, cgiPath, URL, &logEntry, conn)
+			if logEntry.Status != 0 {
 				return
 			}
 		}
 	}
 
-	// Fail if file does not exist or perms aren't right
-	info, err := os.Stat(path)
-	if os.IsNotExist(err) || os.IsPermission(err) {
+	// Check whether this URL is mapped to an SCGI app
+	for scgiPath, scgiSocket := range sysConfig.SCGIPaths {
+		if strings.HasPrefix(URL.Path, scgiPath) {
+			handleSCGI(URL, scgiPath, scgiSocket, sysConfig, &logEntry, conn)
+			return
+		}
+	}
+
+	// Okay, at this point we really are committed to looking on disk for `path`.
+	// Make sure it exists, and is world readable, and if it's a symbolic link,
+	// follow it and check these things again!
+	rawPath := path
+	var info os.FileInfo
+	for {
+		info, err = os.Stat(path)
+		if os.IsNotExist(err) || os.IsPermission(err) {
+			conn.Write([]byte("51 Not found!\r\n"))
+			logEntry.Status = 51
+			return
+		} else if err != nil {
+			log.Println("Error getting info for file " + path + ": " + err.Error())
+			conn.Write([]byte("40 Temporary failure!\r\n"))
+			logEntry.Status = 40
+			return
+		} else if uint64(info.Mode().Perm())&0444 != 0444 {
+			conn.Write([]byte("51 Not found!\r\n"))
+			logEntry.Status = 51
+			return
+		}
+		newPath, err := filepath.EvalSymlinks(path)
+		if err!= nil {
+			log.Println("Error evaluating path " + path + " for symlinks: " + err.Error())
+			conn.Write([]byte("51 Not found!\r\n"))
+			logEntry.Status = 51
+			return
+		}
+		if newPath == path {
+			break
+		}
+		path = newPath
+	}
+
+	// If symbolic links have been used to escape the intended document directory,
+	// deny all knowledge
+	isSub, err := isSubdir(path, sysConfig.DocBase)
+	if err != nil {
+		log.Println("Error testing whether path " + path + " is below DocBase: " + err.Error())
+	}
+	if !isSub {
+		log.Println("Refusing to follow symlink from " + rawPath + " outside of DocBase!")
+	}
+	if err != nil || !isSub {
 		conn.Write([]byte("51 Not found!\r\n"))
-		log.Status = 51
-		return
-	} else if err != nil {
-		errorLog.Println("Error getting info for file " + path + ": " + err.Error())
-		conn.Write([]byte("40 Temporary failure!\r\n"))
-		log.Status = 40
-		return
-	} else if uint64(info.Mode().Perm())&0444 != 0444 {
-		conn.Write([]byte("51 Not found!\r\n"))
-		log.Status = 51
+		logEntry.Status = 51
 		return
 	}
 
-	// Finally, serve the file or directory
+	// Refuse to serve sensitive files even if they are inside DocBase and
+	// world-readable because if they are it's likely a mistake
+	if path == sysConfig.KeyPath || path == sysConfig.AccessLog || path == sysConfig.ErrorLog || filepath.Base(path) == ".molly" {
+		conn.Write([]byte("51 Not found!\r\n"))
+		logEntry.Status = 51
+		return
+	}
+
+	// Finally, serve a simple static file or directory
 	if info.IsDir() {
-		serveDirectory(URL, path, &log, conn, config, errorLog)
+		serveDirectory(URL, path, &logEntry, conn, config)
 	} else {
-		serveFile(path, &log, conn, config, errorLog)
+		serveFile(path, info, &logEntry, conn, config)
 	}
 }
 
-func readRequest(conn net.Conn, log *LogEntry, errorLog *log.Logger) (*url.URL, error) {
+func readRequest(conn net.Conn, logEntry *LogEntry) (*url.URL, error) {
+	err := conn.SetReadDeadline(time.Now().Add(30 * time.Second))
+	if err != nil {
+		log.Println("Error setting read deadline: " + err.Error())
+		return nil, err
+	}
+
 	reader := bufio.NewReaderSize(conn, 1024)
 	request, overflow, err := reader.ReadLine()
+
 	if overflow {
 		conn.Write([]byte("59 Request too long!\r\n"))
-		log.Status = 59
+		logEntry.Status = 59
 		return nil, errors.New("Request too long")
 	} else if err != nil {
-		errorLog.Println("Error reading request from " + conn.RemoteAddr().String() + ": " + err.Error())
-		conn.Write([]byte("40 Unknown error reading request!\r\n"))
-		log.Status = 40
-		return nil, errors.New("Error reading request")
+		if errors.Is(err, os.ErrDeadlineExceeded) {
+			conn.Write([]byte("40 Request timed out!\r\n"))
+		} else {
+			log.Println("Error reading request from " + conn.RemoteAddr().String() + ": " + err.Error())
+			conn.Write([]byte("40 Unknown error reading request!\r\n"))
+		}
+		logEntry.Status = 40
+		return nil, err
 	}
 
 	// Parse request as URL
 	URL, err := url.Parse(string(request))
 	if err != nil {
-		errorLog.Println("Error parsing request URL " + string(request) + ": " + err.Error())
+		log.Println("Error parsing request URL " + string(request) + ": " + err.Error())
 		conn.Write([]byte("59 Error parsing URL!\r\n"))
-		log.Status = 59
+		logEntry.Status = 59
 		return nil, errors.New("Bad URL in request")
 	}
-	log.RequestURL = URL.String()
+	logEntry.RequestURL = URL.String()
 
 	// Set implicit scheme
 	if URL.Scheme == "" {
@@ -165,7 +263,7 @@ func readRequest(conn net.Conn, log *LogEntry, errorLog *log.Logger) (*url.URL,
 	return URL, nil
 }
 
-func resolvePath(path string, config Config) string {
+func resolvePath(path string, config SysConfig) string {
 	// Handle tildes
 	if strings.HasPrefix(path, "/~") {
 		bits := strings.Split(path, "/")
@@ -179,58 +277,65 @@ func resolvePath(path string, config Config) string {
 	return path
 }
 
-func handleRedirects(URL *url.URL, config Config, conn net.Conn, log *LogEntry, errorLog *log.Logger) {
-	handleRedirectsInner(URL, config.TempRedirects, 30, conn, log, errorLog)
-	handleRedirectsInner(URL, config.PermRedirects, 31, conn, log, errorLog)
+func handleRedirects(URL *url.URL, config UserConfig, conn net.Conn, logEntry *LogEntry) {
+	handleRedirectsInner(URL, config.TempRedirects, 30, conn, logEntry)
+	handleRedirectsInner(URL, config.PermRedirects, 31, conn, logEntry)
 }
 
-func handleRedirectsInner(URL *url.URL, redirects map[string]string, status int, conn net.Conn, log *LogEntry, errorLog *log.Logger) {
+func handleRedirectsInner(URL *url.URL, redirects map[string]string, status int, conn net.Conn, logEntry *LogEntry) {
 	strStatus := strconv.Itoa(status)
 	for src, dst := range redirects {
 		compiled, err := regexp.Compile(src)
 		if err != nil {
-			errorLog.Println("Error compiling redirect regexp " + src + ": " + err.Error())
+			log.Println("Error compiling redirect regexp " + src + ": " + err.Error())
 			continue
 		}
 		if compiled.MatchString(URL.Path) {
-			URL.Path = compiled.ReplaceAllString(URL.Path, dst)
-			conn.Write([]byte(strStatus + " " + URL.String() + "\r\n"))
-			log.Status = status
+			new_target := compiled.ReplaceAllString(URL.Path, dst)
+			if !strings.HasPrefix(new_target, "gemini://") {
+				URL.Path = new_target
+				new_target = URL.String()
+			}
+			conn.Write([]byte(strStatus + " " + new_target + "\r\n"))
+			logEntry.Status = status
 			return
 		}
 	}
 }
 
-func serveDirectory(URL *url.URL, path string, log *LogEntry, conn net.Conn, config Config, errorLog *log.Logger) {
+func serveDirectory(URL *url.URL, path string, logEntry *LogEntry, conn net.Conn, config UserConfig) {
 	// Redirect to add trailing slash if missing
 	// (otherwise relative links don't work properly)
 	if !strings.HasSuffix(URL.Path, "/") {
 		URL.Path += "/"
 		conn.Write([]byte(fmt.Sprintf("31 %s\r\n", URL.String())))
-		log.Status = 31
+		logEntry.Status = 31
 		return
 	}
 	// Check for index.gmi if path is a directory
 	index_path := filepath.Join(path, "index."+config.GeminiExt)
 	index_info, err := os.Stat(index_path)
 	if err == nil && uint64(index_info.Mode().Perm())&0444 == 0444 {
-		serveFile(index_path, log, conn, config, errorLog)
+		serveFile(index_path, index_info, logEntry, conn, config)
 		// Serve a generated listing
-	} else {
+	} else if config.DirectoryListing {
 		listing, err := generateDirectoryListing(URL, path, config)
 		if err != nil {
-			errorLog.Println("Error generating listing for directory " + path + ": " + err.Error())
+			log.Println("Error generating listing for directory " + path + ": " + err.Error())
 			conn.Write([]byte("40 Server error!\r\n"))
-			log.Status = 40
+			logEntry.Status = 40
 			return
 		}
 		conn.Write([]byte("20 text/gemini\r\n"))
-		log.Status = 20
+		logEntry.Status = 20
 		conn.Write([]byte(listing))
+	} else {
+		conn.Write([]byte("51 Not found!\r\n"))
+		logEntry.Status = 51
 	}
 }
 
-func serveFile(path string, log *LogEntry, conn net.Conn, config Config, errorLog *log.Logger) {
+func serveFile(path string, info os.FileInfo, logEntry *LogEntry, conn net.Conn, config UserConfig) {
 	// Get MIME type of files
 	ext := filepath.Ext(path)
 	var mimeType string
@@ -239,6 +344,7 @@ func serveFile(path string, log *LogEntry, conn net.Conn, config Config, errorLo
 	} else {
 		mimeType = mime.TypeByExtension(ext)
 	}
+
 	// Override extension-based MIME type
 	for pathRegex, newType := range config.MimeOverrides {
 		overridden, err := regexp.Match(pathRegex, []byte(path))
@@ -246,25 +352,79 @@ func serveFile(path string, log *LogEntry, conn net.Conn, config Config, errorLo
 			mimeType = newType
 		}
 	}
-	// Set a generic MIME type if the extension wasn't recognised
-	if mimeType == "" {
-		mimeType = "application/octet-stream"
-	}
-	// Add lang parameter
-	if mimeType == "text/gemini" && config.DefaultLang != "" {
-		mimeType += "; lang=" + config.DefaultLang
-	}
 
+	// Try to open the file
 	f, err := os.Open(path)
 	if err != nil {
-		errorLog.Println("Error reading file " + path + ": " + err.Error())
+		log.Println("Error reading file " + path + ": " + err.Error())
 		conn.Write([]byte("50 Error!\r\n"))
-		log.Status = 50
+		logEntry.Status = 50
 		return
 	}
 	defer f.Close()
 
+	// If the file extension wasn't recognised, or there's not one, use bytes
+	// from the now open file to sniff!
+	if mimeType == "" {
+		buffer := make([]byte, 512)
+		n, err := f.Read(buffer)
+		if err == nil {
+			_, err = f.Seek(0, 0)
+		}
+		if err != nil {
+			log.Println("Error peeking into file " + path + ": " + err.Error())
+			conn.Write([]byte("50 Error!\r\n"))
+			logEntry.Status = 50
+			return
+		}
+		mimeType = http.DetectContentType(buffer[0:n])
+	}
+
+	// Add charset parameter
+	if strings.HasPrefix(mimeType, "text/gemini") && config.DefaultEncoding != "" {
+		mimeType += "; charset=" + config.DefaultEncoding
+	}
+	// Add lang parameter
+	if strings.HasPrefix(mimeType, "text/gemini") && config.DefaultLang != "" {
+		mimeType += "; lang=" + config.DefaultLang
+	}
+
+	// Derive a maximum allowed download time from the filesyize.
+	// Assume non-malicious clients can manage an average of 0.5 KB/s or better.
+	// But always allow at least 30 seconds
+	allowedTime := int(info.Size() / 512)
+	if allowedTime < 30 {
+		allowedTime = 30
+	}
+	err = conn.SetWriteDeadline(time.Now().Add(time.Duration(allowedTime) * time.Second))
+	if err != nil {
+		log.Println("Error setting write deadline: " + err.Error())
+		conn.Write([]byte("40 Error!\r\n"))
+		logEntry.Status = 40
+		return
+	}
+
+	// Send response
 	conn.Write([]byte(fmt.Sprintf("20 %s\r\n", mimeType)))
-	io.Copy(conn, f)
-	log.Status = 20
+	_, err = io.Copy(conn, f)
+	if err != nil {
+		// Prepare to close the connection *without* TLS Close Notify so the client
+		// knows something has gone wrong!
+		tlsConn, _ := conn.(*tls.Conn)
+		netConn := tlsConn.NetConn()
+		tcpConn := netConn.(*net.TCPConn)
+		remoteAddr := conn.RemoteAddr().String()
+		if errors.Is(err, os.ErrDeadlineExceeded) {
+			log.Println("Writing to " + remoteAddr + " timed out.")
+			// Make sure Close() below takes immediate effect in
+			// the case of a timeout as a defence against
+			// socket exhaustion attacks
+			tcpConn.SetLinger(0)
+		} else {
+			log.Println("Error writing response to " + remoteAddr + ": " + err.Error())
+		}
+		tcpConn.Close()
+		return
+	}
+	logEntry.Status = 20
 }

launch.go

@@ -0,0 +1,185 @@
+package main
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/pem"
+	"io/ioutil"
+	"log"
+	"os"
+	"os/signal"
+	"strconv"
+	"sync"
+	"syscall"
+	"time"
+)
+
+var VERSION = "0.0.0"
+
+func launch(sysConfig SysConfig, userConfig UserConfig, privInfo userInfo) int {
+	var err error
+
+	// Open log files
+	if sysConfig.ErrorLog != "" {
+		errorLogFile, err := os.OpenFile(sysConfig.ErrorLog, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
+		if err != nil {
+			log.Println("Error opening error log file: " + err.Error())
+			return 1
+		}
+		defer errorLogFile.Close()
+		log.SetOutput(errorLogFile)
+	}
+	log.SetFlags(log.Ldate|log.Ltime)
+
+	var accessLogFile *os.File
+	if sysConfig.AccessLog == "-" {
+		accessLogFile = os.Stdout
+	} else if sysConfig.AccessLog != "" {
+		accessLogFile, err = os.OpenFile(sysConfig.AccessLog, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
+		if err != nil {
+			log.Println("Error opening access log file: " + err.Error())
+			return 1
+		}
+		defer accessLogFile.Close()
+	}
+
+	// Read TLS files, create TLS config
+	// Check key file permissions first
+	info, err := os.Stat(sysConfig.KeyPath)
+	if err != nil {
+		log.Println("Error opening TLS key file: " + err.Error())
+		return 1
+	}
+	if uint64(info.Mode().Perm())&0444 == 0444 {
+		log.Println("Refusing to use world-readable TLS key file " + sysConfig.KeyPath)
+		return 1
+	}
+	// Check certificate hostname matches server hostname
+	info, err = os.Stat(sysConfig.CertPath)
+	if err != nil {
+		log.Println("Error opening TLS certificate file: " + err.Error())
+		return 1
+	}
+	certFile, err := os.Open(sysConfig.CertPath)
+	if err != nil {
+		log.Println("Error opening TLS certificate file: " + err.Error())
+		return 1
+	}
+	certBytes, err := ioutil.ReadAll(certFile)
+	if err != nil {
+		log.Println("Error reading TLS certificate file: " + err.Error())
+		return 1
+	}
+	certDer, _ := pem.Decode(certBytes)
+	if certDer == nil {
+		log.Println("Error decoding TLS certificate file: " + err.Error())
+		return 1
+	}
+	certx509, err := x509.ParseCertificate(certDer.Bytes)
+	if err != nil {
+		log.Println("Error parsing TLS certificate: " + err.Error())
+		return 1
+	}
+	err = certx509.VerifyHostname(sysConfig.Hostname)
+	if err != nil {
+		log.Println("Invalid TLS certificate: " + err.Error())
+		return 1
+	}
+	// Warn if certificate is expired
+	now := time.Now()
+	if now.After(certx509.NotAfter) {
+		log.Println("Hey, your certificate expired on " + certx509.NotAfter.String() + "!!!")
+	}
+
+	// Load certificate and private key
+	cert, err := tls.LoadX509KeyPair(sysConfig.CertPath, sysConfig.KeyPath)
+	if err != nil {
+		log.Println("Error loading TLS keypair: " + err.Error())
+		return 1
+	}
+	var tlscfg tls.Config
+	tlscfg.Certificates = []tls.Certificate{cert}
+	tlscfg.ClientAuth = tls.RequestClientCert
+	if sysConfig.AllowTLS12 {
+		tlscfg.MinVersion = tls.VersionTLS12
+	} else {
+		tlscfg.MinVersion = tls.VersionTLS13
+	}
+	if len(userConfig.CertificateZones) > 0 || sysConfig.ReadMollyFiles {
+		tlscfg.ClientAuth = tls.RequestClientCert
+	}
+
+	// Try to chdir to /, so we don't block any mountpoints
+	// But if we can't for some reason it's no big deal
+        err = os.Chdir("/")
+        if err != nil {
+                log.Println("Could not change working directory to /: " + err.Error())
+        }
+
+	// Apply security restrictions
+	err = enableSecurityRestrictions(sysConfig, privInfo)
+	if err != nil {
+		log.Println("Exiting due to failure to apply security restrictions.")
+		return 1
+	}
+
+	// Create TLS listener
+	listener, err := tls.Listen("tcp", ":"+strconv.Itoa(sysConfig.Port), &tlscfg)
+	if err != nil {
+		log.Println("Error creating TLS listener: " + err.Error())
+		return 1
+	}
+	defer listener.Close()
+
+	// Start log handling routines
+	var accessLogEntries chan LogEntry
+	if sysConfig.AccessLog == "" {
+		accessLogEntries = nil
+	} else {
+		accessLogEntries = make(chan LogEntry, 10)
+		go func() {
+			for {
+				entry := <-accessLogEntries
+				if entry.Status != 0 {
+					writeLogEntry(accessLogFile, entry)
+				}
+			}
+		}()
+	}
+
+	// Start listening for signals
+	shutdown := make(chan struct{})
+	sigterm := make(chan os.Signal, 1)
+	signal.Notify(sigterm, syscall.SIGTERM)
+	go func() {
+		<-sigterm
+		log.Println("Caught SIGTERM.  Waiting for handlers to finish...")
+		close(shutdown)
+		listener.Close()
+	}()
+
+	// Infinite serve loop (SIGTERM breaks out)
+	running := true
+	var wg sync.WaitGroup
+	rl := newRateLimiter(sysConfig.RateLimitAverage, sysConfig.RateLimitSoft, sysConfig.RateLimitHard)
+	for running {
+		conn, err := listener.Accept()
+		if err == nil {
+			wg.Add(1)
+			go handleGeminiRequest(conn, sysConfig, userConfig, accessLogEntries, &rl, &wg)
+		} else {
+			select {
+			case <-shutdown:
+				running = false
+			default:
+				log.Println("Error accepting connection: " + err.Error())
+			}
+		}
+	}
+	// Wait for still-running handler Go routines to finish
+	wg.Wait()
+	log.Println("Exiting.")
+
+	// Exit successfully
+	return 0
+}

main.go

@@ -1,92 +1,36 @@
+// +build js nacl plan9 windows
+
 package main
 
 import (
-	"crypto/tls"
 	"flag"
+	"fmt"
 	"log"
 	"os"
-	"strconv"
 )
 
 func main() {
 	var conf_file string
+	var version bool
 
-	// Parse args and read config
-	flag.StringVar(&conf_file, "c", "", "Path to config file")
+	// Parse args
+	flag.StringVar(&conf_file, "c", "/etc/molly.conf", "Path to config file")
+	flag.BoolVar(&version, "v", false, "Print version and exit")
 	flag.Parse()
-	if conf_file == "" {
-		_, err := os.Stat("/etc/molly.conf")
-		if err == nil {
-			conf_file = "/etc/molly.conf"
-		}
+
+	// If requested, print version and exit
+	if version {
+		fmt.Println("Molly Brown version", VERSION)
+		os.Exit(0)
 	}
-	config, err := getConfig(conf_file)
+
+	// Read config
+	sysConfig, userConfig, err := getConfig(conf_file)
 	if err != nil {
 		log.Fatal(err)
 	}
 
-	// Open log files
-	var errorLogFile *os.File
-	if config.ErrorLog == "-" {
-		errorLogFile = os.Stdout
-	} else {
-		errorLogFile, err = os.OpenFile(config.ErrorLog, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
-		if err != nil {
-			log.Fatal(err)
-		}
-		defer errorLogFile.Close()
-	}
-	errorLog := log.New(errorLogFile, "", log.Ldate | log.Ltime)
-
-	var accessLogFile *os.File
-	if config.AccessLog == "-" {
-		accessLogFile = os.Stdout
-	} else {
-		accessLogFile, err = os.OpenFile(config.AccessLog, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
-		if err != nil {
-			errorLog.Println("Error opening access log file: " + err.Error())
-			log.Fatal(err)
-		}
-		defer accessLogFile.Close()
-	}
-
-	// Read TLS files, create TLS config
-	cert, err := tls.LoadX509KeyPair(config.CertPath, config.KeyPath)
-	if err != nil {
-		errorLog.Println("Error loading TLS keypair: " + err.Error())
-		log.Fatal(err)
-	}
-	tlscfg := &tls.Config{
-		Certificates: []tls.Certificate{cert},
-		MinVersion:   tls.VersionTLS12,
-		ClientAuth:   tls.RequestClientCert,
-	}
-
-	// Create TLS listener
-	listener, err := tls.Listen("tcp", ":"+strconv.Itoa(config.Port), tlscfg)
-	if err != nil {
-		errorLog.Println("Error creating TLS listener: " + err.Error())
-		log.Fatal(err)
-	}
-	defer listener.Close()
-
-	// Start log handling routines
-	accessLogEntries := make(chan LogEntry, 10)
-	go func() {
-		for {
-			entry := <-accessLogEntries
-			writeLogEntry(accessLogFile, entry)
-		}
-	}()
-
-	// Infinite serve loop
-	for {
-		conn, err := listener.Accept()
-		if err != nil {
-			errorLog.Println("Error accepting connection: " + err.Error())
-			log.Fatal(err)
-		}
-		go handleGeminiRequest(conn, config, accessLogEntries, errorLog)
-	}
-
+	// Run server and exit
+	var dummy userInfo
+	os.Exit(launch(sysConfig, userConfig, dummy))
 }

main_unix.go

@@ -0,0 +1,55 @@
+// +build aix darwin dragonfly freebsd illumos linux netbsd openbsd solaris
+
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"os"
+	"syscall"
+)
+
+func main() {
+	var conf_file string
+	var chroot string
+	var user string
+	var version bool
+
+	// Parse args
+	flag.StringVar(&conf_file, "c", "/etc/molly.conf", "Path to config file")
+	flag.StringVar(&chroot, "C", "", "Path to chroot into")
+	flag.StringVar(&user, "u", "nobody", "Unprivileged user")
+	flag.BoolVar(&version, "v", false, "Print version and exit")
+	flag.Parse()
+
+	// If requested, print version and exit
+	if version {
+		fmt.Println("Molly Brown version", VERSION)
+		os.Exit(0)
+	}
+
+	// Read config
+	sysConfig, userConfig, err := getConfig(conf_file)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	// Read user info
+	privInfo, err := getUserInfo(user)
+
+	// Chroot, if asked
+	if chroot != "" {
+		err := syscall.Chroot(chroot)
+		if err == nil {
+			err = os.Chdir("/")
+		}
+		if err != nil {
+			log.Println("Could not chroot to " + chroot + ": " + err.Error())
+			os.Exit(1)
+		}
+	}
+
+	// Run server and exit
+	os.Exit(launch(sysConfig, userConfig, privInfo))
+}

ratelim.go

@@ -0,0 +1,87 @@
+package main
+
+import (
+	"log"
+	"sync"
+	"strconv"
+	"time"
+)
+
+type RateLimiter struct {
+	mu sync.Mutex
+	bucket map[string]int
+	bans map[string]time.Time
+	banCounts map[string]int
+	rate int
+	softLimit int
+	hardLimit int
+}
+
+func newRateLimiter(rate int, softLimit int, hardLimit int) RateLimiter {
+	var rl = new(RateLimiter)
+	rl.bucket = make(map[string]int)
+	rl.bans = make(map[string]time.Time)
+	rl.banCounts = make(map[string]int)
+	rl.rate = rate
+	rl.softLimit = softLimit
+	rl.hardLimit = hardLimit
+
+	// Leak periodically
+	go func () {
+		for(true) {
+			rl.mu.Lock()
+			// Leak the buckets
+			for addr, drips := range rl.bucket {
+				if drips <= rate {
+					delete(rl.bucket, addr)
+				} else {
+					rl.bucket[addr] = drips - rl.rate
+				}
+			}
+			// Expire bans
+			now := time.Now()
+			for addr, expiry := range rl.bans {
+				if now.After(expiry) {
+					delete(rl.bans, addr)
+				}
+			}
+
+			// Wait
+			rl.mu.Unlock()
+			time.Sleep(time.Second)
+		}
+	}()
+	return *rl
+}
+
+func  (rl *RateLimiter) softLimited(addr string) (int, bool) {
+	rl.mu.Lock()
+	defer rl.mu.Unlock()
+	drips, present := rl.bucket[addr]
+	if !present {
+		rl.bucket[addr] = 1
+		return 1, false
+	}
+	drips += 1
+	rl.bucket[addr] = drips
+	if drips > rl.hardLimit {
+		banCount, present := rl.banCounts[addr]
+		if present {
+			banCount += 1
+		} else {
+			banCount = 1
+		}
+		rl.banCounts[addr] = banCount
+		banDuration := 1 << (banCount - 1)
+		now := time.Now()
+		expiry := now.Add(time.Duration(banDuration)*time.Hour)
+		rl.bans[addr] = expiry
+		log.Println("Banning " + addr + " for " + strconv.Itoa(banDuration) + " hours due to ignoring rate limiting.")
+	}
+	return drips, drips > rl.softLimit
+}
+
+func  (rl *RateLimiter) hardLimited(addr string) bool {
+	_, present := rl.bans[addr]
+	return present
+}

security.go

@@ -0,0 +1,14 @@
+// +build js nacl plan9 windows
+
+package main
+
+type userInfo struct {
+}
+
+// Restrict access to the files specified in config in an OS-dependent way.
+// This is intended to be called immediately prior to accepting client
+// connections and may be used to establish a security "jail" for the molly
+// brown executable.
+func enableSecurityRestrictions(config SysConfig, ui userInfo) error {
+	return nil
+}

security_dropprivs.go

@@ -0,0 +1,119 @@
+// +build linux,go1.16 aix darwin dragonfly freebsd illumos netbsd openbsd solaris
+
+package main
+
+import (
+	"log"
+	"os"
+	"os/user"
+	"strconv"
+	"syscall"
+)
+
+type userInfo struct {
+	uid int
+	euid int
+	gid int
+	egid int
+	supp_groups []int
+	is_setuid bool
+	is_setgid bool
+	root_user bool
+	root_prim_group bool
+	root_supp_group bool
+	need_drop bool
+	unpriv_uid int
+	unpriv_gid int
+}
+
+func getUserInfo(unprivUser string) (userInfo, error) {
+	var ui userInfo
+	ui.uid = os.Getuid()
+	ui.euid = os.Geteuid()
+	ui.gid = os.Getgid()
+	ui.egid = os.Getegid()
+	supp_groups, err := os.Getgroups()
+	if err != nil {
+		log.Println("Could not get supplementary groups: ", err.Error())
+		return ui, err
+	}
+	ui.supp_groups = supp_groups
+	ui.unpriv_uid = -1
+	ui.unpriv_gid = -1
+
+	ui.is_setuid = ui.uid != ui.euid
+	ui.is_setgid = ui.gid != ui.egid
+	ui.root_user = ui.uid == 0 || ui.euid == 0
+	ui.root_prim_group = ui.gid == 0 || ui.egid == 0
+	for _, gid := range ui.supp_groups {
+		if gid == 0 {
+			ui.root_supp_group = true
+			break
+		}
+	}
+	ui.need_drop = ui.is_setuid || ui.is_setgid || ui.root_user || ui.root_prim_group || ui.root_supp_group
+
+	if ui.root_user || ui.root_prim_group {
+		nobody_user, err := user.Lookup(unprivUser)
+		if err != nil {
+			log.Println("Running as root but could not lookup UID for user " + unprivUser + ": " + err.Error())
+			return ui, err
+		}
+		ui.unpriv_uid, err = strconv.Atoi(nobody_user.Uid)
+		ui.unpriv_gid, err = strconv.Atoi(nobody_user.Gid)
+		if err != nil {
+			log.Println("Running as root but could not lookup UID for user " + unprivUser + ": " + err.Error())
+			return ui, err
+		}
+	}
+
+	return ui, nil
+}
+func DropPrivs(ui userInfo) error {
+
+	// If we're already unprivileged, all good
+	if !ui.need_drop {
+		return nil
+	}
+
+	// Drop supplementary groups
+	if ui.root_supp_group {
+		err := syscall.Setgroups([]int{})
+		if err != nil {
+			log.Println("Could not unset supplementary groups: " + err.Error())
+			return err
+		}
+	}
+
+	// Setgid()
+	if ui.root_prim_group || ui.is_setgid {
+		var target_gid int
+		if ui.root_prim_group {
+			target_gid = ui.unpriv_gid
+		} else {
+			target_gid = ui.gid
+		}
+		err := syscall.Setgid(target_gid)
+		if err != nil {
+			log.Println("Could not setgid to " + strconv.Itoa(target_gid) + ": " + err.Error())
+			return err
+		}
+	}
+
+	// Setuid()
+	if ui.root_user || ui.is_setuid {
+		var target_uid int
+		if ui.root_user {
+			target_uid = ui.unpriv_uid
+		} else {
+			target_uid = ui.uid
+		}
+		err := syscall.Setuid(target_uid)
+		if err != nil {
+			log.Println("Could not setuid to " + strconv.Itoa(target_uid) + ": " + err.Error())
+			return err
+		}
+	}
+
+	return nil
+}

security_oldgolinux.go

@@ -0,0 +1,32 @@
+// +build linux,!go1.16
+
+package main
+
+import (
+	"errors"
+	"log"
+	"os"
+)
+
+type userInfo struct {
+}
+
+func getUserInfo(unprivUser string) (userInfo, error) {
+       var dummy userInfo
+       return dummy, nil
+}
+
+func enableSecurityRestrictions(config SysConfig, ui userInfo) error {
+
+	// Prior to Go 1.6, setuid did not work reliably on Linux
+	// So, absolutely refuse to run as root
+	uid := os.Getuid()
+	euid := os.Geteuid()
+	if uid == 0 || euid == 0 {
+		setuid_err := "Refusing to run with root privileges when setuid() will not work!"
+		log.Println(setuid_err)
+		return errors.New(setuid_err)
+	}
+
+	return nil
+}

security_openbsd.go

@@ -0,0 +1,77 @@
+package main
+
+import (
+	"golang.org/x/sys/unix"
+	"log"
+	"path/filepath"
+)
+
+// Restrict access to the files specified in config in an OS-dependent way.
+// The OpenBSD implementation uses pledge(2) and unveil(2) to restrict the
+// operations available to the molly brown executable. Please note that (S)CGI
+// processes that molly brown spawns or communicates with are unrestricted
+// and should pledge their own restrictions and unveil their own files.
+func enableSecurityRestrictions(config SysConfig, ui userInfo) error {
+
+	// Setuid to an unprivileged user
+	err := DropPrivs(ui)
+	if err != nil {
+		return err
+	}
+
+	// Unveil the configured document base as readable.
+	log.Println("Unveiling \"" + config.DocBase + "\" as readable.")
+	err = unix.Unveil(config.DocBase, "r")
+	if err != nil {
+		log.Println("Could not unveil DocBase: " + err.Error())
+		return err
+	}
+
+	// Unveil cgi path globs as executable.
+	for _, cgiPath := range config.CGIPaths {
+		cgiGlobbedPaths, err := filepath.Glob(cgiPath)
+		for _, cgiGlobbedPath := range cgiGlobbedPaths {
+			log.Println("Unveiling \"" + cgiGlobbedPath + "\" as executable.")
+			err = unix.Unveil(cgiGlobbedPath, "rx")
+			if err != nil {
+				log.Println("Could not unveil CGIPaths: " + err.Error())
+				return err
+			}
+		}
+	}
+
+	// Unveil scgi socket paths as readable and writeable.
+	for _, scgiSocket := range config.SCGIPaths {
+		log.Println("Unveiling \"" + scgiSocket + "\" as read/write.")
+		err = unix.Unveil(scgiSocket, "rw")
+		if err != nil {
+			return err
+		}
+	}
+
+	// Finalize the unveil list.
+	// Any files not whitelisted above won't be accessible to molly brown.
+	err = unix.UnveilBlock()
+	if err != nil {
+		log.Println("Could not block unveil: " + err.Error())
+		return err
+	}
+
+	// Pledge to only use stdio, inet, and rpath syscalls.
+	promises := "stdio inet rpath"
+	if len(config.CGIPaths) > 0 {
+		// If CGI paths have been specified, also allow exec syscalls.
+		promises += " exec proc"
+	}
+	if len(config.SCGIPaths) > 0 {
+		// If SCGI paths have been specified, also allow unix sockets.
+		promises += " unix"
+	}
+	err = unix.PledgePromises(promises)
+	if err != nil {
+		log.Println("Could not pledge: " + err.Error())
+		return err
+	}
+
+	return nil
+}

security_other_unix.go

@@ -0,0 +1,10 @@
+// +build linux,go1.16 aix darwin dragonfly freebsd illumos netbsd solaris
+
+package main
+
+func enableSecurityRestrictions(config SysConfig, ui userInfo) error {
+
+	// Setuid to an unprivileged user
+	return DropPrivs(ui)
+
+}