Merge branch 'master' into desugar

Readme.md

@@ -2,9 +2,24 @@
 
 * Not designed to operate in large clusters providing services for millions of
   people.
-* Designed for _you_, to run one computer. (Or a few.)
+* Designed for _you_, to run one computer. (Or a few.) Running the code you
+  want to run, and nothing else.
 
-Goals (in priority order):
+  ```sh
+  $ git clone https://github.com/akkartik/mu
+  $ cd mu
+  # package up a "hello world" binary and Linux kernel into mu.iso
+  $ ./gen_iso examples/ex6.subx
+  # try it out
+  $ qemu-system-x86_64 -m 256M -cdrom mu.iso -boot d
+  # print the message followed by kernel panic
+  ```
+
+[![Build Status](https://api.travis-ci.org/akkartik/mu.svg?branch=master)](https://travis-ci.org/akkartik/mu)
+
+## Goals
+
+In priority order:
 
 * [Reward curiosity.](http://akkartik.name/about)
   * Easy to build, easy to run. [Minimal dependencies](https://news.ycombinator.com/item?id=16882140#16882555),
@@ -21,7 +36,8 @@ Goals (in priority order):
   * Memory leaks over memory corruption.
 * Teach the computer bottom-up.
 
-Non-goals:
+## Non-goals
+
 * Efficiency. Clear programs over fast programs.
 * Portability. Runs on any computer as long as it's x86.
 * Compatibility. The goal is to get off mainstream stacks, not to perpetuate
@@ -30,49 +46,39 @@ Non-goals:
   For now it's a thin veneer over machine code. I'm working on memory safety
   before expressive syntax.
 
-So far I have a self-hosted tool (SubX) for writing thoroughly tested x86
-machine code atop a bare Linux kernel.
-Eventually you will be able to program in higher-level notations.
-Eventually Mu won't need Linux or C.
-Eventually the OS interfaces for screen, keyboard, file system and network
-will be _dependency-injected_ so that tests can easily insert a fake screen,
-keyboard, file system or network.
+## What works so far
 
-The rest of this Readme describes SubX.
-
-## SubX is a simple, minimalist stack for programming your computer.
+You get a thin syntax called SubX for programming in (a subset of) x86 machine
+code. Here's a program (`examples/ex1.subx`) that returns 42:
 
   ```sh
-  $ git clone https://github.com/akkartik/mu
-  $ cd mu
-  $ ./subx  # print out a help message
+  bb/copy-to-EBX  0x2a/imm32  # 42 in hex
+  b8/copy-to-EAX  1/imm32/exit
+  cd/syscall  0x80/imm8
   ```
 
-SubX requires a Unix-like environment with a C++ compiler (Linux or BSD or Mac
-OS). Running `subx` will transparently compile it as necessary.
-
-[![Build Status](https://api.travis-ci.org/akkartik/mu.svg?branch=master)](https://travis-ci.org/akkartik/mu)
-
-You can generate native ELF binaries with it that run on a bare Linux
-kernel. No other dependencies needed.
+You can generate tiny zero-dependency ELF binaries with it that run on Linux.
 
   ```sh
-  $ ./subx translate examples/ex1.subx -o examples/ex1
+  $ ./subx translate examples/ex1.subx -o examples/ex1  # on Linux or BSD or Mac
   $ ./examples/ex1  # only on Linux
   $ echo $?
   42
  ```
 
+(Running `subx` requires a C++ compiler, transparently invoking it as
+necessary.)
+
 You can run the generated binaries on an interpreter/VM for better error
 messages.
 
   ```sh
-  $ ./subx run examples/ex1  # on Linux or BSD or OS X
+  $ ./subx run examples/ex1  # on Linux or BSD or Mac
   $ echo $?
   42
   ```
 
-Emulated runs generate a trace that permits [time-travel debugging](https://github.com/akkartik/mu/blob/master/browse_trace/Readme.md).
+Emulated runs can generate a trace that permits [time-travel debugging](https://github.com/akkartik/mu/blob/master/browse_trace/Readme.md).
 
   ```sh
   $ ./subx --debug translate examples/factorial.subx -o examples/factorial
@@ -85,8 +91,8 @@ Emulated runs generate a trace that permits [time-travel debugging](https://gith
   $ ../browse_trace/browse_trace last_run  # text-mode debugger UI
   ```
 
-You can write tests for your assembly programs. The entire stack is thoroughly
-covered by automated tests. SubX's tagline: tests before syntax.
+You can write tests for your programs. The entire stack is thoroughly covered
+by automated tests. SubX's tagline: tests before syntax.
 
   ```sh
   $ ./subx test
@@ -129,9 +135,24 @@ Or, running in a VM on other platforms:
   42
   ```
 
+Finally, as described at the top, you can turn it into a bootable disk image
+containing just your code and a Linux kernel. You can run the disk image on a
+cloud server that supports custom images. [Instructions for Linode.](http://akkartik.name/post/iso-on-linode)
+
+  ```sh
+  $ sudo apt install build-essential flex bison wget libelf-dev libssl-dev xorriso
+  $ ./gen_iso examples/ex6.subx
+  $ qemu-system-x86_64 -m 256M -cdrom mu.iso -boot d
+  ```
+
+(`gen_iso` only came into existence 2019-08-09, and has a flabby laundry list
+of dependencies that I will gradually prune. It currently takes 12 minutes to
+run on a single core with 8 GB RAM, mostly to compile its fork of the Linux
+kernel.)
+
 ## What it looks like
 
-Here is the first example we ran above, a program that just returns 42:
+Here is the above example again:
 
   ```sh
   bb/copy-to-EBX  0x2a/imm32  # 42 in hex
@@ -167,7 +188,7 @@ a few registers:
   - carry flag CF
 
 SubX programs consist of instructions like `89/copy`, `01/add`, `3d/compare`
-and `52/push-ECX` which modify these registers as well as a byte-addressable
+and `51/push-ECX` which modify these registers as well as a byte-addressable
 memory. For a complete list of supported instructions, run `subx help opcodes`.
 
 (SubX doesn't support floating-point registers yet. Intel processors support
@@ -229,10 +250,10 @@ them.
 In the last three cases, one exception occurs when the `/rm32` argument
 contains `4`. Rather than encoding register `ESP`, it means the address is
 provided by three _whole new_ arguments (`/base`, `/index` and `/scale`) in a
-_totally_ different way:
+_totally_ different way (where `<<` is the left-shift operator):
 
   ```
-  reg/mem = *(/base + /index * (2 ^ /scale))
+  reg/mem = *(base + (index << scale))
   ```
 
 (There are a couple more exceptions ☹; see [Table 2-2](modrm.pdf) and [Table 2-3](sib.pdf)
@@ -769,9 +790,6 @@ can replicate:
 
 To falsify these hypotheses, here's a roadmap of the next few planned features:
 
-* A script to package SubX together with a minimal Linux kernel image
-  (compiled from source, of course).
-
 * Testable, dependency-injected vocabulary of primitives
   - Streams: `read()`, `write()`. (✓)
   - `exit()` (✓)
@@ -789,6 +807,8 @@ To falsify these hypotheses, here's a roadmap of the next few planned features:
     variables; verifier checks that register reads are for the same type that
     was last written -- across all control flow paths.
 
+* Gradually streamline the bundled kernel, stripping away code we don't need.
+
 ## Credits
 
 Mu builds on many ideas that have come before, especially:

clean

@@ -7,3 +7,4 @@ rm -rf .until
 test $# -gt 0 && exit 0  # convenience: 'clean top-level' to leave subsidiary tools alone
 rm -rf enumerate/enumerate tangle/tangle tangle/*_list */*.dSYM termbox/*.[oa]
 rm -rf browse_trace/browse_trace_bin browse_trace/*_list
+rm -rf tmp mu.iso

gen_iso

@@ -0,0 +1,61 @@
+#!/bin/sh
+# Build one or more .subx files into an ELF binary, and package it up into a
+# bootable ISO image.
+#
+# Must be run on Linux.
+#
+# Dependencies:
+#   apt install build-essential flex bison wget libelf-dev libssl-dev xorriso
+#
+# Based on http://minimal.linux-bg.org (GPLv3)
+
+set -e
+
+if [ $# -eq 0 ]
+then
+  echo "Usage: `basename $0` file.subx ..."
+  exit 1
+fi
+
+echo "=== constructing initramfs out of SubX binary"
+./ntranslate $*
+mv a.elf init
+chmod +x init
+rm -rf tmp/isoimage
+mkdir -p tmp/isoimage/boot
+echo init | cpio -R root:root -H newc -o | xz -9 --check=none > tmp/isoimage/boot/rootfs.xz
+
+if [ ! -d kernel ]
+then
+  echo "=== cloning kernel"
+  git clone https://github.com/akkartik/kernel
+fi
+
+echo "=== building kernel"
+cp kernel.config kernel/.config
+( cd kernel
+  make bzImage -j $(grep ^processor /proc/cpuinfo | wc -l)
+)
+cp kernel/arch/x86/boot/bzImage tmp/isoimage/boot/kernel.xz
+
+echo "=== downloading syslinux"
+test -f tmp/syslinux-6.03.tar.xz  ||  wget https://kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.xz -P tmp
+echo "=== unpacking syslinux"
+tar xf tmp/syslinux-*.tar.xz -C tmp
+
+mkdir -p tmp/isoimage/boot/syslinux
+cp syslinux.cfg \
+   tmp/syslinux-*/bios/core/isolinux.bin \
+   tmp/syslinux-*/bios/com32/elflink/ldlinux/ldlinux.c32 \
+   tmp/isoimage/boot/syslinux
+
+echo "=== generating ISO"
+# 'hybrid' ISO can also be used on non-optical media such as a disk or USB stick
+xorriso -as mkisofs \
+  -isohybrid-mbr tmp/syslinux-*/bios/mbr/isohdpfx.bin \
+  -c boot/syslinux/boot.cat \
+  -b boot/syslinux/isolinux.bin \
+    -no-emul-boot \
+    -boot-load-size 4 \
+    -boot-info-table \
+  tmp/isoimage -o mu.iso

run_one_test

@@ -16,7 +16,7 @@ else
 fi
 
 set -e
-                                      # turn newlines into spaces
+                                        # turn newlines into spaces
 CFLAGS=$CFLAGS ./subx --debug translate $(echo $FILES) /tmp/run_one_test.subx -o /tmp/a.elf
 
 ./subx --debug --trace run /tmp/a.elf

syslinux.cfg

@@ -0,0 +1,6 @@
+DEFAULT mu
+
+LABEL mu
+        LINUX /boot/kernel.xz
+        APPEND vga=nomodeset
+        INITRD /boot/rootfs.xz

vimrc.vim

@@ -86,11 +86,11 @@ command! -nargs=0 L exec "%!grep -a label |grep -v clear-stream:loop"
 "   cursor on '#' causes error
 noremap <Leader>t {j0:call RunTestMoveCursor("<C-r><C-w>")<CR>
 function RunTestMoveCursor(arg)
-  exec "!run_one_test.sh ".expand("%")." ".a:arg
+  exec "!./run_one_test ".expand("%")." ".a:arg
   exec "normal \<C-o>"
 endfunction
 function RunTestMoveCursorAndMaybeOpenTrace(arg)
-  exec "!run_one_test.sh ".expand("%")." ".a:arg
+  exec "!./run_one_test ".expand("%")." ".a:arg
   exec "normal \<C-o>"
   if v:shell_error
     noautocmd vertical split last_run