Examples:
- you try to write file but disk is full
- you have two Teliva files being edited at the same time
Both are situations where it's impossible to avoid some data loss.
However, we should now at least have some valid state of the .tlv file
saved to disk where we'd previously end up with a zero-size file or
garbage.
Teliva emits timestamps in multi-line format end in a newline. As a
result, notes get rendered on the next line and are then immediately
overwritten by the contents of the definition.
This bug was masked by my hacky 'original' timestamps which don't use
multi-line format.
This reverts commit 7c1b9d0b91.
The 'big hammer' isn't good enough. The recent changes view seems to
need state on the stack across invocations of the editor.
This is a complete mess. I want to abstract reading multiline strings
behind a function, but the lookahead requirements for that are quite
stringent. What's a reasonable abstraction here?
I really wanted to avoid getting into defining or parsing new file
formats. However, using the entire power of Lua is not ideal, as
described earlier in Konrad Hinsen's bug. In addition to everything
else, it's a vector for arbitrary code execution when someone loads an
untrusted image.
I could use JSON, but it requires ugly string escaping. Seems cleaner to
just use YAML. But YAML is complex and needs its own dependencies. If
I'm going to do my own, might as well make the multi-line string format
really clear.
I can't yet write the new format.
Steps to reproduce:
* Run teliva with some app.
* Press ctrl-e to edit the app.
* Select some function.
* Press ctrl-g and type in some Lua keyword like 'function' or 'while'
(Since the first word in a function is often 'function', it becomes
the default if you press ctrl-g immediately after entering the editor
for a function.)
* Type nothing. Run the app.
Desired behavior: app continues to run. The definition for the keyword
is silently ignored (in future we may want to provide an error message)
Behavior before this commit: app silently exited with non-zero status,
and refused to restart thereafter until the .tlv file was manually
edited to delete the definition for the Lua keyword.
Behavior after this commit: app throws an error message like these:
* For `function`:
```
src/teliva: x.tlv:99: '(' expected near '='
sorry, you'll need to edit the image directly. press any key to exit.
```
* For `while`:
```
src/teliva: x.tlv:99: unexpected symbol near 'while'
sorry, you'll need to edit the image directly. press any key to exit.
```
You still need to edit the .tlv file manually, but the steps for
recovery are a bit more discoverable.
To fix this properly I also need to fix a looming security hole I've
been thinking about for some time. The long-term goal of Teliva is to
put the human running apps in control of what they do, by sandboxing
accesses to the file system, network and so on. However, even after we
build gates on all of Lua's standard libraries, we're still parsing .tlv
files as Lua, with all of its power available.
Solution: load .tlv files as some sort of JSON-like subset of Lua. Maybe
I should just use JSON, and rely on code that's already in Teliva, even
if I'm introducing a new notation in the process.
It turns out Lua has been providing us this information all along! I'd
just not created the space on screen to show it. Make it persist better.
Kilo now no longer tracks its own status messages, which is a regression
in a rare condition.
^/ works on Linux but not on Mac
^- emits the same character code on Mac
^_ seems to be the underlying character code, and works on both
ctrl-7 also emits the same character code
On a Thinkpad X13, the `delete` key emits `^[[3~` outside of Teliva.
Within Teliva, ncurses converts it to character code 330 (0x14a), which
it fails to recognize as KEY_BACKSPACE. Why?
My backspace is converted to character code 263, which ncurses does
recognize as KEY_BACKSPACE.
ctrl-h is character code 8.
Both 330 and 263 are valid Unicode code points, which feels really ugly
and ambiguous.
I still don't understand the entire state space here, so I'm trying to
err on the side of improving discoverability of the `ctrl-h` escape
hatch. Without requiring too wide a window to show all hotkeys on the
menu.
So far I've been trying to make Teliva follow the default colorscheme of
the terminal, but that easily ends up with illegible color combinations.
New plan: always start with a light background within Teliva. People who
want a dark background will currently need to mess with C sources.
This should somewhat fix https://github.com/akkartik/teliva/issues/1.
It's still not clear whether the default should be a dark or light
background. While dark background is more common in terminals, I believe
newcomers to terminals will prefer a light background. Then again, I'm
biased since I use a light background in my terminals.
I wish I could just hide KEY_BACKSPACE and prevent myself from using it
by accident.
Then again, I'm not making this smarts available in Teliva programs
themselves. Just for the Teliva environment.
Short of syntax errors that keep us from parsing the teliva_program
table, we should now be able to recover gracefully from everything.
Yesterday I started to try to add this to load_definitions before
realizing most errors are only noticed while running `main`. But I
didn't think of recovering from the docall of `main` until this morning.
It makes me very nervous now that there's save_editor_state within
editor event loop, when the editor could be editing notes. Things are
slightly better than this morning, but this prototype still suxxors.
I think I've gotten rid of all the segfaults, but it's still pretty
messed up: if you hit ctrl-g and go edit some definition, it doesn't get
saved. You're just storing the edit in the note.
I'm mindful of the way abstractions can create duplicate effort:
https://flak.tedunangst.com/post/browser-ktrace-browsing
== Kartik's SAD theorem
As programs grow complex, you will be repeatedly forced to either:
- maintain some State,
- perform some computations Again,
- or Duplicate some code.
Here a small amount of duplication seems like the best alternative.
Particularly since no syscalls are involved.
All the spaghetti is hiding another issue: when we load editor state,
that code path currently never leads to importing the edited buffer back
into the image.
Yet another attempt at drawing the state diagram:
Wgetch -> switch_to_editor -> select_view
select_view -> load_editor_state | big_picture_view
load_editor_state -> edit_from -> editorProcessKeypress
big_picture_view -> edit_image -> edit_buffer -> resumeEdit* -> load_editor_buffer -> editorProcessKeypress
big_picture_view -> recent_changes
recent_changes -> big_picture_view | edit_buffer
The problem is that load_editor_state doesn't eventually call
load_editor_buffer the way its sibling big_picture_view does.
For starters, it's confusing that switch_to_editor calls
big_picture_view which calls other editor functions. What is 'editor'
here, anyway?
Let's rename switch_to_editor to developer_mode. So the app starts out
in user mode, and might eventually transition to developer mode.
Developer mode is a black hole; to leave it and return to user mode we
restart the entire app.
The architecture in my mind is now:
- Teliva consists of user mode and developer mode
- Developer mode consists of multiple views
- Each view, when it needs to edit something:
- initializes kilo
- loads a buffer into it
- resumes editing the buffer as necessary
My code is already at spaghetti levels. Some coping mechanisms.
===
The big problem with the Teliva approach compared to my previous Mu
project: no tests. At this point I should document the growing list of
manual tests I've been maintaining:
run a program
run a program, edit
run a program, edit, make an edit, run | edit takes effect
run a program with error
run a program, edit, make an error, run
run a program, edit, ^g to a different definition, make an edit, ^e to run again
run a program, edit, ^g to a non-existent definition
run a program, edit, ^g to a different definition, ^g to a different definition, ^e to run again
start -> big picture -> edit -> move cursor -> run -> edit | cursor preserved
start -> big picture -> edit A -> move cursor -> big picture -> edit B | cursor initialized
start -> big picture -> edit A -> move cursor -> run -> exit -> start -> big picture -> edit B | cursor initialized
start -> big picture -> edit A -> move cursor -> run -> exit -> start -> big picture -> edit B -> big picture (*)
syntax highlighting for line comments
syntax highlighting for multiline comments
(*) - fixed in this commit
===
Coarse-grained state diagram (ignoring recent_changes_view):
app -> big picture on ^e
big picture -> editor when selecting a definition
editor -> app on e
editor -> big picture on ^b
Fine-grained sequence diagram:
main -> pmain -> ... -> Wgetch -> switch_to_editor -> select_view
select_view -> load_editor_state, falling through to big_picture_view if needed
load_editor_state -> edit_from -> editorProcessKeypress
The consequence I hadn't fully internalized was the return path:
editorProcessKeypress -> edit_from -> big_picture_view
Which implies that load_editor_state fails in two ways:
- when the state doesn't exist or is not applicable or is corrupted
- when editing from the state explicitly requested the big picture view
Switching the return value semantics for load_editor_state now supports
both ways.
Making changes to lcurses directory was causing it to be compiled, but
not causing teliva to be relinked.
Now I'm:
- unconditionally running `make` on subdirectories
- conditionally linking their outputs
Seems reasonable when I put it like that. Hopefully this is working now.
I used to know `make` down cold a decade ago, but it's evaporated from
my brain.
Teliva's constantly restarting without the user being aware of it. So
far I figured saving history across the user actually exiting and
restarting Teliva was just a happy "feature". However, it looks like
that's actually more complex to implement. Keeping editor state across
user-visible restarts results in these problems:
- opening the editor after restart has the cursor position messed up, no
matter what definition you open.
- more seriously, opening the editor after restart can't seem to get to
the big-picture view anymore.
Rather than try to debug what's going on, I'm going to just cordon off
that part of the state space for now.
We're not using this yet.
I agonized over this decision for several weeks. Is Teliva's need to
restart with execve an utter hack or a good thing? I'm leaning towards
the latter. Constantly exercising the initial flow makes Teliva more
crash-only. We can build Steve Yegge's idea of immortality (http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html)
out of crash-only primitives, just by making reboots instantaneous. But
focusing directly on immortality tends to compromise crash-only by
exercising it more rarely.
One other issue this brings up: loading these Lua tables from disk is a
vector for arbitrary code execution. I need to fix these when I get to
sandboxing.
I can't select C99 in luasocket, because I don't know how to include
the definition of struct timespec. All this fucking complexity. But
hopefully things will build on OpenBSD now.
Teliva is never intended to be "installed" somewhere. Just work inside
its directory and separately share the .tlv files you create. (Though I
don't yet have a good flow for starting a new .tlv file.)
For starters, put Linux-specific stuff in a Linux-specific target.
By not resetting MYCFLAGS and MYLDFLAGS, I'm unnecessarily passing in
-DLUA_USE_LINUX. But that'll make it easier to get things running on Mac
and BSD.
I'm still unclear on precisely what the experience should be here. We
probably don't need all of a version control system. The goal is just to
be able to answer the question, "what did I change recently that caused
things to break?"
For now let's just start with letting people see past versions.
The problem: if ever I hit ctrl-e to go to the big picture view and then
hit Esc to go back to running the app, my terminal was messed up after
exiting the app.
Why did I even have this gunk? Perhaps it dates from the time when kilo
was emitting raw escape sequences rather than using ncurses.
At least in short runs.
Encouraging that the problem was in a recent commit (5a63a5ca40 from
yesterday when I introduced version control).
Disabling Address Sanitizer again.
I'm starting to see some heap buffer overruns, which means we have too
much C code.
I noticed this because editing life.tlv no longer works after commit
5a63a5ca4. However, the offending heap overrun has been around long
before that. It's just been a silent bug until now.
One old drawback now has a new look. Before, we loaded definitions in
order, so global definitions had to exist before other global
definitions that used them. See window and grid in life.tlv. Now we load
definitions in reverse order, so initialization needs to change. Worse,
if we update window, we need to edit grid just to fix the order.
This implies that we can't yet optimize away bindings where there are no
new changes.
Since everything is in my control there's no need to parameterize
include paths.
It's a struggle to get make to run when it should. Lying that something
is phony stops working when it's a dependency. Commands get
unnecessarily run. Just fucking run recursive makes directly in the
target that depends on them.
I'd like to enable -Wextra as well, but that creates some false
positives.
I've at least made my changes clean w.r.t. -Wextra.
Now we have 4 remaining warnings with gcc 9.3 that seem genuine. Need to
fix those.
Still extremely ugly:
- I've inlined all the namespaces under ssl, so you need to know that
context and config are related to ssl.
- luasec comes with its own copy of luasocket. I haven't deduped that
yet.
Until now we had just the bare minimum bindings needed for the demos
built so far. Now we have all of lcurses building in place with minimal
changes.
The changes in this commit can run hanoi.lua when inlined into Lua 5.1,
but don't work with Teliva.
This was on my todo list. What made it urgent was finding that calling
getch() even once while in ncurses caused Kilo to stop detecting arrow
keys. No need to debug that sort of nonsense.
I was saving an address on the stack to a global, and it was getting
clobbered later. This is the sort of thing I completely eliminated in
https://github.com/akkartik/mu :/
Now I'm taking a leaf out of the Mu playbook and leaking a little bit of
memory every time I switch definitions.
src/teliva counter.tlv
C-e # switch to editor
C-e # save and quit
C-x # exit
counter.tlv now has the same logical contents, though the whitespace has
changed, and the order of keys is different.
The implementation is utterly ghastly. For one, I'm unnecessarily
interfacing with kilo through the file system.
Plan is for this to be the default representation for Teliva programs.
Text-friendly but not meant to be edited directly as text. Will
eventually include both code and data definitions, both current snapshot
and past revision history.
Right now .tlv files seem to run. Error checking is non-existent,
because I don't understand Lua's idioms around 'status' yet. Opening the
editor expectedly segfaults.
This commit is the most mind-bending bit of code I've written in a long
time.
Kartik K. Agaram