Extremely cruddy implementation:
- I'm still unclear on how to represent the advice function:
- How to handle errors when loading user configuration?
Currently I refuse to start.
- Whole function? More errors to handle in header and so on. What if
the function is renamed?
- Just body? Needs more structured editing support.
- Lots of duplication, particularly between the permissions in the menu
and the permissions screen.
I don't know how to show the hostname at the time of connect() or
bind(), so networking is going to remain a boolean for now. It's also
unclear what effective constraints we can impose on what gets discussed
with a specific hostname. Everything outside the computer is out of
one's control.
One trick I learned is for consistently grabbing ASan logs on abort:
It's always safe to redirect stderr with ncurses!
I _think_ I don't need to gate other socket calls; you can't do anything
without bind() and connect(). And they should be good places to dump
more precise details later about the kind of server or client connection
being attempted.
In the process we now also have a mechanism for Teliva to overlay errors
while apps run. Might make sense to make that available to apps as well.
But I'm starting to realize that any app access to the Teliva areas of
the screen is fraught with risk.
Our sandboxing model is a blunt caricature, just two booleans. But let's
see how far this gets us.
Still doesn't persist, and definitely has no effect.
Current plan:
- two booleans to gate file and network access, respectively
- false shows as green, true shows as orange
- if _both_ booleans are true, then both show as red to indicate that
there are no protections.
It looks like attron doesn't actually enable colors near 256, even
though https://linux.die.net/man/3/attron suggests it does.
> COLOR_PAIR values can only be OR'd with attributes if the pair
> number is less than 256.
Put stuff people messing with Teliva apps are likely to need above the C
interface.
The state of documentation for Teliva app creators is still quite poor.
All they really have to go on is the example apps.
This isn't necessarily for sandboxing, but they don't really work right
now in the presence of ncurses, and it seems better to not include
broken stuff. Maybe we can get them to coexist with ncurses down the
road.
I should have documented that I'd never actually seen that code path
trigger before. Here's a minimal test that did it just now:
function test_foo()
return a+1
end
E2: [string "test_foo"]:2: attempt to perform arithmetic on global 'a' (a nil value)
A simple missing variable doesn't do it since it just evaluates to nil.
Without this commit, the above test was silently continuing to the main
app after failing tests.
In each session, Teliva has to bootstrap a trusted channel with the
computer owner while running arbitrarily untrusted code. So let's get
really, really precise about what the trusted channel consists of:
- the bottom-most row of screen containing the menu
- the keystrokes the owner types in
- ncurses COLOR_PAIR slots 254 (menu) and 255 (error)
One reason the menu colors are important: we don't want people to get
used to apps that hide the menu colors by setting default
foreground/background to invisible and then drawing their own menu one
row up.
The error COLOR_PAIR I don't see any reason to carve out right now, but
it seems like a good idea for Teliva the framework to not get into the
habit of apps doing some things for it.
I'm not sure how realistic all this is (I feel quite ill-equipped to
think about security), but it seems worthwhile to err on the side of
paranoia. Teliva will be paranoid so people don't have to be.
Kind of a subtle idea. Teliva the framework is intended to be
trustworthy software that people install on their computers. The apps
people run using Teliva may be less trustworthy. The whole point of
Teliva is to provide a sandbox for running code before you trust it.
One way (of many) apps can be malicious is by subtly getting between
what people see and reality. Imagine, for example, an app that draws a
fake menu bar and offers a different hotkey to edit source code. When
someone presses that hotkey they think they're using the standard Teliva
editor but they're really using an editor within the app, which the app
uses to hide its most malicious bits from view.
Down the road Teliva will have more bits of UI, such as for asking for
permission to read sensitive data. It's important that people understand
what they're granting permission to, that apps can't tamper with the
communications channel between them and Teliva.
This is likely just one of many ways for an app to break out of its
sandbox. Teliva isn't sandboxed yet. I'm just taking my first steps on
this journey. In particular, there are other mechanisms for asking for
user input besides `getch()`. I don't yet have a big-picture view of the
Teliva sandbox.
It seems clear that I need to educate people on the difference between
different parts of screen. Depending on the app you install, most of the
screen may be a dark forest. It'll be important to know where the safe
path is, where you can speak to trusted advisors while in the forest.
Examples:
- you try to write file but disk is full
- you have two Teliva files being edited at the same time
Both are situations where it's impossible to avoid some data loss.
However, we should now at least have some valid state of the .tlv file
saved to disk where we'd previously end up with a zero-size file or
garbage.
Teliva emits timestamps in multi-line format end in a newline. As a
result, notes get rendered on the next line and are then immediately
overwritten by the contents of the definition.
This bug was masked by my hacky 'original' timestamps which don't use
multi-line format.
This reverts commit 7c1b9d0b91.
The 'big hammer' isn't good enough. The recent changes view seems to
need state on the stack across invocations of the editor.
This is a complete mess. I want to abstract reading multiline strings
behind a function, but the lookahead requirements for that are quite
stringent. What's a reasonable abstraction here?
I really wanted to avoid getting into defining or parsing new file
formats. However, using the entire power of Lua is not ideal, as
described earlier in Konrad Hinsen's bug. In addition to everything
else, it's a vector for arbitrary code execution when someone loads an
untrusted image.
I could use JSON, but it requires ugly string escaping. Seems cleaner to
just use YAML. But YAML is complex and needs its own dependencies. If
I'm going to do my own, might as well make the multi-line string format
really clear.
I can't yet write the new format.
Steps to reproduce:
* Run teliva with some app.
* Press ctrl-e to edit the app.
* Select some function.
* Press ctrl-g and type in some Lua keyword like 'function' or 'while'
(Since the first word in a function is often 'function', it becomes
the default if you press ctrl-g immediately after entering the editor
for a function.)
* Type nothing. Run the app.
Desired behavior: app continues to run. The definition for the keyword
is silently ignored (in future we may want to provide an error message)
Behavior before this commit: app silently exited with non-zero status,
and refused to restart thereafter until the .tlv file was manually
edited to delete the definition for the Lua keyword.
Behavior after this commit: app throws an error message like these:
* For `function`:
```
src/teliva: x.tlv:99: '(' expected near '='
sorry, you'll need to edit the image directly. press any key to exit.
```
* For `while`:
```
src/teliva: x.tlv:99: unexpected symbol near 'while'
sorry, you'll need to edit the image directly. press any key to exit.
```
You still need to edit the .tlv file manually, but the steps for
recovery are a bit more discoverable.
To fix this properly I also need to fix a looming security hole I've
been thinking about for some time. The long-term goal of Teliva is to
put the human running apps in control of what they do, by sandboxing
accesses to the file system, network and so on. However, even after we
build gates on all of Lua's standard libraries, we're still parsing .tlv
files as Lua, with all of its power available.
Solution: load .tlv files as some sort of JSON-like subset of Lua. Maybe
I should just use JSON, and rely on code that's already in Teliva, even
if I'm introducing a new notation in the process.
It turns out Lua has been providing us this information all along! I'd
just not created the space on screen to show it. Make it persist better.
Kilo now no longer tracks its own status messages, which is a regression
in a rare condition.
^/ works on Linux but not on Mac
^- emits the same character code on Mac
^_ seems to be the underlying character code, and works on both
ctrl-7 also emits the same character code
On a Thinkpad X13, the `delete` key emits `^[[3~` outside of Teliva.
Within Teliva, ncurses converts it to character code 330 (0x14a), which
it fails to recognize as KEY_BACKSPACE. Why?
My backspace is converted to character code 263, which ncurses does
recognize as KEY_BACKSPACE.
ctrl-h is character code 8.
Both 330 and 263 are valid Unicode code points, which feels really ugly
and ambiguous.
I still don't understand the entire state space here, so I'm trying to
err on the side of improving discoverability of the `ctrl-h` escape
hatch. Without requiring too wide a window to show all hotkeys on the
menu.
So far I've been trying to make Teliva follow the default colorscheme of
the terminal, but that easily ends up with illegible color combinations.
New plan: always start with a light background within Teliva. People who
want a dark background will currently need to mess with C sources.
This should somewhat fix https://github.com/akkartik/teliva/issues/1.
It's still not clear whether the default should be a dark or light
background. While dark background is more common in terminals, I believe
newcomers to terminals will prefer a light background. Then again, I'm
biased since I use a light background in my terminals.
I wish I could just hide KEY_BACKSPACE and prevent myself from using it
by accident.
Then again, I'm not making this smarts available in Teliva programs
themselves. Just for the Teliva environment.
Short of syntax errors that keep us from parsing the teliva_program
table, we should now be able to recover gracefully from everything.
Yesterday I started to try to add this to load_definitions before
realizing most errors are only noticed while running `main`. But I
didn't think of recovering from the docall of `main` until this morning.
It makes me very nervous now that there's save_editor_state within
editor event loop, when the editor could be editing notes. Things are
slightly better than this morning, but this prototype still suxxors.
I think I've gotten rid of all the segfaults, but it's still pretty
messed up: if you hit ctrl-g and go edit some definition, it doesn't get
saved. You're just storing the edit in the note.
I'm mindful of the way abstractions can create duplicate effort:
https://flak.tedunangst.com/post/browser-ktrace-browsing
== Kartik's SAD theorem
As programs grow complex, you will be repeatedly forced to either:
- maintain some State,
- perform some computations Again,
- or Duplicate some code.
Here a small amount of duplication seems like the best alternative.
Particularly since no syscalls are involved.
All the spaghetti is hiding another issue: when we load editor state,
that code path currently never leads to importing the edited buffer back
into the image.
Yet another attempt at drawing the state diagram:
Wgetch -> switch_to_editor -> select_view
select_view -> load_editor_state | big_picture_view
load_editor_state -> edit_from -> editorProcessKeypress
big_picture_view -> edit_image -> edit_buffer -> resumeEdit* -> load_editor_buffer -> editorProcessKeypress
big_picture_view -> recent_changes
recent_changes -> big_picture_view | edit_buffer
The problem is that load_editor_state doesn't eventually call
load_editor_buffer the way its sibling big_picture_view does.
For starters, it's confusing that switch_to_editor calls
big_picture_view which calls other editor functions. What is 'editor'
here, anyway?
Let's rename switch_to_editor to developer_mode. So the app starts out
in user mode, and might eventually transition to developer mode.
Developer mode is a black hole; to leave it and return to user mode we
restart the entire app.
The architecture in my mind is now:
- Teliva consists of user mode and developer mode
- Developer mode consists of multiple views
- Each view, when it needs to edit something:
- initializes kilo
- loads a buffer into it
- resumes editing the buffer as necessary
My code is already at spaghetti levels. Some coping mechanisms.
===
The big problem with the Teliva approach compared to my previous Mu
project: no tests. At this point I should document the growing list of
manual tests I've been maintaining:
run a program
run a program, edit
run a program, edit, make an edit, run | edit takes effect
run a program with error
run a program, edit, make an error, run
run a program, edit, ^g to a different definition, make an edit, ^e to run again
run a program, edit, ^g to a non-existent definition
run a program, edit, ^g to a different definition, ^g to a different definition, ^e to run again
start -> big picture -> edit -> move cursor -> run -> edit | cursor preserved
start -> big picture -> edit A -> move cursor -> big picture -> edit B | cursor initialized
start -> big picture -> edit A -> move cursor -> run -> exit -> start -> big picture -> edit B | cursor initialized
start -> big picture -> edit A -> move cursor -> run -> exit -> start -> big picture -> edit B -> big picture (*)
syntax highlighting for line comments
syntax highlighting for multiline comments
(*) - fixed in this commit
===
Coarse-grained state diagram (ignoring recent_changes_view):
app -> big picture on ^e
big picture -> editor when selecting a definition
editor -> app on e
editor -> big picture on ^b
Fine-grained sequence diagram:
main -> pmain -> ... -> Wgetch -> switch_to_editor -> select_view
select_view -> load_editor_state, falling through to big_picture_view if needed
load_editor_state -> edit_from -> editorProcessKeypress
The consequence I hadn't fully internalized was the return path:
editorProcessKeypress -> edit_from -> big_picture_view
Which implies that load_editor_state fails in two ways:
- when the state doesn't exist or is not applicable or is corrupted
- when editing from the state explicitly requested the big picture view
Switching the return value semantics for load_editor_state now supports
both ways.
Making changes to lcurses directory was causing it to be compiled, but
not causing teliva to be relinked.
Now I'm:
- unconditionally running `make` on subdirectories
- conditionally linking their outputs
Seems reasonable when I put it like that. Hopefully this is working now.
I used to know `make` down cold a decade ago, but it's evaporated from
my brain.
Teliva's constantly restarting without the user being aware of it. So
far I figured saving history across the user actually exiting and
restarting Teliva was just a happy "feature". However, it looks like
that's actually more complex to implement. Keeping editor state across
user-visible restarts results in these problems:
- opening the editor after restart has the cursor position messed up, no
matter what definition you open.
- more seriously, opening the editor after restart can't seem to get to
the big-picture view anymore.
Rather than try to debug what's going on, I'm going to just cordon off
that part of the state space for now.
We're not using this yet.
I agonized over this decision for several weeks. Is Teliva's need to
restart with execve an utter hack or a good thing? I'm leaning towards
the latter. Constantly exercising the initial flow makes Teliva more
crash-only. We can build Steve Yegge's idea of immortality (http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html)
out of crash-only primitives, just by making reboots instantaneous. But
focusing directly on immortality tends to compromise crash-only by
exercising it more rarely.
One other issue this brings up: loading these Lua tables from disk is a
vector for arbitrary code execution. I need to fix these when I get to
sandboxing.
I can't select C99 in luasocket, because I don't know how to include
the definition of struct timespec. All this fucking complexity. But
hopefully things will build on OpenBSD now.
Teliva is never intended to be "installed" somewhere. Just work inside
its directory and separately share the .tlv files you create. (Though I
don't yet have a good flow for starting a new .tlv file.)
For starters, put Linux-specific stuff in a Linux-specific target.
By not resetting MYCFLAGS and MYLDFLAGS, I'm unnecessarily passing in
-DLUA_USE_LINUX. But that'll make it easier to get things running on Mac
and BSD.
I'm still unclear on precisely what the experience should be here. We
probably don't need all of a version control system. The goal is just to
be able to answer the question, "what did I change recently that caused
things to break?"
For now let's just start with letting people see past versions.
The problem: if ever I hit ctrl-e to go to the big picture view and then
hit Esc to go back to running the app, my terminal was messed up after
exiting the app.
Why did I even have this gunk? Perhaps it dates from the time when kilo
was emitting raw escape sequences rather than using ncurses.
At least in short runs.
Encouraging that the problem was in a recent commit (5a63a5ca40 from
yesterday when I introduced version control).
Disabling Address Sanitizer again.
I'm starting to see some heap buffer overruns, which means we have too
much C code.
I noticed this because editing life.tlv no longer works after commit
5a63a5ca4. However, the offending heap overrun has been around long
before that. It's just been a silent bug until now.
One old drawback now has a new look. Before, we loaded definitions in
order, so global definitions had to exist before other global
definitions that used them. See window and grid in life.tlv. Now we load
definitions in reverse order, so initialization needs to change. Worse,
if we update window, we need to edit grid just to fix the order.
This implies that we can't yet optimize away bindings where there are no
new changes.
Since everything is in my control there's no need to parameterize
include paths.
It's a struggle to get make to run when it should. Lying that something
is phony stops working when it's a dependency. Commands get
unnecessarily run. Just fucking run recursive makes directly in the
target that depends on them.
Kartik K. Agaram