It turns out (bowboard screen 128) on a real screen massively slowed down
and ran out of memory since commit e2ab1b30b1 on May 19. The culprit was
these changes, which created memory allocations for a new trace on every
recursive call.
I originally had some vague desire to isolate these calls from the user-visible
trace. That's expensive enough that I'll wait until it becomes a concern
before trying to isolate again.
It turns out I have a problem with trace depth somewhere which I just wasn't
noticing before. Running certain sandboxes (line; maybe loops?) twice was
causing traces to no longer start at depth 1, which implies that they weren't
terminating at depth 1. This became a lot more obvious since I instituted
a max-depth.
This was quite difficult to diagnose. The issue I noticed was that brline
had stopped working. All the bugs in previous commits were hiding the cause.
Once I cleaned them up, I realized the problem was that the `(,x0 ,y0)
was triggering the nested-backquote check. The fix was fairly straightforward
then (even though I didn't yet understand why). But how to write a test
for this? I spent some time trying to do so without defining a macro using
literal macros, before I realized:
You can't call literal macros; we don't have first-class macros.
Trying to insert literal macro support just breaks everything because we
have no way to distinguish between a literal macro call and the stage in
macroexpand where a symbol has been replaced with its macro definition.
How do you explain stuff like this? I grow weary of Lisp.
There's still some issue in loading the entire definition of brline from
data.limg.
Now that we never have a null trace, tracing errors is always safe. And
now that we're running with low trace max-depth we're more likely to run
into problems with missing errors in the trace.
I was aware of some complications. The various indexes and y coordinates
in the trace's cache would be unstable and need to be recomputed. But it's
surprising that the trace _completely disappears_.
The goal: the sandbox initially maintains a shallow trace. As you expand
into the trace, the environment reruns the sandbox at greater depth as
needed.
The challenge: expanding happens within edit-trace, which doesn't have
the whole sandbox needed to re-run the sandbox. We'll either need to expand
the trace's capabilities to include the whole sandbox, or duplicate some
logic to decide when to run the sandbox.
So far we were only doing so for the first few lines, just enough to render
one page's worth of lines. We'd have probably noticed if we collapsed some
lines after re-evaluating.
I've been stymied for a week wondering how to reliably compute trace-line
identity. A trace can have multiple identical lines. Only some of them
may be visible at any point. How to remember which is which across re-evaluations?
There's no easy answer. I'm just going to keep things ad hoc. When you
re-evaluate, new lines can currently pop into visibility. However we guarantee
that just moving around the trace will be stable, thanks to the visible
bit being cached within each trace-line. Scrolling will be similar. Reevaluating
may cause the trace to be perturbed up or down. However, just scrolling
around will work reliably.
We now use traces everywhere for error-checking. Null traces introduce
the possibility of changing a functions error response, and therefore its
semantics.
Another commit, another bugfix.
Some snippets from my currently exploding todo list:
- always investigate lookup errors immediately. Beyond the root cause, they should never happen at the moment, while we aren't reclaiming memory.
we should always return a more precise error message. Usually involving null pointer checks.
- on abort, print out stack trace
- emit mapping of labels to addresses during survey
- store a mapping of symbols somewhere in the code image
- stop allocating 1KB per token; expand space for tokens as needed
In the process I spent a long time tracking down a stray TODO in 108write.subx
that I thought would abort but didn't since the switch to baremetal.
Then after I reintroduced that assertion I had to go track down a bunch
of buffer sizes. Stream sizes continue to be a huge mess.
Current plan:
- some way to define macros. For now:
(def f (litmac litfn () (a b) `(+ ,a , b)))
- macroexpand will expand calls by passing them through the cdr
(f 3 4)
macroexpand: ((litfn () (a b) `(+ ,a ,b)) 3 4)
=> (+ 3 4)
eval: (+ 3 4) => 7
I don't understand why a second line in the keyboard is visible now where
it wasn't before. That whole aspect has unclear desires. What exactly do
I want to happen on newlines?
I keep running into one hole in Mu's memory-safety since dropping the Linux
dependency: null pointers no longer error when dereferenced. Here the problem
manifests as aliasing: lots of gap buffers share the same exact data near
address 0, because it was never initialized.
It requires more than 1GB to fill the screen with a chessboard pattern
using the definition in shell/iterative-definitions.limg.
I also speed up the chessboard program by clearing the screen up front
and then only rendering the white pixels.
It took me _way_ too long to realize that I'm not checking for errors within
the loop, and that will cause it to manifest as an infinite loop as inner
evaluations fail to run.
Debugging notes, for posterity:
printing one row of a chessboard pattern over fake screen (chessboard screen 4 0 0 15) gets stuck in an infinite loop halfway through
debug pattern during infinite loop: VWEX. It's still in the loop but it's not executing the body
raw (fill_rect screen 16 0 20 4 15) works fine
same number of calls to fill_rect work fine
replacing calls to fill_rect with pixel inside chessboard2 works fine
at the point of the infinite loop it's repeatedly going through the hline loop
-- BUT it never executes the check of the loop (< lo hi) with lo=20, hi=20. Something is returning 1, but it's not inside <
stream optimization is not implicated
simple test case with a single loop
(
(globals . (
(foo . (fn () (screen i n)
(while (< i n)
(pixel screen 4 4 i)
(pixel screen 5 4 i)
(pixel screen 6 4 i)
(pixel screen 7 4 i)
(set i (+ i 1)))))
))
(sandbox . (foo screen 0 100))
)
simpler (if you reset cursor position before every print):
(
(globals . (
(foo . (fn () (screen i n)
(while (< i n)
(print screen i)
(set i (+ i 1)))))
))
(sandbox . (foo screen 0 210))
)
I now believe it has nothing to do with the check. The check always works.
Sometimes no body is evaluated. And so the set has no effect.
All highly experimental. Current constraints:
* No tail recursion elimination
* No heap reuse
* Keep implementation simple
So it's slow, and I don't want to complicate it to speed it up. So I'm
investing in affordances to help deal with the slowness. However, in the
process I've taken the clean abstraction of a trace ("all you need to do
is add to the trace") and bolted on call counts and debug-prints as independent
mechanisms.
Before: we always drew pixels atop characters, and we only drew pixels
that were explicitly requested.
After: we always draw pixels atop characters, and we only draw pixels that
don't have color 0.
Both semantics should be identical as long as pixels are never drawn atop
characters.
Filling pixels isn't a rare corner case. I'm going to switch to a dense
rather than sparse representation for pixels, but callers will have to
explicitly request the additional memory.
We now have a couple of protections:
- if we get close to running out of space in the trace we drop in an
error
- if we run out of space in the trace we stop trying to append
- if there are errors we cancel future evaluations
This is already much nicer. You can't do much on the Mu computer, but at
least it gracefully gives up and shows its limitations. On my computer
the Mu shell tries to run computations for about 20s before giving up.
That seems at the outer limit of what interactivity supports. If things
take too long, test smaller chunks.
Among other things, we turned off the trace to significantly speed up the
debug cycle.
State as of https://merveilles.town/@akkartik/106079258606146213
Ohhh, as I save the commit I notice a big problem: I've been editing the
disk image directly because writes to the Mu disk lose indentation. But
I've been forgetting that the state in the Mu disk needs to be pre-evaluated.
So function bindings need extra parens for the environment. The `pixel`
calls in the previous commit message are the first statement in the body,
and they aren't actually considered part of the body right now. No wonder
they don't run.
There are lots of other problems, but this will clarify a lot.
I tried building a function to draw a horizontal line across the screen.
Here's what I have in data.txt:
(
(globals . (
(horline . (fn () (screen y)
(horline_1 screen y 0 (width screen))))
(horline_1 . (fn () (screen y lo hi)
(if (>= lo hi)
()
((fn ()
(pixel screen lo y 12)
(horline_1 screen y (+ lo 1) hi))))))
))
(sandbox . (horline_1 screen 0 0 20))
)
$ dd if=/dev/zero of=data.img count=20160
$ cat data.txt |dd of=data.img conv=notrunc
$ ./translate shell/*.mu && qemu-system-i386 -hda disk.img -hdb data.img
Result: I can't call (horline screen 0) over a fake screen of width 40.
Some stream overflows somewhere after all the tweaks to various fixed-size
buffers scattered throughout the app. Calling horline_1 gets to a 'hi'
column of 20, but not to 30.
Kartik K. Agaram