chronometrist/cl/chronometrist.org

# -*- mode: poly-org; -*-
#+TITLE: Chronometrist
#+SUBTITLE: A friendly and powerful personal time tracker and analyzer
#+DESCRIPTION: Common Lisp implementation
#+AUTHOR: contrapunctus
#+PROPERTY: header-args :tangle yes :comments link

* Introduction
This is a book about Chronometrist, a time tracker for Emacs, written by a humble hobbyist hacker. It also happens to contain the canonical copy of the source code, and can be loaded as an Emacs Lisp program using the =literate-elisp= library.

I hope this book—when completed—passes Tim Daly's [[https://www.youtube.com/watch?v=Av0PQDVTP4A&t=8m52s]["Hawaii Test"]], in which a programmer with no knowledge of this program whatsover can read this book end-to-end, and come out as much of an expert in its maintenance as the original author.

—contrapunctus

* Explanation
:PROPERTIES:
:DESCRIPTION: The design, the implementation, and a little history
:END:
** Why I wrote Chronometrist
It probably started off with a desire for introspection and self-awareness - what did I do all day? How much time have I spent doing X? It is also a tool to help me stay focused on a single task at a time, and to not go overboard and work on something for 7 hours straight.

At first I tried an Android application, "A Time Tracker". The designs of Chronometrist's main buffer and the =report= buffer still resemble that of A Time Tracker. However, every now and then I'd forget to start or stop tracking. Each time I did, I had to open an SQLite database and edit UNIX timestamps to correct it, which was not fun :\

Later, I discovered John Wiegley's =timeclock=. It turned out that since it was an Emacs extension (actually, a part of Emacs), I was more likely to use it. Chronometrist started out as an "A Time Tracker"-like UI for =timeclock=, moving to an s-expression backend later.

Quite recently, after around two years of Chronometrist developement and use, I discovered that Org mode has a time tracking feature, too. Even though I've embraced some of [[#explanation-literate-programming][the joys of Org]], I'm not quite at ease with the idea of storing data in a complex text format with only one complete implementation.

** Design goals
:PROPERTIES:
:DESCRIPTION: Some vague objectives which guided the project
:END:
1. Don't make assumptions about the user's profession
   - e.g. timeclock seems to assume you're using it for a 9-to-5/contractor job
2. Incentivize use
   * Hooks allow the time tracker to automate tasks and become a useful part of your workflow
3. Make it easy to edit data using existing, familiar tools
   * We don't use an SQL database, where changing a single field is tricky [fn:1]
   * We use a text file containing s-expressions (easy for humans to read and write)
   * We use ISO-8601 for timestamps (easy for humans to read and write) rather than UNIX epoch time
4. Reduce human errors in tracking
5. Have a useful, informative, interactive interface
6. Support mouse and keyboard use equally

[fn:1] I still have doubts about this. Having SQL as a query language would be very useful in perusing the stored data. Maybe we should have tried to create a companion mode to edit SQL databases interactively?

** Terminology
:PROPERTIES:
:DESCRIPTION: Explanation of some terms used later
:END:
Chronometrist records /time intervals/ (earlier called "events") as plists. Each plist contains at least a =:name "<name>"=, a =:start "<iso-timestamp>"=, and (except in case of an ongoing task) a =:stop "<iso-timestamp>"=.

+ row :: a row of a table in a =tabulated-list-mode= buffer; an element of =tabulated-list-entries=.
+ schema :: the column descriptor of a table in a =tabulated-list-mode= buffer; the value of =tabulated-list-format=.

See also [[#explanation-time-formats][Currently-Used Time Formats]]

** Overview
At its most basic, we read data from a [[#program-backend][backend]], store it in a [[#program-data-structures][hash table]], and [[#program-frontend-chronometrist][display it]] as a [[elisp:(find-library "tabulated-list-mode")][=tabulated-list-mode=]] buffer. When the file is changed—whether by the program or the user—we [[refresh-file][update the hash table]] and the [[#program-frontend-chronometrist-refresh][buffer]].

In addition, we implement a [[#program-pretty-printer][plist pretty-printer]] and some [[#program-migration][migration commands]].

Extensions exist for -
1. [[file:chronometrist-key-values.org][attaching arbitrary metadata]] to time intervals, and
2. support for the [[file:chronometrist-third.org][Third Time system]]
3. [[https://tildegit.org/contrapunctus/chronometrist-goal][time goals and alerts]]

** Optimization
It is of great importance that Chronometrist be responsive -
+ A responsive program is more likely to be used; recall our design goal of 'incentivizing use'.
+ Being an Emacs program, freezing the UI for any human-noticeable length of time is unacceptable - it prevents the user from working on anything in their environment.
Thus, I have considered various optimization strategies, and so far implemented two.

*** Prevent excess creation of file watchers
One of the earliest 'optimizations' of great importance turned out to simply be a bug - turns out, if you run an identical call to [[elisp:(describe-function 'file-notify-add-watch)][=file-notify-add-watch=]] twice, you create /two/ file watchers and your callback will be called /twice./ We were creating a file watcher /each time the =chronometrist= command was run./ 🤦 This was causing humongous slowdowns each time the file changed. 😅
+ It was fixed in v0.2.2 by making the watch creation conditional, using =-fs-watch= to store the watch object.

*** Preserve hash table state for some commands
NOTE - this has been replaced with a more general optimization - see next section.

The next one was released in v0.5. Till then, any time the [[* chronometrist-file][=file=]] was modified, we'd clear the =events= hash table and read data into it again. The reading itself is nearly-instant, even with ~2 years' worth of data [fn:2] (it uses Emacs' [[elisp:(describe-function 'read)][=read=]], after all), but the splitting of [[#explanation-midnight-spanning-intervals][midnight-spanning events]] is the real performance killer.

After the optimization...
1. Two backend functions (=sexp-new= and =sexp-replace-last=) were modified to set a flag (=-inhibit-read-p=) before saving the file.
2. If this flag is non-nil, [[* refresh-file][=refresh-file=]] skips the expensive calls to =events-populate=, =tasks-from-table=, and =tags-history-populate=, and resets the flag.
3. Instead, the aforementioned backend functions modify the relevant variables - =events=, =task-list=, and =tags-history= - via...
   * =events-add= / =events-replace-last=
   * =task-list-add=, and
   * =tags-history-add= / =tags-history-replace-last=, respectively.

There are still some operations which [[* refresh-file][=refresh-file=]] runs unconditionally - which is to say there is scope for further optimization, if or when required.

[fn:2] As indicated by exploratory work in the =parsimonious-reading= branch, where I made a loop to only =read= and collect s-expressions from the file. It was near-instant...until I added event splitting to it.

*** Determine type of change made to file
Most changes, whether made through user-editing or by Chronometrist commands, happen at the end of the file. We try to detect the kind of change made - whether the last expression was modified, removed, or whether a new expression was added to the end - and make the corresponding change to =events=, instead of doing a full parse again (=events-populate=). The increase in responsiveness has been significant.

When =refresh-file= is run by the file system watcher, it uses =file-hash= to assign indices and a hash to =-file-state=. The next time the file changes, =file-change-type= compares this state to the current state of the file to determine the type of change made.

Challenges -
1. Correctly detecting the type of change
2. Updating =task-list= and the Chronometrist buffer, when a new task is added or the last interval for a task is removed (v0.6.4)
3. Handling changes made to an active interval after midnight
   * use the date from the plist's =:start= timestamp instead of the date today
   * =:append= - normally, add to table; for spanning intervals, invalid operation
   * =:modify= - normally, replace in table; for spanning intervals, split and replace
   * =:remove= - normally, remove from table; for spanning intervals, split and remove

Effects on the task list
1. When a plist is added, the =:name= might be new, in which case we need to add it to the task list.
2. When the last plist is modified, the =:name= may have changed -
   1. the =:name= might be new and require addition to the task list.
   2. the old plist may have been the only plist for the old =:name=, so we need to check if there are any other plists with the old =:name=. If there are none, the old =:name= needs to be removed from the task list.
3. When the last plist is removed, it may have been the only plist for the old =:name=, so we need to check if there are any other plists with the old =:name=. If there are none, the old =:name= needs to be removed from the task list.

** Midnight-spanning intervals
:PROPERTIES:
:DESCRIPTION: Events starting on one day and ending on another
:CUSTOM_ID: explanation-midnight-spanning-intervals
:END:
A unique problem in working with Chronometrist, one I had never foreseen, was tasks which start on one day and end on another. For instance, you start working on something at =2021-01-01 23:00= hours and stop on =2021-01-02 01:00=.

These mess up data consumption in all sorts of unforeseen ways, especially interval calculations and acquiring intervals for a specific date. In case of two of the most common operations throughout the program -
1. finding the intervals recorded on a given date
2. finding the time spent on a task on a given day - if the day's intervals used for this contain a midnight-spanning interval, you'll have inaccurate results - it will include yesterday's time from the interval as well as today's.

There are a few different approaches of dealing with them. (Currently, Chronometrist uses #3.)

*** Check the code of the first event of the day (timeclock format)
:PROPERTIES:
:DESCRIPTION: When the code of the first event in the day is "o", it's a midnight-spanning event.
:END:
+ Advantage - very simple to detect
+ Disadvantage - "in" and "out" events must be represented separately

*** Split them at the file level
+ Advantage - operation is performed only once for each such event + simpler data-consuming code + reduced post-parsing load.
+ What happens when the user changes their day-start-time? The split-up events are now split wrongly, and the second event may get split /again./

Possible solutions -
1. Add function to check if, for two events A and B, the =:stop= of A is the same as the =:start= of B, and that all their other tags are identical. Then we can re-split them according to the new day-start-time.
   + Implemented as [[#program-data-structures-plists-split-p][plist-split-p]]
2. Add a =:split= tag to split events. It can denote that the next event was originally a part of this one.
3. Re-check and update the file when the day-start-time changes.
   - Possible with ~add-variable-watcher~ or ~:custom-set~ in Customize (thanks bpalmer)

This strategy is implemented in the [[#program-backend-plist-group][plist-group]] backend. Custom day-start time is not yet implemented - if ever implemented, it will probably require exporting the file, with all split intervals being combined and re-split.

*** Split them at the hash-table-level
Handled by ~sexp-events-populate~
+ Advantage - simpler data-consuming code.

*** Split them at the data-consumer level (e.g. when calculating time for one day/getting events for one day)
+ Advantage - reduced repetitive post-parsing load.

** Point restore behaviour
:PROPERTIES:
:DESCRIPTION: The desired behaviour of point in Chronometrist
:END:
After hacking, always test for and ensure the following -
1. Toggling the buffer via [[#program-frontend-chronometrist-command][chronometrist]]/[[#program-frontend-report-command][report]]/[[#program-frontend-statistics-command][statistics]] should preserve point
2. The timer function should preserve point when the buffer is current
3. The timer function should preserve point when the buffer is not current, but is visible in another window
4. The next/previous week keys and buttons should preserve point.

** report date range logic
:PROPERTIES:
:DESCRIPTION: Deriving dates in the current week
:END:
A quick description, starting from the first time [[#program-frontend-report-command][report]] is run in an Emacs session -
1. We get the current date as a =ts= struct, using date.
2. The variable =week-start-day= stores the day we consider the week to start with. The default is "Sunday".

   We check if the date from #2 is on the week start day, else decrement it till we are, using =(previous-week-start)=.
3. We store the date from #3 in the global variable =-ui-date=.
4. By counting up from =-ui-date=, we get dates for the days in the next 7 days using =(date->dates-in-week)=. We store them in =-ui-week-dates=.

   The dates in =-ui-week-dates= are what is finally used to query the data displayed in the buffer.
5. To get data for the previous/next weeks, we decrement/increment the date in =-ui-date= by 7 days and repeat the above process (via =(previous-week)= / =(next-week)=).

** Literate programming
:PROPERTIES:
:CUSTOM_ID: explanation-literate-programming
:END:
The shift from a bunch of Elisp files to a single Org literate program was born out of frustration with programs stored as text files, which are expensive to restructure (especially in the presence of a VCS). While some dissatisfactions remain, I generally prefer the outcome - tree and source-block folding, tags, properties, and =org-match= have made it trivial to get different views of the program, and literate programming may allow me to express the "explanation" documentation in the same context as the program, without having to try to link between documentation and source.

*** Tangling
At first, I tried tangling. Back when I used =benchmark.el= to test it, =org-babel-tangle= took about 30 seconds to tangle this file. Thus, I wrote a little sed one-liner (in the file-local variables) to do the tangling, which was nearly instant. It emitted anything between lines matching the exact strings ="#+BEGIN_SRC lisp"= and ="#+END_SRC"= -

#+BEGIN_SRC org :tangle no
# eval: (progn (make-local-variable 'after-save-hook) (add-hook 'after-save-hook (lambda () (start-process-shell-command "sed-tangle" "sed-tangle" "sed -n -e '/#+BEGIN_SRC lisp$/,/#+END_SRC$/{//!p;};/#+END_SRC/i\\ ' chronometrist.org | sed -E 's/^ +$//' > chronometrist.el"))))
#+END_SRC

*** literate-elisp-load
Later, we switched from tangling to using the =literate-elisp= package to loading this Org file directly - a file =chronometrist.el= would be used to load =chronometrist.org=.

#+BEGIN_SRC lisp :tangle no :load no
(literate-elisp-load
 (format "%schronometrist.org" (file-name-directory load-file-name)))
#+END_SRC

This way, source links (e.g. help buffers, stack traces) would lead to this Org file, and this documentation was available to each user, within the comfort of their Emacs. The presence of the =.el= file meant that users of =use-package= did not need to make any changes to their configuration.

*** Reject modernity, return to tangling
For all its benefits, the previous approach broke autoloads and no sane way could be devised to make them work, so back we came to tangling. =org-babel-tangle-file= seems to be quicker when run as a Git pre-commit hook - a few seconds' delay before I write a commit message.

Certain tools like =checkdoc= remain a pain to use with any kind of literate program. This will probably continue to be the case until these tools are fixed or extended.

*** Definition metadata
Each definition has its own heading. The type of definition is stored in tags -
1. custom group
2. [custom|hook|internal] variable
3. keymap (use variable instead?)
4. macro
5. function
   * does not refer to external state
   * primarily used for the return value
6. reader
   * reads external state without modifying it
   * primarily used for the return value
7. writer
   * modifies external state, namely a data structure or file
   * primarily used for side-effects
8. procedure
   * any other impure function
   * usually affects the display
   * primarily used for side-effects
9. major/minor mode
10. command

Further details are stored in properties -
1. :INPUT: (for functions)
2. :VALUE: list|hash table|...
   * for functions, this is the return value
3. :STATE: <external file or data structure read or written to>

*** TODO Issues [40%]
1. [X] When opening this file, Emacs may freeze at the prompt for file-local variable values; if so, C-g will quit the prompt, and permanently marking them as safe will make the freezing stop. [fn:3]
2. [ ] I like =visual-fill-column-mode= for natural language, but I don't want it applied to code blocks. =polymode.el= may hold answers.
3. [X] Is there a tangling solution which requires only one command (e.g. currently we use two =sed= s) but is equally fast? [fn:3]
   * Perhaps we can get rid of the requirement of adding newlines after each source block, and add the newlines ourselves. That gives us control, and also makes it possible to insert Org text in the middle of a definition without unnecessary newlines.
4. [ ] =nameless-insert-name= does not work in source blocks.
5. [ ] Some source blocks don't get syntax highlighted.
   * A workaround is to press =M-o M-o=

[fn:3] No longer a problem since we switched to =literate-elisp=

** Currently-Used Time Formats
:PROPERTIES:
:CUSTOM_ID: explanation-time-formats
:END:

*** ts
  ts.el struct
  * Used by nearly all internal functions

*** iso-timestamp
 ="YYYY-MM-DDTHH:MM:SSZ"=
  * Used in the s-expression file format
  * Read by sexp-events-populate
  * Used in the plists in the events hash table values

*** iso-date
 ="YYYY-MM-DD"=
  * Used as hash table keys in events - can't use ts structs for keys, you'd have to make a hash table predicate which uses ts=

*** seconds
  integer seconds as duration
  * Used for most durations
  * May be changed to floating point to allow larger durations. The minimum range of =most-positive-fixnum= is 536870911, which seems to be enough to represent durations of 17 years.
  * Used for update intervals (update-interval, change-update-interval)

*** minutes
  integer minutes as duration
  * Used by [[https://tildegit.org/contrapunctus/chronometrist-goal][goal]] (goals-list, get-goal) - minutes seems like the ideal unit for users to enter

*** list-duration
 =(hours minute seconds)=
  * Only returned by =seconds-to-hms=, called by =format-time=

** Backend protocol
:PROPERTIES:
:CREATED:  2022-01-05T19:00:12+0530
:END:
The protocol as of now, with remarks -
1. =backend-run-assertions (backend)=
2. =view-backend (backend)=
3. =edit-backend (backend)= - these two would make sense if there was only one way to view/edit a backend. But if we want [[file:TODO.org::#viewing-editing-frontends][viewing/editing frontends]], there would be many.
4. =backend-empty-p (backend)=
5. =backend-modified-p (backend)=
6. =create-file (backend &optional file)=
7. =latest-date-records (backend)=
8. =insert (backend plist)=
9. =remove-last (backend)=
10. =latest-record (backend)=
11. =task-records-for-date (backend task date-ts)=
12. =replace-last (backend plist)=
13. =to-file (input-hash-table output-backend output-file)=
14. =on-add (backend)=
15. =on-modify (backend)=
16. =on-remove (backend)=
17. =on-change (backend)=
18. =verify (backend)=
19. =on-file-path-change (backend old-path new-path)=
20. =reset-backend (backend)= - probably rename to "initialize"
21. =memory-layer-empty-p (backend)= - needs a more generic name; perhaps "initialized-p", to go with #20
22. =to-hash-table (backend)=
23. =to-list (backend)=

There are many operations which are file-oriented, whereas I have tried to treat files as implementation details. =create-file=, for instance, is used by =to-file=; I could make creation of files implicit by moving it into =initialize-instance=, but that would mean creation of files in =to-file= would require creation of a backend object. That seems to me to be an abuse of implicit behaviour; and what would backends which are not file-backed do in =to-file=, then? There's probably a way to do it, but I had other things I preferred to tackle first.

** generic loop interface for iterating over records
Of all the ways to work with Chronometrist data, both as part of the program and as part of my occasional "queries", my favorite was to use =cl-loop=.

First, there was the =loop-file= macro, which handled the sole backend at that time - the plist backend. It took care of the common logic (=read= ing each plist in the file, checking loop termination conditions), and let the client code specify (with the terseness of =cl-loop=) what they wanted to do with the data.

During the migration to the CLOS-based backend design began the quest to make =loop-file= work with generic backends - it eventually became =loop-records= and =loop-days=. The idea was to call a generic function (=record-iterator= and =day-iterator=) which would return a new record on each call. Internal state of each of these generic functions was stored in backend slots. No list would be built up, unless the client code specified an accumulation clause.

Most recently, gilberth of #lispcafe suggested an alternate approach - trying to build a list of records first, and using =cl-loop= (or any other iteration mechanism) on that. Testing the two approaches yielded a clear advantage for this new suggestion. The test was to generate key-values suitable for completion history from my full Chronometrist data to date (the plist backend had ~6.6k plists in a 1.2M file), using almost identical =cl-loop= client code for both cases. Here was the output from =(benchmark 1 ...)= -

| approach | backend     | benchmark output                                 |
|----------+-------------+--------------------------------------------------|
| current  | plist       | "Elapsed time: 5.322056s (0.709023s in 4 GCs)"   |
| current  | plist-group | "Elapsed time: 107.159170s (1.064125s in 6 GCs)" |
| new      | plist       | "Elapsed time: 0.559264s (0.172344s in 1 GCs)"   |
| new      | plist-group | "Elapsed time: 0.671106s (0.179435s in 1 GCs)"   |

In addition, with this approach, client code can use any kind of iteration constructs they fancy - not just =cl-loop= but also =dolist=, higher-order functions (including those from =dash= and =seq=), =loopy=, etc.

The macro still exists in its non-generic form as =loop-sexp-file=, providing a common way to loop over s-expressions in a text file, used by =to-list= in both backends and =to-hash-table= in the plist group backend.

* How-to guides for maintainers
** How to set up Emacs to contribute
# Different approaches to this setup -
# 1. Document it and let contributors do it voluntarily.
#    * Downside - if a contributor does not do it, it can lead to inconsistency and an additional headache for the maintainer.
# 2. Put it in file variables and/or =.dir-locals.el=.
#    * Downside - if the contributor does not have a dependency installed (e.g. nameless), Emacs will create an error. That's not the UX I want for someone opening the file for the first time - it does not tell them that they need to install something.
#      + Technically, a friendlier error message could be displayed. But you want to read/edit a document and it asks you to install something first...kind of a flow breaker.
# 3. Use a tool which installs developement dependencies for you, e.g. Cask, Eldev.
#    * May look into this in the future. But as of now I don't even foresee any contributors 😓

1. Install [[https://github.com/Malabarba/Nameless][nameless-mode]] for easier reading of Emacs Lisp code, and [[https://github.com/jingtaozf/literate-elisp][literate-elisp]] to load this file directly without tangling.

   #+BEGIN_SRC lisp :tangle no :load no
     (mapcar #'package-install '(nameless literate-elisp))
   #+END_SRC

2. Create a =.dir-locals-2.el= in the project root, containing -

      #+BEGIN_SRC lisp :tangle no :load no
        ((org-mode
          .
          ((eval . (nameless-mode))
           (eval . (progn
             (make-local-variable 'after-save-hook)
             ;; you can't `defun' in one `eval' and use the
             ;; function in another `eval', apparently
             (add-hook
              'after-save-hook
              (defun tangle ()
                (interactive)
                (compile
                 (mapconcat #'shell-quote-argument
                            `("emacs" "-q" "-Q" "--batch"
                              "--eval=(require 'ob-tangle)"
                              ,(format "--eval=(org-babel-tangle-file \"%s\")"
                                       (buffer-file-name)))
                            " ")))))))))
      #+END_SRC

3. Set up compiling, linting, and testing with =makem.sh=. First, define this command -

   #+BEGIN_SRC lisp :tangle no :load no
   (defun run-makem ()
     (interactive)
     (cd (locate-dominating-file default-directory "makem.sh"))
     (compile "./makem.sh compile lint test-ert"))
   #+END_SRC

   Then, run it after staging the files -
   #+BEGIN_SRC lisp :tangle no :load no
   (add-hook 'magit-post-stage-hook #'run-makem)
   #+END_SRC

   Or after tangling ends -
   #+BEGIN_SRC lisp :tangle no :load no
     (add-hook 'org-babel-post-tangle-hook #'run-makem)
   #+END_SRC

** How to tangle this file
Use =org-babel= (=org-babel-tangle= / =org-babel-tangle-file=), /not/ =literate-elisp-tangle=. The file emitted by the latter does not contain comments - thus, it does not contain library headers or abide by =checkdoc='s comment conventions.

* The Program
** Package
#+BEGIN_SRC lisp
(in-package :cl)
(defpackage :chronometrist
  (:use :cl)
  (:export
   ;; customizable variables
   :day-start-time
   :week-start-day
   :weekday-number-alist
   :file
   :active-backend
   :sexp-pretty-print-function))
(in-package :chronometrist)
#+END_SRC

** Common definitions
*** format-time                                                  :function:
#+BEGIN_SRC lisp
(defun format-duration (seconds &optional (blank (make-string 3 ?\s)))
  "Format SECONDS as a string suitable for display in Chronometrist buffers.
SECONDS must be a positive integer.

BLANK is a string to display in place of blank values. If not
supplied, 3 spaces are used."
  (-let [(h m s) (seconds-to-hms seconds)]
    (if (and (zerop h) (zerop m) (zerop s))
        (concat (make-string 7 ?\s) "-")
      (let ((h (if (zerop h) blank (format "%2d:" h)))
            (m (cond ((and (zerop h) (zerop m))  blank)
                     ((zerop h)  (format "%2d:" m))
                     (t  (format "%02d:" m))))
            (s (if (and (zerop h) (zerop m))
                   (format "%2d" s)
                 (format "%02d" s))))
        (concat h m s)))))
#+END_SRC

*** file-empty-p                                                   :reader:
#+BEGIN_SRC lisp
(defun file-empty-p (file)
  "Return t if FILE is empty."
  (zerop (nth 7 (file-attributes file))))
#+END_SRC

*** format-keybinds                                              :function:
#+BEGIN_SRC lisp
(defun format-keybinds (command map &optional firstonly)
  "Return the keybindings for COMMAND in MAP as a string.
If FIRSTONLY is non-nil, return only the first keybinding found."
  (if firstonly
      (key-description
       (where-is-internal command map firstonly))
    (->> (where-is-internal command map)
         (mapcar #'key-description)
         (-take 2)
         (-interpose ", ")
         (apply #'concat))))
#+END_SRC

*** day-start-time                                        :custom:variable:
=events-maybe-split= refers to this, but I'm not sure this has the desired effect at the moment—haven't even tried using it.
#+BEGIN_SRC lisp
(defcustom day-start-time "00:00:00"
  "The time at which a day is considered to start, in \"HH:MM:SS\".

The default is midnight, i.e. \"00:00:00\"."
  :type 'string)
#+END_SRC

*** week-start-day                                        :custom:variable:
#+BEGIN_SRC lisp
(defcustom week-start-day "Sunday"
  "The day used for start of week by `report'."
  :type 'string)
#+END_SRC

*** weekday-number-alist                                  :custom:variable:
#+BEGIN_SRC lisp
(defcustom weekday-number-alist
  '(("Sunday"    . 0)
    ("Monday"    . 1)
    ("Tuesday"   . 2)
    ("Wednesday" . 3)
    ("Thursday"  . 4)
    ("Friday"    . 5)
    ("Saturday"  . 6))
  "Alist in the form (\"NAME\" . NUMBER), where \"NAME\" is the name of a weekday and NUMBER its associated number."
  :type 'alist)
#+END_SRC

*** previous-week-start                                            :reader:
#+BEGIN_SRC lisp
(defun previous-week-start (ts)
  "Find the previous `week-start-day' from TS.
Return a ts struct for said day's beginning.

If the day of TS is the same as the
`week-start-day', return TS.

TS must be a ts struct (see `ts.el')."
  (loop with week-start = (alist-get week-start-day
                                     weekday-number-alist nil nil #'equal)
        until (= week-start (ts-dow ts))
        do (ts-decf (ts-day ts))
        finally return ts))
#+END_SRC

*** plist-remove                                                 :function:
#+BEGIN_SRC lisp
(defun plist-remove (plist &rest keys)
  "Return PLIST with KEYS and their associated values removed."
  (let ((keys (--filter (plist-member plist it) keys)))
    (mapc (lambda (key)
            (let ((pos (seq-position plist key)))
              (setq plist (append (seq-take plist pos)
                                  (seq-drop plist (+ 2 pos))))))
          keys)
    plist))
#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest plist-remove ()
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :a)
          '(:b 2 :c 3 :d 4)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :b)
          '(:a 1 :c 3 :d 4)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :c)
          '(:a 1 :b 2 :d 4)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :d)
          '(:a 1 :b 2 :c 3)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :a :b)
          '(:c 3 :d 4)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :a :d)
          '(:b 2 :c 3)))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :c :d)
          '(:a 1 :b 2)))
  (should (equal
           (plist-remove '(:a 1 :b 2 :c 3 :d 4) :a :b :c :d)
           nil))
  (should
   (equal (plist-remove '(:a 1 :b 2 :c 3 :d 4) :d :a)
          '(:b 2 :c 3))))

#+END_SRC

*** plist-key-values                                             :function:
#+BEGIN_SRC lisp
(defun plist-key-values (plist)
  "Return user key-values from PLIST."
  (plist-remove plist :name :tags :start :stop))
#+END_SRC

*** plist-p                                                      :function:
[[file:../tests/chronometrist-tests.org::#tests-common-plist-p][tests]]

#+BEGIN_SRC lisp
(defun plist-p (list)
  "Return non-nil if LIST is a property list, i.e. (:KEYWORD VALUE ...)"
  (when list
    (while (consp list)
      (setq list (if (and (keywordp (first list)) (consp (rest list)))
                     (cddr list)
                   'not-plist)))
    (null list)))
#+END_SRC

*** plist type                                                       :type:
#+BEGIN_SRC lisp
(deftype plist ()
  '(satisfies plist-p)
  '(satisfies (lambda (plist)
                (and (plist-get plist :name)
                     (plist-get plist :start)))))
#+END_SRC

*** delete-list                                                    :writer:
#+BEGIN_SRC lisp
(defun sexp-delete-list (&optional arg)
  "Delete ARG lists after point.
Return new position of point."
  (let ((point-1 (point)))
    ;; try to preserve the file-local variable prop line in case this
    ;; is run from the start of buffer
    (while (forward-comment 1) nil)
    (forward-sexp (or arg 1))
    (delete-region point-1 (point))
    (point)))
#+END_SRC

*** make-hash-table-1                                            :function:
#+BEGIN_SRC lisp
(defun make-hash-table-1 ()
  "Return an empty hash table with `equal' as test."
  (make-hash-table :test #'equal))
#+END_SRC

*** current-task                                            :reader:method:
[[file:../tests/chronometrist-tests.org::#tests-common-current-task][tests]]

#+BEGIN_SRC lisp
(defun current-task (&optional (backend (active-backend)))
  "Return the name of the active task as a string, or nil if not clocked in."
  (let ((last-event (latest-record backend)))
    (if (plist-member last-event :stop)
        nil
      (plist-get last-event :name))))
#+END_SRC

*** install-directory                                            :variable:
#+BEGIN_SRC lisp
(defvar install-directory
  (when load-file-name
    (file-name-directory load-file-name))
  "Directory where Chronometrist has been installed.")
#+END_SRC

*** doc-paths                                                    :variable:
#+BEGIN_SRC lisp
(defvar doc-paths '(:lp "chronometrist.org")
  "Plist of names of Chronometrist's documentation files.")
#+END_SRC

*** open-literate-source                                          :command:
#+BEGIN_SRC lisp
(defun open-literate-source ()
  "Visit the Org literate program for Chronometrist."
  (interactive)
  (find-file (concat install-directory (plist-get doc-paths :lp))))
#+END_SRC

*** debug logging
#+BEGIN_SRC lisp
(defcustom debug-enable nil
  "Whether to log debugging messages."
  :type 'boolean
  :group 'chronometrist)

(defcustom debug-buffer "*debug*"
  "Name of buffer to log debug messages to."
  :type 'string
  :group 'chronometrist)

(define-derived-mode debug-log-mode special-mode "debug-log")

(defun debug-message (format-string &rest args)
  "Log a debug message to `debug-buffer'.
FORMAT-STRING and ARGS are passed to `format'."
  (when debug-enable
    (with-current-buffer (get-buffer-create debug-buffer)
      (goto-char (point-max))
      (debug-log-mode)
      (let ((inhibit-read-only t))
        (insert
         (apply #'format
                (concat (format-time-string "[%T] ")
                        format-string)
                args)
         "\n")))))
#+END_SRC

** Data structures
:PROPERTIES:
:CUSTOM_ID: program-data-structures
:END:
Reading directly from the file could be difficult, especially when your most common query is "get all intervals recorded on <date>" [fn:4] - and so, we maintain the hash table =events=, where each key is a date in the ISO-8601 format. The plists in this hash table are free of [[#explanation-midnight-spanning-intervals][midnight-spanning intervals]], making code which consumes it easier to write.

The data from =events= is used by most (all?) interval-consuming functions, but is never written to the user's file itself.

[fn:4] it might be the case that the [[#program-backend][file format]] is not suited to our most frequent operation...

*** reset                                                         :command:
#+BEGIN_SRC lisp
(defun reset ()
  "Reset Chronometrist's internal state."
  (interactive)
  (debug-message "[Command] reset")
  (reset-backend (active-backend))
  (refresh))
#+END_SRC

*** apply-time                                                   :function:
#+BEGIN_SRC lisp
(defun apply-time (time timestamp)
  "Return TIMESTAMP with time modified to TIME.
TIME must be a string in the form \"HH:MM:SS\"

TIMESTAMP must be a time string in the ISO-8601 format.

Return value is a ts struct (see `ts.el')."
  (-let [(h m s) (mapcar #'string-to-number (split-string time ":"))]
    (ts-apply :hour h :minute m :second s
              (iso-to-ts timestamp))))
#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest apply-time ()
  (should
   (equal (ts-format "%FT%T%z" (apply-time "01:02:03" "2021-02-17T01:20:18+0530"))
          "2021-02-17T01:02:03+0530")))
#+END_SRC

*** split-plist                                                  :function:
#+BEGIN_SRC lisp
(defun split-plist (plist)
  "Return a list of two split plists if PLIST spans a midnight, else nil."
  (when (plist-get plist :stop)
    (let ((split-time (split-time (plist-get plist :start)
                                           (plist-get plist :stop)
                                           day-start-time)))
      (when split-time
        (-let* (((&plist :start start-1 :stop stop-1) (first  split-time))
                ((&plist :start start-2 :stop stop-2) (second split-time))
                ;; `plist-put' modifies lists in-place. The resulting bugs
                ;; left me puzzled for a while.
                (event-1      (copy-list plist))
                (event-2      (copy-list plist)))
          (list (-> event-1
                    (plist-put :start start-1)
                    (plist-put :stop  stop-1))
                (-> event-2
                    (plist-put :start start-2)
                    (plist-put :stop  stop-2))))))))
#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest split-plist ()
  (should
   (null (split-plist
          '(:name  "Task"
                   :start "2021-02-17T01:33:12+0530"
                   :stop  "2021-02-17T01:56:08+0530"))))
  (should
   (equal (split-plist
           '(:name  "Guitar"
                    :tags  (classical warm-up)
                    :start "2021-02-12T23:45:21+0530"
                    :stop  "2021-02-13T00:03:46+0530"))
          '((:name "Guitar"
                   :tags (classical warm-up)
                   :start "2021-02-12T23:45:21+0530"
                   :stop "2021-02-13T00:00:00+0530")
            (:name "Guitar"
                   :tags (classical warm-up)
                   :start "2021-02-13T00:00:00+0530"
                   :stop "2021-02-13T00:03:46+0530")))))
#+END_SRC

*** ht-update                                                      :writer:
#+BEGIN_SRC lisp
(defun ht-update (plist hash-table &optional replace)
  "Return HASH-TABLE with PLIST added as the latest interval.
If REPLACE is non-nil, replace the last interval with PLIST."
  (let* ((date (->> (plist-get plist :start)
                    (iso-to-ts )
                    (ts-format "%F" )))
         (events-today (gethash date hash-table)))
    (--> (if replace (-drop-last 1 events-today) events-today)
         (append it (list plist))
         (puthash date it hash-table))
    hash-table))
#+END_SRC

*** ht-last-date                                                   :reader:
#+BEGIN_SRC lisp
(defun ht-last-date (hash-table)
  "Return an ISO-8601 date string for the latest date present in `events'."
  (--> (hash-table-keys hash-table)
       (sort it #'string-lessp)
       (last it)
       (first it)))
#+END_SRC

*** ht-last                                                        :reader:
#+BEGIN_SRC lisp
(defun ht-last (&optional (backend (active-backend)))
  "Return the last plist from `events'."
  (let* ((hash-table (backend-hash-table backend))
         (last-date  (ht-last-date hash-table)))
    (--> (gethash last-date hash-table)
         (last it)
         (car it))))
#+END_SRC

*** ht-subset                                                      :reader:
:PROPERTIES:
:VALUE:    hash table
:CUSTOM_ID: program-data-structures-ht-subset
:END:
#+BEGIN_SRC lisp
(defun ht-subset (start end hash-table)
  "Return a subset of HASH-TABLE.
The subset will contain values between dates START and END (both
inclusive).

START and END must be ts structs (see `ts.el'). They will be
treated as though their time is 00:00:00."
  (let ((subset (make-hash-table-1))
        (start  (date-ts start))
        (end    (date-ts end)))
    (maphash (lambda (key value)
               (when (ts-in start end (iso-to-ts key))
                 (puthash key value subset)))
             hash-table)
    subset))
#+END_SRC

*** task-time-one-day                                              :reader:
#+BEGIN_SRC lisp
(defun task-time-one-day (task &optional (date (date-ts)) (backend (active-backend)))
  "Return total time spent on TASK today or on DATE, an ISO-8601 date.
The return value is seconds, as an integer."
  (let ((task-events (task-records-for-date backend task date)))
    (if task-events
        (->> (plists-to-durations task-events)
             (-reduce #'+)
             (truncate))
      ;; no events for this task on DATE, i.e. no time spent
      0)))
#+END_SRC

*** active-time-on                                                 :reader:
#+BEGIN_SRC lisp
(defvar task-list)
(defun active-time-on (&optional (date (date-ts)))
  "Return the total active time today, or on DATE.
Return value is seconds as an integer."
  (->> (--map (task-time-one-day it date) (task-list))
       (-reduce #'+)
       (truncate)))
#+END_SRC

*** count-active-days                                            :function:
#+BEGIN_SRC lisp
(defun statistics-count-active-days (task table)
  "Return the number of days the user spent any time on TASK.
  TABLE must be a hash table - if not supplied, `events' is used.

  This will not return correct results if TABLE contains records
which span midnights."
  (loop for events being the hash-values of table
    count (seq-find (lambda (event)
                      (equal task (plist-get event :name)))
                    events)))
#+END_SRC

*** task-list                                                    :variable:
:PROPERTIES:
:VALUE:    list
:END:
#+BEGIN_SRC lisp
(defcustom task-list nil
  "List of tasks used by `chronometrist'.
Value may be either nil or a list of strings.

If nil, the task list is generated from user data in
`file' and stored in the task-list slot of the
active backend."
  :type '(choice (repeat string) nil)
  :group 'chronometrist)
#+END_SRC

** Time functions
*** iso-to-ts                                                    :function:
#+BEGIN_SRC lisp
(defun iso-to-ts (timestamp)
  "Convert TIMESTAMP to a TS struct. (see `ts.el')
TIMESTAMP must be an ISO-8601 timestamp, as handled by
`parse-iso8601-time-string'."
  (-let [(second minute hour day month year dow _dst utcoff)
         (decode-time
          (parse-iso8601-time-string timestamp))]
    (ts-update
     (make-ts :hour hour :minute minute :second second
              :day day   :month month   :year year
              :dow dow   :tz-offset utcoff))))

#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest iso-to-ts ()
  (should (ts= (iso-to-ts "2021-01-01")
               (make-ts :year 2021 :month 1  :day 1
                        :hour 0    :minute 0 :second 0)))
  (should (ts= (iso-to-ts "2021-01-01T01:01:01")
               (make-ts :year 2021 :month 1  :day 1
                        :hour 1    :minute 1 :second 1))))

#+END_SRC

*** plists-to-durations                                          :function:
#+BEGIN_SRC lisp
(defun plists-to-durations (plists)
  "Convert PLISTS into a list of durations in seconds.
PLISTS must be a list of valid Chronometrist property lists (see
`file'). Return 0 if PLISTS is nil."
  (if plists (mapcar #'interval plists) 0))
#+END_SRC

*** date-iso                                                     :function:
#+BEGIN_SRC lisp
(defun date-iso (&optional (ts (ts-now)))
  (ts-format "%F" ts))
#+END_SRC

*** date-ts                                                      :function:
#+BEGIN_SRC lisp
(defun date-ts (&optional (ts (ts-now)))
  "Return a ts struct representing the time 00:00:00 on today's date.
If TS is supplied, use that date instead of today.
TS should be a ts struct (see `ts.el')."
  (ts-apply :hour 0 :minute 0 :second 0 ts))
#+END_SRC

*** format-time-iso8601                                          :function:
#+BEGIN_SRC lisp
(defun format-time-iso8601 (&optional unix-time)
  "Return current date and time as an ISO-8601 timestamp.
Optional argument UNIX-TIME should be a time value (see
`current-time') accepted by `format-time-string'."
  (format-time-string "%FT%T%z" unix-time))

;; Note - this assumes that an event never crosses >1 day. This seems
;; sufficient for all conceivable cases.
#+END_SRC

*** FIXME split-time                                               :reader:
It does not matter here that the =:stop= dates in the returned plists are different from the =:start=, because =events-populate= uses only the date segment of the =:start= values as hash table keys. (The hash table keys form the rest of the program's notion of "days", and that of which plists belong to which day.)

#+BEGIN_SRC lisp
(defun split-time (start-time stop-time day-start-time)
  "If START-TIME and STOP-TIME intersect DAY-START-TIME, split them into two intervals.
START-TIME and STOP-TIME must be ISO-8601 timestamps e.g. \"YYYY-MM-DDTHH:MM:SSZ\".

DAY-START-TIME must be a string in the form \"HH:MM:SS\" (see
`day-start-time')

Return a list in the form
\((:start START-TIME
  :stop <day-start time on initial day>)
 (:start <day start time on second day>
  :stop STOP-TIME))"
  ;; FIXME - time zones are ignored; may cause issues with
  ;; time-zone-spanning events

  ;; The time on which the first provided day starts (according to `day-start-time')
  (let* ((stop-ts         (iso-to-ts stop-time))
         (first-day-start (apply-time day-start-time start-time))
         (next-day-start  (ts-adjust 'hour 24 first-day-start)))
    ;; Does the event stop time exceed the next day start time?
    (when (ts< next-day-start stop-ts)
      (let ((split-time (ts-format "%FT%T%z" next-day-start)))
        (list `(:start ,start-time :stop ,split-time)
              `(:start ,split-time :stop ,stop-time))))))
#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest split-time ()
  (should
   (null
    (split-time "2021-02-17T01:33:12+0530"
                 "2021-02-17T01:56:08+0530"
                 "00:00:00")))
  (should
   (equal
    (split-time "2021-02-19T23:45:36+0530"
                 "2021-02-20T00:18:40+0530"
                 "00:00:00")
    '((:start "2021-02-19T23:45:36+0530"
              :stop "2021-02-20T00:00:00+0530")
      (:start "2021-02-20T00:00:00+0530"
              :stop "2021-02-20T00:18:40+0530"))))
  (should
   (equal
    (split-time "2021-02-19T23:45:36+0530"
                 "2021-02-20T03:18:40+0530"
                 "01:20:30")
    '((:start "2021-02-19T23:45:36+0530"
              :stop "2021-02-20T01:20:30+0530")
      (:start "2021-02-20T01:20:30+0530"
              :stop "2021-02-20T03:18:40+0530")))))
#+END_SRC

*** seconds-to-hms                                               :function:
#+BEGIN_SRC lisp
(defun seconds-to-hms (seconds)
  "Convert SECONDS to a vector in the form [HOURS MINUTES SECONDS].
SECONDS must be a positive integer."
  (let* ((seconds (truncate seconds))
         (s       (% seconds 60))
         (m       (% (/ seconds 60) 60))
         (h       (/ seconds 3600)))
    (list h m s)))
#+END_SRC

*** interval                                                     :function:
#+BEGIN_SRC lisp
(defun interval (plist)
  "Return the period of time covered by EVENT as a time value.
EVENT should be a plist (see `file')."
  (let* ((start-ts  (iso-to-ts (plist-get plist :start)))
         (stop-iso  (plist-get plist :stop))
         ;; Add a stop time if it does not exist.
         (stop-ts   (if stop-iso (iso-to-ts stop-iso) (ts-now))))
    (ts-diff stop-ts start-ts)))
#+END_SRC

*** format-duration-long                                         :function:
#+BEGIN_SRC lisp
(defun format-duration-long (seconds)
  "Return SECONDS as a human-friendly duration string.
e.g. \"2 hours, 10 minutes\". SECONDS must be an integer. If
SECONDS is less than 60, return a blank string."
  (let* ((hours         (/ seconds 60 60))
         (minutes       (% (/ seconds 60) 60))
         (hour-string   (if (= 1 hours) "hour" "hours"))
         (minute-string (if (= 1 minutes) "minute" "minutes")))
    (cond ((and (zerop hours) (zerop minutes)) "")
          ((zerop hours)
           (format "%s %s" minutes minute-string))
          ((zerop minutes)
           (format "%s %s" hours hour-string))
          (t (format "%s %s, %s %s"
                     hours hour-string
                     minutes minute-string)))))
#+END_SRC

**** tests
#+BEGIN_SRC lisp :tangle ../tests/tests.el :load test
(ert-deftest format-duration-long ()
  (should (equal (format-duration-long 5) ""))
  (should (equal (format-duration-long 65) "1 minute"))
  (should (equal (format-duration-long 125) "2 minutes"))

  (should (equal (format-duration-long 3605) "1 hour"))
  (should (equal (format-duration-long 3660) "1 hour, 1 minute"))
  (should (equal (format-duration-long 3725) "1 hour, 2 minutes"))

  (should (equal (format-duration-long 7200) "2 hours"))
  (should (equal (format-duration-long 7260) "2 hours, 1 minute"))
  (should (equal (format-duration-long 7320) "2 hours, 2 minutes")))
#+END_SRC

*** iso-to-date                                                  :function:
#+BEGIN_SRC lisp
(defun iso-to-date (timestamp)
  (first (split-string timestamp "T")))
#+END_SRC

** Backends
:PROPERTIES:
:CUSTOM_ID: program-backend
:END:
*** file                                                 :custom:variable:
#+BEGIN_SRC lisp
(defcustom file (locate-user-emacs-file "chronometrist")
  "Name (without extension) and full path of the Chronometrist database."
  :type 'file)

(defun file-variable-watcher (_symbol newval _operation _where)
  "Update slots of the active backend when `file' is changed.
For SYMBOL, NEWVAL, OPERATION, and WHERE, see `add-variable-watcher'."
  (on-file-path-change (active-backend) file newval))

(add-variable-watcher 'file #'file-variable-watcher)
#+END_SRC

*** protocol
**** backend                                                       :class:
The backend may use no files, a single file, or multiple files. Thus, =backend= makes no reference to files, and [[#file-backend-mixin][=file-backend-mixin=]] may be used by single file backends.

#+BEGIN_SRC lisp
(defclass backend ()
  ((task-list :initform nil
              :initarg :task-list
              :accessor backend-task-list)))
#+END_SRC

**** backends-alist                                             :variable:
#+BEGIN_SRC lisp
(defvar backends-alist nil
  "Alist of Chronometrist backends.
Each element must be in the form `(KEYWORD TAG OBJECT)', where
TAG is a string used as a tag in customization, and OBJECT is an
EIEIO object such as one returned by `make-instance'.")
#+END_SRC

**** active-backend                                      :custom:variable:
#+BEGIN_SRC lisp
(defcustom active-backend :plist
  "The backend currently in use.
Value must be a keyword corresponding to a key in
`backends-alist'."
  :type `(choice
          ,@(loop for elt in backends-alist
              collect `(const :tag ,(second elt) ,(first elt)))))
#+END_SRC

**** active-backend                                               :reader:
#+BEGIN_SRC lisp
(defun active-backend ()
  "Return an object representing the currently active backend."
  (second (alist-get active-backend backends-alist)))
#+END_SRC

**** switch-backend                                              :command:
#+BEGIN_SRC lisp
(defun switch-backend ()
  (interactive)
  (debug-message "[Command] switch-backend")
  (let* ((prompt (format "Switch to backend (current - %s): "
                         active-backend))
         (choice (read-backend-name prompt
                                     backends-alist
                                     (lambda (keyword)
                                       (not (eq active-backend
                                                keyword)))
                                     t)))
    (setq active-backend choice)
    (reset-backend (active-backend))
    ;; timer function is backend-dependent
    (force-restart-timer)))
#+END_SRC

**** register-backend                                             :writer:
#+BEGIN_SRC lisp
(defun register-backend (keyword tag object)
  "Add backend to `backends-alist'.
For values of KEYWORD, TAG, and OBJECT, see `backends-alist'.

If a backend with KEYWORD already exists, the existing entry will
be replaced."
  (setq backends-alist
        (assq-delete-all keyword backends-alist))
  (add-to-list 'backends-alist
               (list keyword tag object)
               t))
#+END_SRC

**** read-backend-name                                         :procedure:
#+BEGIN_SRC lisp
(defun read-backend-name (prompt backend-alist
                                  &optional predicate return-keyword)
  "Prompt user for a Chronometrist backend name.
BACKEND-ALIST should be an alist similar to `backends-alist'.

RETURN-KEYWORD, if non-nil, means return only the keyword of the
selected backend; otherwise, return the CLOS object for the
backend.

PROMPT and PREDICATE have the same meanings as in
`completing-read'."
  (let ((backend-keyword
         (read
          (completing-read prompt
                           (loop for list in backend-alist
                             collect (first list))
                           predicate t))))
    (if return-keyword
        backend-keyword
      (second (alist-get backend-keyword backend-alist)))))
#+END_SRC

**** task-list                                                  :function:
#+BEGIN_SRC lisp
(defun task-list ()
  "Return the list of tasks to be used.
If `task-list' is non-nil, return its value; else,
return a list of tasks from the active backend."
  (let ((backend (active-backend)))
    (with-slots (task-list) backend
      (or task-list
          task-list
          (setf task-list (list-tasks backend))))))
#+END_SRC

**** list-tasks                                                 :function:
#+BEGIN_SRC lisp
(defun list-tasks (backend)
  "Return all tasks recorded in BACKEND as a list of strings."
  (loop for plist in (to-list backend)
    collect (plist-get plist :name) into names
    finally return
    (sort (remove-duplicates names :test #'equal)
          #'string-lessp)))
#+END_SRC

**** run-assertions                                     :generic:function:
#+BEGIN_SRC lisp
(defgeneric backend-run-assertions (backend)
  "Check common preconditions for any operations on BACKEND.
Signal errors for any unmet preconditions.")
#+END_SRC

**** view-backend                                       :generic:function:
#+BEGIN_SRC lisp
(defgeneric view-backend (backend)
  "Open BACKEND for interactive viewing.")
#+END_SRC

**** edit-backend                                       :generic:function:
#+BEGIN_SRC lisp
(defgeneric edit-backend (backend)
  "Open BACKEND for interactive editing.")
#+END_SRC

**** backend-empty-p                                    :generic:function:
#+BEGIN_SRC lisp
(defgeneric backend-empty-p (backend)
  "Return non-nil if BACKEND contains no records, else nil.")
#+END_SRC

**** backend-modified-p                                 :generic:function:
#+BEGIN_SRC lisp
(defgeneric backend-modified-p (backend)
  "Return non-nil if BACKEND is being modified.
For instance, a file-based backend could be undergoing editing by
a user.")
#+END_SRC

**** file operations
***** create-file                                      :generic:function:
[[file:../tests/tests.org::#tests-backend-create-file][tests]]

#+BEGIN_SRC lisp
(defgeneric create-file (backend &optional file)
  "Create file associated with BACKEND.
Use FILE as a path, if provided.
Return path of new file if successfully created, and nil if it already exists.")
#+END_SRC

***** latest-date-records                              :generic:function:
#+BEGIN_SRC lisp
(defgeneric latest-date-records (backend)
  "Return intervals of latest day in BACKEND as a tagged list (\"DATE\" PLIST*).
Return nil if BACKEND contains no records.")
#+END_SRC

***** insert                                           :generic:function:
#+BEGIN_SRC lisp
(defgeneric insert (backend plist)
  "Insert PLIST as new record in BACKEND.
Return non-nil if record is inserted successfully.

PLIST may be an interval which crosses days.")

(defmethod insert :before ((_backend t) plist &key &allow-other-keys)
  (unless (typep plist 'plist)
    (error "Not a valid plist: %S" plist)))
#+END_SRC

***** remove-last                                      :generic:function:
#+BEGIN_SRC lisp
(defgeneric remove-last (backend)
  "Remove last record from BACKEND.
Return non-nil if record is successfully removed.
Signal an error if there is no record to remove.")
#+END_SRC

***** latest-record                                    :generic:function:
#+BEGIN_SRC lisp
(defgeneric latest-record (backend)
  "Return the latest record from BACKEND as a plist, or nil if BACKEND contains no records.
Return value may be active, i.e. it may or may not have a `:stop'
key-value.

If the latest record starts on one day and ends on another, the
entire (unsplit) record must be returned.")
#+END_SRC

***** task-records-for-date                            :generic:function:
#+BEGIN_SRC lisp
(defgeneric task-records-for-date (backend task date-ts)
  "From BACKEND, return records for TASK on DATE-TS as a list of plists.
DATE-TS must be a `ts.el' struct.

Return nil if BACKEND contains no records.")

(defmethod task-records-for-date :before ((_backend t) task date-ts &key &allow-other-keys)
  (unless (typep task 'string)
    (error "task %S is not a string" task))
  (unless (typep date-ts 'ts)
    (error "date-ts %S is not a `ts' struct" date-ts)))
#+END_SRC

***** replace-last                                     :generic:function:
#+BEGIN_SRC lisp
(defgeneric replace-last (backend plist)
  "Replace last record in BACKEND with PLIST.
Return non-nil if successful.")

(defmethod replace-last :before ((_backend t) plist &key &allow-other-keys)
  (unless (typep plist 'plist)
    (error "Not a valid plist: %S" plist)))
#+END_SRC

***** to-file                                          :generic:function:
#+BEGIN_SRC lisp
(defgeneric to-file (input-hash-table output-backend output-file)
  "Save data from INPUT-HASH-TABLE to OUTPUT-FILE, in OUTPUT-BACKEND format.
Any existing data in OUTPUT-FILE is overwritten.")
#+END_SRC

***** on-add                                           :generic:function:
#+BEGIN_SRC lisp
(defgeneric on-add (backend)
  "Function called when data is added to BACKEND.
This may happen within Chronometrist (e.g. via
`insert') or outside it (e.g. a user editing the
backend file).

NEW-DATA is the data that was added.")
#+END_SRC

***** on-modify                                        :generic:function:
#+BEGIN_SRC lisp
(defgeneric on-modify (backend)
  "Function called when data in BACKEND is modified (rather than added or removed).
This may happen within Chronometrist (e.g. via
`replace-last') or outside it (e.g. a user editing
the backend file).

OLD-DATA and NEW-DATA is the data before and after the changes,
respectively.")
#+END_SRC

***** on-remove                                        :generic:function:
#+BEGIN_SRC lisp
(defgeneric on-remove (backend)
  "Function called when data is removed from BACKEND.
This may happen within Chronometrist (e.g. via
`remove-last') or outside it (e.g. a user editing
the backend file).

OLD-DATA is the data that was modified.")
#+END_SRC

***** on-change                                        :generic:function:
#+BEGIN_SRC lisp
(defgeneric on-change (backend)
  "Function to be run when BACKEND changes on disk.
This may happen within Chronometrist (e.g. via
`insert') or outside it (e.g. a user editing the
backend file).")
#+END_SRC

***** verify                                           :generic:function:
#+BEGIN_SRC lisp
(defgeneric verify (backend)
  "Check BACKEND for errors in data.
Return nil if no errors are found.

If an error is found, return (LINE-NUMBER . COLUMN-NUMBER) for file-based backends.")
#+END_SRC

***** on-file-path-change                              :generic:function:
#+BEGIN_SRC lisp
(defgeneric on-file-path-change (backend old-path new-path)
  "Function run when the value of `file' is changed.
OLD-PATH and NEW-PATH are the old and new values of
`file', respectively.")
#+END_SRC

**** memory operations
***** reset-backend                                    :generic:function:
#+BEGIN_SRC lisp
(defgeneric reset-backend (backend)
  "Reset data structures for BACKEND.")
#+END_SRC

***** to-hash-table                                    :generic:function:
#+BEGIN_SRC lisp
(defgeneric to-hash-table (backend)
  "Return data in BACKEND as a hash table in chronological order.
Hash table keys are ISO-8601 date strings. Hash table values are
lists of records, represented by plists. Both hash table keys and
hash table values must be in chronological order.")
#+END_SRC

***** to-list                                          :generic:function:
#+BEGIN_SRC lisp
(defgeneric to-list (backend)
  "Return all records in BACKEND as a list of plists.")
#+END_SRC

***** memory-layer-empty-p                             :generic:function:
#+BEGIN_SRC lisp
(defgeneric memory-layer-empty-p (backend)
  "Return non-nil if memory layer of BACKEND contains no records, else nil.")
#+END_SRC

**** extended protocol (unimplemented)
These can be implemented in terms of the minimal protocol above.

***** active-days                                      :generic:function:
#+BEGIN_SRC lisp
(defgeneric active-days (backend task &key start end)
  "From BACKEND, return number of days on which TASK had recorded time.")
#+END_SRC

***** count-records                                    :generic:function:
#+BEGIN_SRC lisp
(defgeneric count-records (backend)
  "Return number of records in BACKEND.")
#+END_SRC

*** Common definitions for s-expression backends
:PROPERTIES:
:CUSTOM_ID: program-backend-sexp-common
:END:
**** file-backend-mixin                                            :mixin:
:PROPERTIES:
:CUSTOM_ID: file-backend-mixin
:END:

#+BEGIN_SRC lisp
(defclass file-backend-mixin ()
  ((path :initform nil
    :initarg :path
    :accessor backend-path
    :custom 'string
    :documentation
    "Path to backend file, without extension.")
   (extension :initform nil
    :initarg :extension
    :accessor backend-ext
    :custom 'string
    :documentation
    "Extension of backend file.")
   (file :initarg :file
         :initform nil
         :accessor backend-file
         :custom 'string
         :documentation "Full path to backend file, with extension.")
   (hash-table :initform (make-hash-table-1)
               :initarg :hash-table
               :accessor backend-hash-table)
   (file-watch :initform nil
               :initarg :file-watch
               :accessor backend-file-watch
               :documentation "Filesystem watch object, as returned by `file-notify-add-watch'."))
  :documentation "Mixin for backends storing data in a single file.")
#+END_SRC

**** setup-file-watch                                             :writer:
#+BEGIN_SRC lisp
(defun setup-file-watch (&optional (callback #'refresh-file))
  "Arrange for CALLBACK to be called when the backend file changes."
  (with-slots (file file-watch) (active-backend)
    (unless file-watch
      (setq file-watch
            (file-notify-add-watch file '(change) callback)))))
#+END_SRC

**** edit-backend                                                 :method:
#+BEGIN_SRC lisp
(defmethod edit-backend ((backend file-backend-mixin))
  (find-file-other-window (backend-file backend))
  (goto-char (point-max)))
#+END_SRC

**** initialize-instance                                          :method:
#+BEGIN_SRC lisp
(defmethod initialize-instance :after ((backend file-backend-mixin)
                                          &rest _initargs)
  "Initialize FILE based on PATH and EXTENSION."
  (with-slots (path extension file) backend
    (when (and path extension (not file))
      (setf file (concat path "." extension)))))
#+END_SRC

**** reset-backend                                         :writer:method:
#+BEGIN_SRC lisp
(defmethod reset-backend ((backend file-backend-mixin))
  (with-slots (hash-table file-watch
               rest-start rest-end rest-hash
               file-length last-hash) backend
    (reset-task-list backend)
    (when file-watch
      (file-notify-rm-watch file-watch))
    (setf hash-table  (to-hash-table backend)
          file-watch  nil
          rest-start  nil
          rest-end    nil
          rest-hash   nil
          file-length nil
          last-hash   nil)
    (setup-file-watch)))
#+END_SRC

**** backend-empty-p                                       :reader:method:
#+BEGIN_SRC lisp
(defmethod backend-empty-p ((backend file-backend-mixin))
  (with-slots (file) backend
      (or (not (file-exists-p file))
          (file-empty-p file))))
#+END_SRC

**** memory-layer-empty-p                                  :reader:method:
#+BEGIN_SRC lisp
(defmethod memory-layer-empty-p ((backend file-backend-mixin))
  (with-slots (hash-table) backend
    (zerop (hash-table-count hash-table))))
#+END_SRC

**** backend-modified-p                                    :reader:method:
#+BEGIN_SRC lisp
(defmethod backend-modified-p ((backend file-backend-mixin))
  (with-slots (file) backend
    (buffer-modified-p
     (get-buffer-create
      (find-file-noselect file)))))
#+END_SRC

**** on-file-path-change                                   :writer:method:
#+BEGIN_SRC lisp
(defmethod on-file-path-change ((backend file-backend-mixin) _old-path new-path)
  "Update path and file slots of BACKEND to use NEW-PATH when `file' is changed."
  (with-slots (path extension file) backend
    (setf path new-path
          file (concat path "." extension))))
#+END_SRC

**** elisp-sexp-backend                                            :class:
#+BEGIN_SRC lisp
(defclass elisp-sexp-backend (backend file-backend-mixin)
  ((rest-start :initarg :rest-start
               :initform nil
               :accessor backend-rest-start
               :documentation "Integer denoting start of first s-expression in file.")
   (rest-end :initarg :rest-end
             :initform nil
             :accessor backend-rest-end
             :documentation "Integer denoting end of second-last s-expression in file.")
   (rest-hash :initarg :rest-hash
              :initform nil
              :accessor backend-rest-hash
              :documentation "Hash of content between rest-start and rest-end.")
   (file-length :initarg :file-length
                :initform nil
                :accessor backend-file-length
                :documentation "Integer denoting length of file, as returned by `(point-max)'.")
   (last-hash :initarg :last-hash
              :initform nil
              :accessor backend-last-hash
              :documentation "Hash of content between rest-end and file-length."))
  :documentation "Base class for any text file backend which stores s-expressions readable by Emacs Lisp.")
#+END_SRC

**** sexp-mode                                                :major:mode:
#+BEGIN_SRC lisp
(define-derived-mode sexp-mode
  ;; fundamental-mode
  emacs-lisp-mode
  "sexp")
#+END_SRC

**** create-file                                           :writer:method:
#+BEGIN_SRC lisp
(defmethod create-file ((backend elisp-sexp-backend) &optional file)
  (let ((file (or file (backend-file backend))))
    (unless (file-exists-p file)
      (with-current-buffer (find-file-noselect file)
        (erase-buffer)
        (goto-char (point-min))
        (insert ";;; -*- mode: sexp; -*-\n\n")
        (write-file file))
      file)))
#+END_SRC

**** in-file                                                       :macro:
#+BEGIN_SRC lisp
(defmacro sexp-in-file (file &rest body)
  "Run BODY in a buffer visiting FILE, restoring point afterwards."
  (declare (indent defun) (debug t))
  `(with-current-buffer (find-file-noselect ,file)
     (save-excursion ,@body)))
#+END_SRC

**** pre-read-check                                            :procedure:
#+BEGIN_SRC lisp
(defun sexp-pre-read-check (buffer)
  "Return non-nil if there is an s-expression before point in BUFFER.
Move point to the start of this s-expression."
  (with-current-buffer buffer
    (and (not (bobp))
         (backward-list)
         (or (not (bobp))
             (not (looking-at-p "^[[:blank:]]*;"))))))
#+END_SRC

**** loop-sexp-file                                                :macro:
#+BEGIN_SRC lisp
(defmacro loop-sexp-file (_for sexp _in file &rest loop-clauses)
  "`loop' LOOP-CLAUSES over s-expressions in FILE.
SEXP is bound to each s-expressions in reverse order (last
expression first)."
  (declare (indent defun) (debug 'loop))
  `(sexp-in-file ,file
     (goto-char (point-max))
     (loop with ,sexp
       while (and (sexp-pre-read-check (current-buffer))
                  (setq ,sexp (ignore-errors (read (current-buffer))))
                  (backward-list))
       ,@loop-clauses)))
#+END_SRC

**** backend-empty-p                                       :reader:method:
#+BEGIN_SRC lisp
(defmethod backend-empty-p ((backend elisp-sexp-backend))
  (sexp-in-file (backend-file backend)
    (goto-char (point-min))
    (not (ignore-errors
           (read (current-buffer))))))
#+END_SRC

**** indices and hashes
#+BEGIN_SRC lisp
(defun rest-start (file)
  (sexp-in-file file
    (goto-char (point-min))
    (forward-list)
    (backward-list)
    (point)))

(defun rest-end (file)
  (sexp-in-file file
    (goto-char (point-max))
    (backward-list 2)
    (forward-list)
    (point)))
#+END_SRC

**** file-hash                                                    :reader:
#+BEGIN_SRC lisp
(defun file-hash (start end &optional (file (backend-file (active-backend))))
  "Calculate hash of `file' between START and END."
  (sexp-in-file file
    (secure-hash 'sha1
                 (buffer-substring-no-properties start end))))
#+END_SRC

***** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest file-hash ()
  (-let* ((file test-file)
          ((last-start last-end)
           (file-hash :before-last nil nil file))
          ((rest-start rest-end rest-hash)
           (file-hash nil :before-last t file)))
    (message "chronometrist - file-hash test - file path is %s" file)
    (should (= 1 rest-start))
    (should (= 1254 rest-end))
    (should (= 1256 last-start))
    (should (= 1426 last-end))))
#+END_SRC

**** file-change-type                                             :reader:
+ rest-start - start of first sexp
+ rest-end - end of second last sexp
+ file-length - end of file

+ rest-hash - hash of content between rest-start and rest-end
+ last-hash - hash of content between rest-end and file-length

+ ht-last-sexp - last sexp in memory
+ file-sexp-after-rest - sexp after rest-end
+ file-last-sexp - last sexp in file

| situation    | rest-hash | last-hash | file-sexp-after-rest | file-last-sexp                          | file-length                  |
|--------------+-----------+-----------+----------------------+-----------------------------------------+------------------------------|
| no change    | same      | same      | same as ht-last-sexp | same as ht-last-sexp and file-last-sexp | same                         |
| append       | same      | same      | -                    | (new s-expression)                      | always greater               |
| modify       | same      | changed   | changed              | changed                                 | may be smaller               |
| remove       | same      | changed   | nil                  | same as second last sexp                | always smaller               |
| other change | changed   | -         |                      | -                                       | may be smaller than rest-end |

We avoid comparing s-expressions in the file with the contents of the hash table, since the last s-expression might be represented differently in the hash tables of different elisp-sexp backends. Additionally, in =:modify= as well as =nil= situations, there is no s-expression after old-file-length.

#+BEGIN_SRC lisp
(defun file-change-type (backend)
  "Determine the type of change made to BACKEND's file.
    Return
    :append  if a new s-expression was added to the end,
    :modify  if the last s-expression was modified,
    :remove  if the last s-expression was removed,
        nil  if the contents didn't change, and
          t  for any other change."
  (with-slots
      (file file-watch
            ;; The slots contain the old state of the file.
            hash-table
            rest-start rest-end rest-hash
            file-length last-hash) backend
    (let* ((new-length    (file-length file))
           (new-rest-hash (when (and (>= new-length rest-start)
                                     (>= new-length rest-end))
                            (file-hash rest-start rest-end file)))
           (new-last-hash (when (and (>= new-length rest-end)
                                     (>= new-length file-length))
                            (file-hash rest-end file-length file))))
      ;; (debug-message "File indices - old rest-start: %s rest-end: %s file-length: %s new-length: %s"
      ;;          rest-start rest-end file-length new-length)
      (cond ((and (= file-length new-length)
                  (equal rest-hash new-rest-hash)
                  (equal last-hash new-last-hash))
             nil)
            ((or (< new-length rest-end) ;; File has shrunk so much that we cannot compare rest-hash.
                 (not (equal rest-hash new-rest-hash)))
             t)
            ;; From here on, it is implicit that the change has happened at the end of the file.
            ((and (< file-length new-length) ;; File has grown.
                  (equal last-hash new-last-hash))
             :append)
            ((and (< new-length file-length) ;; File has shrunk.
                  (not (sexp-in-file file
                         (goto-char rest-end)
                         (ignore-errors
                           (read (current-buffer)))))) ;; There is no sexp after rest-end.
             :remove)
            (t :modify)))))
#+END_SRC

***** tests
#+BEGIN_SRC lisp :tangle tests.el :load test
(ert-deftest file-change-type ()
  (with-slots (file hash-table file-state) plist-test-backend
    (let* ((b plist-test-backend)
           (test-contents (with-current-buffer (find-file-noselect file)
                            (buffer-substring (point-min) (point-max)))))
      (reset-backend b)
      (setf file-state
            (list :last (file-hash :before-last nil)
                  :rest (file-hash nil :before-last t)))
      (unwind-protect
          (progn
            (should
             (eq nil (file-change-type file-state)))
            (should
             (eq :append
                 (progn
                   (insert plist-test-backend
                             '(:name "Append Test"
                                     :start "2021-02-01T13:06:46+0530"
                                     :stop "2021-02-01T13:06:49+0530"))
                   (tests--change-type-and-update file-state file))))
            (should
             (eq :modify
                 (progn
                   (replace-last plist-test-backend
                                   '(:name "Modify Test"
                                           :tags (some tags)
                                           :start "2021-02-01T13:06:46+0530"
                                           :stop "2021-02-01T13:06:49+0530"))
                   (tests--change-type-and-update file-state file))))
            (should
             (eq :remove
                 (progn
                   (sexp-in-file file
                     (goto-char (point-max))
                     (backward-list 1)
                     (sexp-delete-list 1)
                     (save-buffer))
                   (tests--change-type-and-update file-state file))))
            (should
             (eq t
                 (progn
                   (sexp-in-file file
                     (goto-char (point-min))
                     (plist-pp '(:name "Other Change Test"
                                         :start "2021-02-02T17:39:40+0530"
                                         :stop "2021-02-02T17:39:44+0530")
                                 (current-buffer))
                     (save-buffer))
                   (tests--change-type-and-update file-state file)))))
        (with-current-buffer
            (find-file-noselect file)
          (delete-region (point-min) (point-max))
          (insert test-contents)
          (save-buffer))
        (reset-backend b)))))
#+END_SRC

**** reset-task-list                                              :writer:
#+BEGIN_SRC lisp
(defun reset-task-list (backend)
  "Regenerate BACKEND's task list from its data.
Only takes effect if `task-list' is nil (i.e. the
user has not defined their own task list)."
  (unless task-list
    (setf (backend-task-list backend) (list-tasks backend))))
#+END_SRC

**** add-to-task-list                                             :writer:
#+BEGIN_SRC lisp
(defun add-to-task-list (task backend)
  "Add TASK to BACKEND's task list, if it is not already present.
Only takes effect if `task-list' is nil (i.e. the
user has not defined their own task list)."
  (with-slots (task-list) backend
    (unless (and (not task-list)
                 (member task task-list :test #'equal))
      (setf task-list
            (sort (cons task task-list)
                  #'string-lessp)))))
#+END_SRC

**** remove-from-task-list                                        :writer:
#+BEGIN_SRC lisp
(defun remove-from-task-list (task backend)
  "Remove TASK from BACKEND's task list if necessary.
TASK is removed if it does not occur in BACKEND's hash table, or
if it only occurs in the newest plist of the same.

Only takes effect if `task-list' is nil (i.e. the
user has not defined their own task list).

Return new value of BACKEND's task list, or nil if
unchanged."
  (with-slots (hash-table task-list) backend
    (unless task-list
      (let (;; number of plists in hash table
            (ht-plist-count (loop with count = 0
                              for intervals being the hash-values of hash-table
                              do (loop for _interval in intervals
                                   do (incf count))
                              finally return count))
            ;; index of first occurrence of TASK in hash table, or nil if not found
            (ht-task-first-result (loop with count = 0
                                    for intervals being the hash-values of hash-table
                                    when (loop for interval in intervals
                                           do (incf count)
                                           when (equal task (plist-get interval :name))
                                           return t)
                                    return count)))
        (when (or (not ht-task-first-result)
                  (= ht-task-first-result ht-plist-count))
          ;; The only interval for TASK is the last expression
          (setf task-list (remove task task-list)))))))
#+END_SRC

**** on-change                                             :writer:method:
#+BEGIN_SRC lisp
(defmethod on-change ((backend elisp-sexp-backend) fs-event)
  "Function called when BACKEND file is changed.
This may happen within Chronometrist (through the backend
protocol) or outside it (e.g. a user editing the backend file).

FS-EVENT is the event passed by the `filenotify' library (see `file-notify-add-watch')."
  (with-slots (file hash-table file-watch
                    rest-start rest-end rest-hash
                    file-length last-hash) backend
    (-let* (((_ action _ _) fs-event)
            (file-state-bound-p (and rest-start rest-end rest-hash
                                     file-length last-hash))
            (change      (when file-state-bound-p
                           (file-change-type backend)))
            (reset-watch-p (or (eq action 'deleted)
                               (eq action 'renamed))))
      (debug-message "[Method] on-change: file change type %s" change)
      ;; If only the last plist was changed, update hash table and
      ;; task list, otherwise clear and repopulate hash table.
      (cond ((or reset-watch-p
                 (not file-state-bound-p) ;; why?
                 (eq change t))
             (reset-backend backend))
            (file-state-bound-p
             (pcase change
               ;; A new s-expression was added at the end of the file
               (:append (on-add backend))
               ;; The last s-expression in the file was changed
               (:modify (on-modify backend))
               ;; The last s-expression in the file was removed
               (:remove (on-remove backend))
               ((pred null) nil))))
      (setf rest-start  (rest-start file)
            rest-end    (rest-end file)
            file-length (file-length file)
            last-hash   (file-hash rest-end file-length file)
            rest-hash   (file-hash rest-start rest-end file)))))
#+END_SRC

*** plist backend
In this format, user data is stored as Elisp plists in a plain text file. A basic plist in this file looks like this -

#+BEGIN_SRC lisp :tangle no :load no
(:name "Task Name"
 [:keyword <value>]*
 :start "<ISO-8601 date-time>"
 :stop "<ISO-8601 date-time>")
#+END_SRC

=:name= and =:start= are essential. =:stop= may be missing if the task is currently active.

The reasons I like this format are -
1. Users can browse and edit the data using the Emacs setup they are accustomed to. We get things like swiper, undo-tree, and a host of other features for free.
2. It is trivial to parse using the =read= built-in to Emacs.
   * =loop-file= is provided as an additional convenience, to iterate through each expression in the file.
3. It is easy to diff and version control.

**** tests
:PROPERTIES:
:header-args:   :tangle tests.el
:END:

#+BEGIN_SRC lisp :tangle tests.el :load test
(defvar plist-test-backend
  (make-instance 'plist-backend
                 :file (make-temp-file "plist-test-" nil ".sexp")))

(let ((file (backend-file plist-test-backend)))
  (with-current-buffer (find-file-noselect file)
    (mapcar
     (lambda (plist)
       ;; to use this, we'd have to move `plist-pp' before this
       ;; definition, and I'm perfectly content with where it is
       ;; right now
       (plist-pp plist (current-buffer))
       (insert "\n\n")
       ;; (print plist) (princ "\n")
       )
     '((:name "Programming"
              :start "2018-01-01T00:00:00+0530"
              :stop  "2018-01-01T01:00:00+0530")
       (:name "Swimming"
              :start "2018-01-01T02:00:00+0530"
              :stop  "2018-01-01T03:00:00+0530")
       (:name "Cooking"
              :start "2018-01-01T04:00:00+0530"
              :stop  "2018-01-01T05:00:00+0530")
       (:name "Guitar"
              :start "2018-01-01T06:00:00+0530"
              :stop  "2018-01-01T07:00:00+0530")
       (:name "Cycling"
              :start "2018-01-01T08:00:00+0530"
              :stop  "2018-01-01T09:00:00+0530")
       (:name "Programming"
              :start "2018-01-02T23:00:00+0530"
              :stop  "2018-01-03T01:00:00+0530")
       (:name "Cooking"
              :start "2018-01-03T23:00:00+0530"
              :stop  "2018-01-04T01:00:00+0530")
       (:name "Programming"
              :tags      (bug-hunting)
              :project   "Chronometrist"
              :component "goals"
              :start     "2020-05-09T20:03:25+0530"
              :stop      "2020-05-09T20:05:55+0530")
       (:name "Arrangement/new edition"
              :tags     (new edition)
              :song     "Songs of Travel"
              :composer "Vaughan Williams, Ralph"
              :start    "2020-05-10T00:04:14+0530"
              :stop     "2020-05-10T00:25:48+0530")
       (:name "Guitar"
              :tags  (classical warm-up)
              :start "2020-05-10T15:41:14+0530"
              :stop  "2020-05-10T15:55:42+0530")
       (:name "Guitar"
              :tags  (classical solo)
              :start "2020-05-10T16:00:00+0530"
              :stop  "2020-05-10T16:30:00+0530")
       (:name "Programming"
              :tags  (reading)
              :book  "Smalltalk-80: The Language and Its Implementation"
              :start "2020-05-10T16:33:17+0530"
              :stop  "2020-05-10T17:10:48+0530")))))
#+END_SRC

1. [X] finding the test input file
   * =buffer-file-name= returns nil when Emacs is run in batch mode; I've tried using =(concat (or (ignore-errors (file-name-directory (buffer-file-name))) default-directory) "test.sexp")=, but that resulted in ="~/.emacs.d/test.sexp"= being used instead, for some reason.
   * maybe we can store the test file contents in a string instead, and create a temporary test file using =make-temp-file=?

Boilerplate for updating state between file operations in tests.
#+BEGIN_SRC lisp :load test
(defmacro tests--change-type-and-update (state file)
  `(prog1 (file-change-type ,state)
     (setq ,state
           (list :last (file-hash :before-last nil nil ,file)
                 :rest (file-hash nil :before-last t ,file)))))
#+END_SRC

**** backend                                                       :class:
#+BEGIN_SRC lisp
(defclass plist-backend (elisp-sexp-backend)
  ((extension :initform "plist"
              :accessor backend-ext
              :custom 'string)))

(register-backend
 :plist "Store records as plists."
 (make-instance 'plist-backend :path file))
#+END_SRC

**** pretty-print-function                               :custom:variable:
#+BEGIN_SRC lisp
(defcustom sexp-pretty-print-function #'plist-pp
  "Function used to pretty print plists in `file'.
Like `pp', it must accept an OBJECT and optionally a
STREAM (which is the value of `current-buffer')."
  :type 'function
  :group 'chronometrist)
#+END_SRC

**** latest-date-records                                   :reader:method:
In this backend, it's easier to implement this in terms of [[#program-backend-plist-latest-record][=latest-record=]] than the other way round.

#+BEGIN_SRC lisp
(defmethod latest-date-records ((backend plist-backend))
  (backend-run-assertions backend)
  (with-slots (hash-table) backend
    (when-let*
        ((latest-date (ht-last-date hash-table))
         (records     (gethash latest-date hash-table)))
      (cons latest-date records))))
#+END_SRC

**** to-hash-table                                         :writer:method:
#+BEGIN_SRC lisp
(defmethod to-hash-table ((backend plist-backend))
  (sexp-in-file (backend-file backend)
    (goto-char (point-min))
    (let ((table (make-hash-table-1))
          expr pending-expr)
      (while (or pending-expr
                 (setq expr (ignore-errors (read (current-buffer)))))
        ;; find and split midnight-spanning events during deserialization itself
        (let* ((split-expr (split-plist expr))
               (new-value  (cond (pending-expr
                                  (prog1 pending-expr
                                    (setq pending-expr nil)))
                                 (split-expr
                                  (setq pending-expr (second split-expr))
                                  (first split-expr))
                                 (t expr)))
               (new-value-date (--> (plist-get new-value :start)
                                    (substring it 0 10)))
               (existing-value (gethash new-value-date table)))
          (puthash new-value-date
                   (if existing-value
                       (append existing-value
                               (list new-value))
                     (list new-value))
                   table)))
      table)))
#+END_SRC

**** insert                                                :writer:method:
#+BEGIN_SRC lisp
(defmethod insert ((backend plist-backend) plist &key (save t))
  (backend-run-assertions backend)
  (debug-message "[Method] insert: %s" plist)
  (sexp-in-file (backend-file backend)
    (goto-char (point-max))
    ;; If we're adding the first s-exp in the file, don't add a
    ;; newline before it
    (unless (bobp) (insert "\n"))
    (unless (bolp) (insert "\n"))
    (funcall sexp-pretty-print-function plist (current-buffer))
    (when save (save-buffer))
    t))
#+END_SRC

**** remove-last                                           :writer:method:
#+BEGIN_SRC lisp
(defmethod remove-last ((backend plist-backend))
  (debug-message "[Method] remove-last")
  (backend-run-assertions backend)
  (when (backend-empty-p backend)
  (error "remove-last has nothing to remove in %s"
         (eieio-object-class-name backend)))
  (sexp-in-file (backend-file backend)
    (goto-char (point-max))
    ;; this condition should never really occur, since we insert a
    ;; file local variable prop line when the file is created...
    (unless (and (bobp) (bolp)) (insert "\n"))
    (backward-list 1)
    (sexp-delete-list)))
#+END_SRC

**** reindent-buffer                                             :command:
#+BEGIN_SRC lisp
(defun sexp-reindent-buffer ()
  "Reindent the current buffer.
This is meant to be run in `file' when using an s-expression backend."
  (interactive)
  (debug-message "[Command] reindent-buffer")
  (let (expr)
    (goto-char (point-min))
    (while (setq expr (ignore-errors (read (current-buffer))))
      (backward-list)
      (sexp-delete-list)
      (when (looking-at "\n*")
        (delete-region (match-beginning 0) (match-end 0)))
      (funcall sexp-pretty-print-function expr (current-buffer))
      (insert "\n")
      (unless (eobp) (insert "\n")))))
#+END_SRC

**** to-file                                               :writer:method:
#+BEGIN_SRC lisp
(defmethod to-file (hash-table (backend plist-backend) file)
  (delete-file file)
  (create-file backend file)
  (reset-backend backend)             ; possibly to ensure BACKEND is up to date
  (sexp-in-file file
    (goto-char (point-max))
    (loop
      for date in (sort (hash-table-keys hash-table) #'string-lessp) do
      (loop for plist in (gethash date hash-table) do
        (insert (plist-pp plist) "\n\n"))
      finally do (save-buffer))))
#+END_SRC

**** to-list                                               :reader:method:
#+BEGIN_SRC lisp
(defmethod to-list ((backend plist-backend))
  (backend-run-assertions backend)
  (loop-sexp-file for expr in (backend-file backend) collect expr))
#+END_SRC

**** on-add                                                :writer:method:
#+BEGIN_SRC lisp
(defmethod on-add ((backend plist-backend))
  "Function run when a new plist is added at the end of a
`plist-backend' file."
  (with-slots (hash-table) backend
    (-let [(new-plist &as &plist :name new-task) (latest-record backend)]
      (setf hash-table (ht-update new-plist hash-table))
      (add-to-task-list new-task backend))))
#+END_SRC

**** on-modify                                             :writer:method:
#+BEGIN_SRC lisp
(defmethod on-modify ((backend plist-backend))
  "Function run when the newest plist in a
`plist-backend' file is modified."
  (with-slots (hash-table) backend
    (-let (((new-plist &as &plist :name new-task) (latest-record backend))
           ((&plist :name old-task) (ht-last backend)))
      (setf hash-table (ht-update new-plist hash-table t))
      (remove-from-task-list old-task backend)
      (add-to-task-list new-task backend))))
#+END_SRC

**** on-remove                                             :writer:method:
#+BEGIN_SRC lisp
(defmethod on-remove ((backend plist-backend))
  "Function run when the newest plist in a
`plist-backend' file is deleted."
  (with-slots (hash-table) backend
    (-let (((&plist :name old-task) (ht-last))
           (date  (ht-last-date hash-table)))
      ;; `remove-from-task-list' checks the hash table to determine
      ;; if `task-list' is to be updated. Thus, the hash table must
      ;; not be updated until the task list is.
      (remove-from-task-list old-task backend)
      (--> (gethash date hash-table)
           (-drop-last 1 it)
           (setf (gethash date hash-table) it)))))
#+END_SRC

**** extended protocol
***** latest-record                                       :reader:method:
:PROPERTIES:
:CUSTOM_ID: program-backend-plist-latest-record
:END:
#+BEGIN_SRC lisp
(defmethod latest-record ((backend plist-backend))
  (backend-run-assertions backend)
  (sexp-in-file (backend-file backend)
    (goto-char (point-max))
    (backward-list)
    (ignore-errors (read (current-buffer)))))
#+END_SRC

***** task-records-for-date                               :reader:method:
#+BEGIN_SRC lisp
(defmethod task-records-for-date ((backend plist-backend) task date-ts)
  (backend-run-assertions backend)
  (let* ((date    (date-iso date-ts))
         (records (gethash date (backend-hash-table backend))))
    (loop for record in records
      when (equal task (plist-get record :name))
      collect record)))
#+END_SRC

***** replace-last                                        :writer:method:
#+BEGIN_SRC lisp
(defmethod replace-last ((backend plist-backend) plist)
  (debug-message "[Method] replace-last with %s" plist)
  (sexp-in-file (backend-file backend)
    (goto-char (remove-last backend))
    (funcall sexp-pretty-print-function plist (current-buffer))
    (save-buffer)
    t))
#+END_SRC

***** count-records                                       :reader:method:
#+BEGIN_SRC lisp
(defmethod count-records ((backend plist-backend))
  (sexp-in-file (backend-file backend)
    (goto-char (point-min))
    (loop with count = 0
      while (ignore-errors (read (current-buffer)))
      do (incf count)
      finally return count)))
#+END_SRC

*** plist group backend
:PROPERTIES:
:CUSTOM_ID: program-backend-plist-group
:END:
This is largely like the plist backend, but plists are grouped by date by wrapping them in a tagged list -

#+BEGIN_SRC lisp :tangle no :load no
("<ISO-8601 date>"
 (:name "Task Name"
  [:keyword <value>]*
  :start "<ISO-8601 time>"
  :stop "<ISO-8601 time>")
 ...)
#+END_SRC

This makes it easy and computationally cheap to perform our most common query - getting the plists on a given day. [[#explanation-midnight-spanning-intervals][Midnight-spanning intervals]] are split in the file itself. The downside is that the user, if editing it by hand, must take care to split the intervals.

Note that migrating from the plist backend to the plist group backend is inherently likely to result in more plists compared to the source, as each midnight-spanning plist is split into two.

Concerns specific to the plist group backend -
1. the last plist may be split across two days. In such a situation -
   * changing key-values for the last plist would only apply to the most recent one
   * +deleting the last plist via =remove-last= would only delete the recent part of the split plist+ fixed
   * resetting is unaffected, since that only applies to the last interval, whether or not it's a split plist
   * restarting is unaffected, since that only applies to the active interval, and split intervals are always inactive ones

**** backend                                                       :class:
#+BEGIN_SRC lisp
(defclass plist-group-backend (elisp-sexp-backend)
  ((extension :initform "plg"
              :accessor backend-ext
              :custom 'string)))

(register-backend
 :plist-group "Store records as plists grouped by date."
 (make-instance 'plist-group-backend
                :path file))
#+END_SRC

**** backward-read-sexp                                           :reader:
#+BEGIN_SRC lisp
(defun backward-read-sexp (buffer)
  (backward-list)
  (save-excursion (read buffer)))
#+END_SRC

**** run-assertions                                        :reader:method:
#+BEGIN_SRC lisp
(defmethod backend-run-assertions ((backend file-backend-mixin))
  (with-slots (file) backend
    (unless (file-exists-p file)
      (error "Backend file %S does not exist" file))))
#+END_SRC

**** latest-date-records                                   :reader:method:
#+BEGIN_SRC lisp
(defmethod latest-date-records ((backend plist-group-backend))
  (backend-run-assertions backend)
  (sexp-in-file (backend-file backend)
    (goto-char (point-max))
    (ignore-errors
      (backward-read-sexp (current-buffer)))))
#+END_SRC

**** HACK insert                                           :writer:method:
<<hack-note-plist-group-insert>>
We just want to insert a plist, but as a hack to avoid updating the pretty-printer to handle indentation of plists being inserted into an outer list, we =append= the plist to a plist group and insert/replace the plist group instead.

Situations -
1. new inactive day-crossing record
   1. first record - split, insert into two new groups
   2. not first record - split, insert into existing group + new group
2. new active record, or new non-day-crossing inactive record
   1. first record - insert into new plist group
   2. not first record
      1. latest recorded date = today - insert into existing group
      2. insert into new group

#+BEGIN_SRC lisp
(defmethod insert ((backend plist-group-backend) plist &key (save t))
  (check-type plist plist)
  (debug-message "[Method] insert: %S" plist)
  (backend-run-assertions backend)
  (if (not plist)
      (error "%s" "`insert' was called with an empty plist")
    (sexp-in-file (backend-file backend)
      (-let* (((plist-1 plist-2)   (split-plist plist))
              ;; Determine if we need to insert a new plist group
              (latest-plist-group  (latest-date-records backend))
              (backend-latest-date (first latest-plist-group))
              (date-today          (date-iso))
              (insert-new-group    (not (equal date-today backend-latest-date)))
              (start-date          (iso-to-date (plist-get plist :start)))
              (new-plist-group-1   (if latest-plist-group
                                       (append latest-plist-group
                                               (list (or plist-1 plist)))
                                     (list start-date (or plist-1 plist))))
              (new-plist-group-2   (when (or plist-2 insert-new-group)
                                     (list date-today (or plist-2 plist)))))
        (goto-char (point-max))
        (when (not latest-plist-group)
          ;; first record
          (while (forward-comment 1) nil))
        (if (and plist-1 plist-2)
            ;; inactive, day-crossing record
            (progn
              (when latest-plist-group
                ;; not the first record
                (sexp-pre-read-check (current-buffer))
                (sexp-delete-list))
              (funcall sexp-pretty-print-function new-plist-group-1 (current-buffer))
              (dotimes (_ 2) (default-indent-new-line))
              (funcall sexp-pretty-print-function new-plist-group-2 (current-buffer)))
          ;; active, or non-day-crossing inactive record
          ;; insert into new group
          (if (or (not latest-plist-group) ;; first record
                  insert-new-group)
              (progn
                (default-indent-new-line)
                (funcall sexp-pretty-print-function new-plist-group-2 (current-buffer)))
            ;; insert into existing group
            (sexp-pre-read-check (current-buffer))
            (sexp-delete-list)
            (funcall sexp-pretty-print-function new-plist-group-1 (current-buffer))))
        (when save (save-buffer))
        t))))
#+END_SRC

**** plists-split-p                                             :function:
:PROPERTIES:
:CUSTOM_ID: program-data-structures-plists-split-p
:END:
[[file:../tests/tests.org::#tests-common-plists-split-p][tests]]

#+BEGIN_SRC lisp
(defun plists-split-p (old-plist new-plist)
  "Return t if OLD-PLIST and NEW-PLIST are split plists.
Split plists means the :stop time of old-plist must be the same as
the :start time of new-plist, and they must have identical
keyword-values (except :start and :stop)."
  (-let* (((&plist :stop  old-stop)  old-plist)
          ((&plist :start new-start) new-plist)
          (old-stop-unix     (parse-iso8601-time-string old-stop))
          (new-start-unix    (parse-iso8601-time-string new-start))
          (old-plist-no-time (plist-remove old-plist :start :stop))
          (new-plist-no-time (plist-remove new-plist :start :stop)))
    (and (time-equal-p old-stop-unix
                       new-start-unix)
         (equal old-plist-no-time
                new-plist-no-time))))
#+END_SRC

**** last-two-split-p                                          :procedure:
#+BEGIN_SRC lisp
(defun last-two-split-p (file)
  "Return non-nil if the latest two plists in FILE are split.
FILE must be a file containing plist groups, as created by
`plist-backend'.

Return value is either a list in the form
(OLDER-PLIST NEWER-PLIST), or nil."
  (sexp-in-file file
    (let* ((newer-group (progn (goto-char (point-max))
                               (backward-list)
                               (read (current-buffer))))
           (older-group (and (= 2 (length newer-group))
                             (backward-list 2)
                             (read (current-buffer))))
           ;; in case there was just one plist-group in the file
           (older-group (unless (equal older-group newer-group)
                          older-group))
           (newer-plist (second newer-group))
           (older-plist (first (last older-group))))
      (when (and older-plist newer-plist
                 (plists-split-p older-plist newer-plist))
        (list older-plist newer-plist)))))
#+END_SRC

**** plist-unify                                                :function:
#+BEGIN_SRC lisp
(defun plist-unify (old-plist new-plist)
  "Return a plist with the :start of OLD-PLIST and the :stop of NEW-PLIST."
  (let ((old-plist-wo-time (plist-remove old-plist :start :stop))
        (new-plist-wo-time (plist-remove new-plist :start :stop)))
    (cond ((not (and old-plist new-plist)) nil)
          ((equal old-plist-wo-time new-plist-wo-time)
           (let* ((plist    (copy-list old-plist))
                  (new-stop (plist-get new-plist :stop)))
             ;; Usually, a split plist has a `:stop' key. However, a
             ;; user may clock out and delete the stop time, resulting
             ;; in a split record without a `:stop' key.
             (if new-stop
                 (plist-put plist :stop new-stop)
               (plist-remove plist :stop))))
          (t (error "Attempt to unify plists with non-identical key-values")))))
#+END_SRC

**** remove-last                                           :writer:method:
#+BEGIN_SRC lisp
(defmethod remove-last ((backend plist-group-backend) &key (save t))
  (with-slots (file) backend
    (sexp-in-file file
      (goto-char (point-max))
      (when (backend-empty-p backend)
        (error "remove-last has nothing to remove in %s"
               (eieio-object-class-name backend)))
      (when (last-two-split-p file) ;; cannot be checked after changing the file
        ;; latest plist-group has only one plist, which is split - delete the group
        (backward-list)
        (sexp-delete-list))
      ;; remove the last plist in the last plist-group
      ;; if the plist-group has only one plist, delete the group
      (let ((plist-group (save-excursion (backward-list)
                                         (read (current-buffer)))))
        (if (= 2 (length plist-group))
            (progn (backward-list)
                   (sexp-delete-list))
          (down-list -1)
          (backward-list)
          (sexp-delete-list)
          (join-line))
        (when save (save-buffer))
        t))))
#+END_SRC

**** to-list                                               :reader:method:
#+BEGIN_SRC lisp
(defmethod to-list ((backend plist-group-backend))
  (backend-run-assertions backend)
  (loop-sexp-file for expr in (backend-file backend)
    append (reverse (rest expr))))
#+END_SRC

**** to-hash-table                                         :reader:method:
#+BEGIN_SRC lisp
(defmethod to-hash-table ((backend plist-group-backend))
  (with-slots (file) backend
    (loop-sexp-file for plist-group in file
      with table = (make-hash-table-1) do
      (puthash (first plist-group) (rest plist-group) table)
      finally return table)))
#+END_SRC

**** to-file                                               :writer:method:
#+BEGIN_SRC lisp
(defmethod to-file (hash-table (backend plist-group-backend) file)
  (check-type hash-table hash-table)
  (delete-file file)
  (create-file backend file)
  (reset-backend backend)
  (sexp-in-file file
    (goto-char (point-max))
    (loop for date being the hash-keys of hash-table
      using (hash-values plists) do
      (insert
       (plist-pp (apply #'list date plists))
       "\n")
      finally do (save-buffer))))
#+END_SRC

**** on-add                                                :writer:method:
#+BEGIN_SRC lisp
(defmethod on-add ((backend plist-group-backend))
  "Function run when a new plist-group is added at the end of a
`plist-group-backend' file."
  (with-slots (hash-table) backend
    (-let [(date plist) (latest-date-records backend)]
      (puthash date plist hash-table)
      (add-to-task-list (plist-get plist :name) backend))))
#+END_SRC

**** on-modify                                             :writer:method:
#+BEGIN_SRC lisp
(defmethod on-modify ((backend plist-group-backend))
  "Function run when the newest plist-group in a
`plist-group-backend' file is modified."
  (with-slots (hash-table) backend
    (-let* (((date . plists) (latest-date-records backend))
            (old-date        (ht-last-date hash-table))
            (old-plists      (gethash old-date hash-table)))
      (puthash date plists hash-table)
      (loop for plist in old-plists
        do (remove-from-task-list (plist-get plist :name) backend))
      (loop for plist in plists
        do (add-to-task-list (plist-get plist :name) backend)))))
#+END_SRC

**** on-remove                                             :writer:method:
#+BEGIN_SRC lisp
(defmethod on-remove ((backend plist-group-backend))
  "Function run when the newest plist-group in a
`plist-group-backend' file is deleted."
  (with-slots (hash-table) backend
    (-let* ((old-date        (ht-last-date hash-table))
            (old-plists      (gethash old-date hash-table)))
      (loop for plist in old-plists
        do (remove-from-task-list (plist-get plist :name) backend))
      (puthash old-date nil hash-table))))
#+END_SRC

**** verify                                                :reader:method:
#+BEGIN_SRC lisp
(defmethod verify ((backend plist-group-backend))
  (with-slots (file hash-table) backend
    ;; incorrectly ordered groups check
    (loop-sexp-file for group in file
      with old-date-iso with old-date-unix
      with new-date-iso with new-date-unix
      ;; while (not (bobp))
      do (setq new-date-iso  (first group)
               new-date-unix (parse-iso8601-time-string new-date-iso))
      when (and old-date-unix
                (time-less-p old-date-unix
                             new-date-unix))
      do (return (format "%s appears before %s on line %s"
                         new-date-iso old-date-iso (line-number-at-pos)))
      else do (setq old-date-iso  new-date-iso
                    old-date-unix new-date-unix)
      finally return "Yay, no errors! (...that I could find 💀)")))
#+END_SRC

**** extended protocol
***** latest-record                                       :reader:method:
#+BEGIN_SRC lisp
(defmethod latest-record ((backend plist-group-backend))
  (with-slots (file) backend
    (if (last-two-split-p file)
        (apply #'plist-unify (last-two-split-p (backend-file (active-backend))))
      (first (last (latest-date-records backend))))))
#+END_SRC

***** task-records-for-date                               :reader:method:
#+BEGIN_SRC lisp
(defmethod task-records-for-date ((backend plist-group-backend)
                                      task date-ts)
  (check-type task string)
  (check-type date-ts ts)
  (backend-run-assertions backend)
  (loop for plist in (gethash (date-iso date-ts)
                                 (backend-hash-table backend))
    when (equal task (plist-get plist :name))
    collect plist))
#+END_SRC

***** TODO active-days                           :reader:method:noexport:
#+BEGIN_SRC lisp :tangle no
(defmethod active-days ((backend plist-group-backend) task &key start end)
  (check-type task string)
  (backend-run-assertions backend))
#+END_SRC

***** replace-last                                        :writer:method:
=replace-last= is what is used for clocking out, so we split midnight-spanning intervals in this operation.

We apply the same hack as in the [[hack-note-plist-group-insert][insert]] method, removing and inserting the plist group instead of just the specific plist, to avoid having to update the pretty printer.

#+BEGIN_SRC lisp
(defmethod replace-last ((backend plist-group-backend) plist)
  (check-type plist plist)
  (when (backend-empty-p backend)
    (error "No record to replace in %s" (eieio-object-class-name backend)))
  (sexp-in-file (backend-file backend)
    (remove-last backend :save nil)
    (delete-trailing-whitespace)
    (insert backend plist :save nil)
    (save-buffer)
    t))
#+END_SRC

***** count-records                              :reader:method:noexport:
#+BEGIN_SRC lisp :tangle no
(defmethod count-records ((backend plist-group-backend)))
#+END_SRC

*** sqlite backend
**** class
#+BEGIN_SRC emacs-lisp
(defclass sqlite-backend (backend file-backend-mixin)
  ((extension :initform "sqlite"
              :accessor backend-ext
              :custom 'string)
   (connection :initform nil
               :initarg :connection
               :accessor backend-connection)))

(register-backend
 :sqlite "Store records in SQLite database."
 (make-instance 'sqlite-backend :path file))
#+END_SRC

**** initialize-instance                                          :method:
#+BEGIN_SRC emacs-lisp
(defmethod initialize-instance :after ((backend sqlite-backend)
                                          &rest _initargs)
  "Initialize connection for BACKEND based on its file."
  (with-slots (file connection) backend
    (when (and file (not connection))
      (setf connection (emacsql-sqlite file)))))
#+END_SRC

**** create-file
#+BEGIN_SRC emacs-lisp
(defmethod create-file ((backend sqlite-backend) &optional file)
  "Create file for BACKEND if it does not already exist.
Return the connection object from `emacsql-sqlite'."
  (let* ((file (or file (backend-file backend)))
         (db   (or (backend-connection backend)
                   (setf (backend-connection backend)
                         (emacsql-sqlite file)))))
    (loop
      for query in
      '(;; Properties are user-defined key-values stored as JSON.
        [:create-table properties
         ([(prop-id integer :primary-key)
           (properties text :unique :not-null)])]
        ;; An event is a timestamp with a name and optional properties.
        [:create-table event-names
         ([(name-id integer :primary-key)
           (name text :unique :not-null)])]
        [:create-table events
         ([(event-id integer :primary-key)
           (name-id integer :not-null :references event-names [name-id])])]
        ;; An interval is a time range with a name and optional properties.
        [:create-table interval-names
         ([(name-id integer :primary-key)
           (name text :unique :not-null)])]
        [:create-table intervals
         ([(interval-id integer :primary-key)
           (name-id integer :not-null :references interval-names [name-id])
           (start-time integer :not-null)
           ;; The latest interval may be ongoing, so the stop time may be NULL.
           (stop-time integer)
           (prop-id integer :references properties [prop-id])]
          (:unique [name-id start-time stop-time]))]
        ;; A date contains one or more events and intervals. It may
        ;; also contain properties.
        [:create-table dates
         ([(date-id integer :primary-key)
           (date integer :unique :not-null)
           (prop-id integer :references properties [prop-id])])]
        [:create-table date-events
         ([(date-id integer :not-null :references dates [date-id])
           (event-id integer :not-null :references events [event-id])])]
        [:create-table date-intervals
         ([(date-id integer :not-null :references dates [date-id])
           (interval-id integer :not-null :references intervals [interval-id])])])
      do (emacsql db query)
      finally return db)))
#+END_SRC

**** to-file                                                      :method:
#+BEGIN_SRC emacs-lisp :load no :tangle no
(defmethod to-file (hash-table (backend sqlite-backend) file)
  (loop with db = (emacsql-sqlite file)
    with count = 0
    for records being the hash-values of hash-table do
    (loop for record in records do
      (insert record db)
      (incf count)
      (when (zerop (% count 5))
        (message "migrate - %s records converted" count)))
    finally return count do
    (message "migrate - finished converting %s events." count)))
#+END_SRC

#+BEGIN_SRC emacs-lisp
(defun iso-to-unix (timestamp)
  (truncate (float-time (parse-iso8601-time-string timestamp))))

(defmethod to-file (hash-table (backend sqlite-backend) file)
  (with-slots (connection) backend
    (delete-file file)
    (emacsql-close connection)
    (setf connection nil)
    (create-file backend file)
    (loop for date in (sort (hash-table-keys hash-table) #'string-lessp) do
      ;; insert date if it does not exist
      (emacsql connection [:insert-or-ignore-into dates [date] :values [$s1]]
               (iso-to-unix date))
      (loop for plist in (gethash date hash-table) do
        (insert backend plist)))))
#+END_SRC

**** insert-properties                                            :writer:
#+BEGIN_SRC emacs-lisp
(defun sqlite-insert-properties (backend plist)
  "Insert properties from PLIST to (SQLite) BACKEND.
Properties are key-values excluding :name, :start, and :stop.

Insert nothing if the properties already exist. Return the
prop-id of the inserted or existing property."
  (with-slots (connection) backend
    (let* ((plist (plist-key-values plist))
           (props (if (functionp sqlite-properties-function)
                      (funcall sqlite-properties-function plist)
                    plist)))
      (emacsql connection
               [:insert-or-ignore-into properties [properties] :values [$s1]]
               props)
      (caar (emacsql connection [:select (funcall max prop-id) :from properties])))))
#+END_SRC

***** properties-to-json                                       :function:
#+BEGIN_SRC emacs-lisp
(defun sqlite-properties-to-json (plist)
  "Return PLIST as a JSON string."
  (json-encode
   ;; `json-encode' throws an error when it thinks
   ;; it sees "alists" which have numbers as
   ;; "keys", so we convert any cons cells and any
   ;; lists starting with a number to vectors
   (-tree-map (lambda (elt)
                (cond ((pp-pair-p elt)
                       (vector (car elt) (cdr elt)))
                      ((consp elt)
                       (vconcat elt))
                      (t elt)))
              plist)))
#+END_SRC

***** properties-function                               :custom:variable:
#+BEGIN_SRC emacs-lisp
(defcustom sqlite-properties-function nil
  "Function used to control the encoding of user key-values.
The function must accept a single argument, the plist of key-values.

Any non-function value results in key-values being inserted as
s-expressions in a text column."
  :type '(choice function (sexp :tag "Insert as s-expressions")))
#+END_SRC

**** insert
#+BEGIN_SRC emacs-lisp
(defmethod insert ((backend sqlite-backend) plist)
  (-let (((plist-1 plist-2)  (split-plist plist))
         (db  (backend-connection backend)))
    (loop for plist in (if (and plist-1 plist-2)
                              (list plist-1 plist-2)
                            (list plist))
      do
      (-let* (((&plist :name name :start start :stop stop) plist)
              (date-unix   (iso-to-unix (iso-to-date start)))
              (start-unix  (iso-to-unix start))
              (stop-unix   (and stop (iso-to-unix stop)))
              name-id interval-id)
        ;; insert name if it does not exist
        (emacsql db [:insert-or-ignore-into interval-names [name]
                     :values [$s1]]
                 name)
        ;; insert interval properties if they do not exist
        (sqlite-insert-properties backend plist)
        ;; insert interval and associate it with the date
        (setq name-id
              (caar (emacsql db [:select [name-id]
                                 :from interval-names
                                 :where (= name $s1)]
                             name)))
        (emacsql db [:insert-or-ignore-into intervals [name-id start-time stop-time]
                     :values [$s1 $s2 $s3]]
                 name-id start-unix stop-unix)
        (emacsql db [:insert-or-ignore-into dates [date]
                     :values [$s1]] date-unix)
        (setq date-id
              (caar (emacsql db [:select [date-id] :from dates
                                 :where (= date $s1)]
                             date-unix))
              interval-id
              (caar (emacsql db [:select (funcall max interval-id) :from intervals])))
        (emacsql db [:insert-into date-intervals [date-id interval-id]
                     :values [$s1 $s2]]
                 date-id interval-id)))))
#+END_SRC

**** open-file
#+BEGIN_SRC emacs-lisp
(defmethod edit-backend ((backend sqlite-backend))
  (require 'sql)
  (switch-to-buffer
   (sql-comint-sqlite 'sqlite (list file))))
#+END_SRC

**** latest-record
#+BEGIN_SRC emacs-lisp
;; SELECT * FROM TABLE WHERE ID = (SELECT MAX(ID) FROM TABLE);
;; SELECT * FROM tablename ORDER BY column DESC LIMIT 1;
(defmethod latest-record ((backend sqlite-backend) db)
  (emacsql db [:select * :from events :order-by rowid :desc :limit 1]))
#+END_SRC

**** task-records-for-date
#+BEGIN_SRC emacs-lisp
(defmethod task-records-for-date ((backend sqlite-backend) task date-ts))
#+END_SRC

**** active-days
#+BEGIN_SRC emacs-lisp
(defmethod active-days ((backend sqlite-backend) task))
#+END_SRC

**** replace-last
#+BEGIN_SRC emacs-lisp
(defmethod replace-last ((backend sqlite-backend) plist)
  (emacsql db [:delete-from events :where ]))
#+END_SRC

** Migration
:PROPERTIES:
:CUSTOM_ID: program-migration
:END:
*** remove-prefix
#+BEGIN_SRC lisp
(defun remove-prefix (string)
  (replace-regexp-in-string "^" "" string))
#+END_SRC

*** migrate                                                       :command:
#+BEGIN_SRC lisp
(defun migrate ()
  "Convert from one Chronometrist backend to another."
  (interactive)
  (let* ((input-backend
          (read-backend-name "Backend to convert: "
                              backends-alist))
         (input-file-suggestion (backend-file input-backend))
         (input-file (read-file-name "File to convert: " nil
                                     input-file-suggestion t
                                     input-file-suggestion))
         (output-backend (read-backend-name
                          "Backend to write: "
                          backends-alist
                          (lambda (keyword)
                            (not (equal (second
                                         (alist-get keyword backends-alist))
                                        input-backend)))))
         (output-file-suggestion (backend-file output-backend))
         (output-file (read-file-name "File to write: " nil nil nil
                                      output-file-suggestion))
         (input-backend-name  (remove-prefix
                               (symbol-name
                                (eieio-object-class-name input-backend))))
         (output-backend-name (remove-prefix
                               (symbol-name
                                (eieio-object-class-name output-backend))))
         (confirm (yes-or-no-p
                   (format "Convert %s (%s) to %s (%s)? "
                           input-file
                           input-backend-name
                           output-file
                           output-backend-name)))
         (confirm-overwrite
          (if (and confirm
                   (file-exists-p output-file)
                   (not (file-empty-p output-file)))
              (yes-or-no-p
               (format "Overwrite existing non-empty file %s ?"
                       output-file))
            t)))
    (if (and confirm confirm-overwrite)
        (to-file (backend-hash-table input-backend)
                  output-backend
                  output-file)
      (message "Conversion aborted."))))
#+END_SRC

*** table                                                        :variable:
:PROPERTIES:
:VALUE:    hash table
:END:
#+BEGIN_SRC lisp
(defvar migrate-table (make-hash-table-1))
#+END_SRC

*** EXTEND populate                                                :writer:
:PROPERTIES:
:STATE: migrate-table
:END:

1. [ ] support other timeclock codes - currently only "i" and "o" are supported.

#+BEGIN_SRC lisp
(defun migrate-populate (in-file)
  "Read data from IN-FILE to `migrate-table'.
IN-FILE should be a file in the format supported by timeclock.el.
See `timeclock-log-data' for a description."
  (clrhash migrate-table)
  (with-current-buffer (find-file-noselect in-file)
    (save-excursion
      (goto-char (point-min))
      (let ((key-counter 0))
        (while (not (eobp))
          (let* ((event-string (buffer-substring-no-properties (point-at-bol)
                                                               (point-at-eol)))
                 (event-list   (split-string event-string "[ /:]"))
                 (code         (first event-list))
                 (date-time    (--> (seq-drop event-list 1)
                                    (seq-take it 6)
                                    (mapcar #'string-to-number it)
                                    (reverse it)
                                    (apply #'encode-time it)
                                    (format-time-iso8601 it)))
                 (project-or-comment
                  (replace-regexp-in-string
                   (rx (and (or "i" "o") " "
                            (and (= 4 digit) "/" (= 2 digit) "/" (= 2 digit) " ")
                            (and (= 2 digit) ":" (= 2 digit) ":" (= 2 digit))
                            (opt " ")))
                   ""
                   event-string)))
            (pcase code
              ("i"
               (incf key-counter)
               (puthash key-counter
                        `(:name ,project-or-comment :start ,date-time)
                        migrate-table))
              ("o"
               (--> (gethash key-counter migrate-table)
                    (append it
                            `(:stop ,date-time)
                            (when (and (stringp project-or-comment)
                                       (not
                                        (string= project-or-comment "")))
                              `(:comment ,project-or-comment)))
                    (puthash key-counter it migrate-table)))))
          (forward-line)
          (goto-char (point-at-bol))))
      nil)))
#+END_SRC

*** timelog-file-to-sexp-file                                      :writer:
#+BEGIN_SRC lisp
(defvar timeclock-file)

(defun migrate-timelog-file-to-sexp-file (&optional in-file out-file)
  "Migrate your existing `timeclock-file' to the Chronometrist file format.
IN-FILE and OUT-FILE, if provided, are used as input and output
file names respectively."
  (interactive `(,(if (featurep 'timeclock)
                      (read-file-name (concat "timeclock file (default: "
                                              timeclock-file
                                              "): ")
                                      user-emacs-directory
                                      timeclock-file t)
                    (read-file-name (concat "timeclock file: ")
                                    user-emacs-directory
                                    nil t))
                 ,(read-file-name (concat "Output file (default: "
                                          (locate-user-emacs-file "chronometrist.sexp")
                                          "): ")
                                  user-emacs-directory
                                  (locate-user-emacs-file "chronometrist.sexp"))))
  (when (if (file-exists-p out-file)
            (yes-or-no-p (concat "Output file "
                                 out-file
                                 " already exists - overwrite? "))
          t)
    (let ((output (find-file-noselect out-file)))
      (with-current-buffer output
        (erase-buffer)
        (migrate-populate in-file)
        (maphash (lambda (_key value)
                   (plist-pp value output)
                   (insert "\n\n"))
                 migrate-table)
        (save-buffer)))))
#+END_SRC

*** check                                                          :writer:
#+BEGIN_SRC lisp
(defun migrate-check ()
  "Offer to import data from `timeclock-file' if `file' does not exist."
  (when (and (bound-and-true-p timeclock-file)
             (not (file-exists-p file)))
    (if (yes-or-no-p (format (concat "Chronometrist v0.3+ uses a new file format;"
                                     " import data from %s ? ")
                             timeclock-file))
        (migrate-timelog-file-to-sexp-file timeclock-file file)
      (message "You can migrate later using `migrate-timelog-file-to-sexp-file'."))))
#+END_SRC

** Provide
#+BEGIN_SRC lisp :comments no
(provide 'chronometrist)

;;; chronometrist.el ends here
#+END_SRC