16 KiB
+++ title = "What happened to our dreams of real-time collaborative writing?" date = 2019-07-31 [taxonomies] skills = ["html5", "markdown"] +++
Lately, i've been experimenting with real-time collaborative writing (pads) for editing a static website. Here's my story down the rabbithole of the Etherpad ecosystem and why writing Markdown with Etherpad in 2019 is still so complicated. Don't worry, we won't go too deep because i want to keep whatever's left of my sanity after spending a few days crawlin through a somewhat-antique codebase riddled with dynamic typing.
Pad all the things!
Etherpad is a tool for real-time edition of documents. You create a pad on a pad server, and by giving other people its address, they can join in on the fun and start writing with you. Each person gets their own color, and all edits are saved so we can always roll back to a previous version if we need to. Also, pad URLs are predictable and easy to memorize: https://SERVER/p/NAME will take me to pad NAME (and create it if it doesn't exist yet) on Etherpad instance SERVER.
Back in the late 2000s, when Etherpad was released as an open source project, everyone was really thrilled about it. Many server operators started hosting public instances of Etherpad so anybody could collaborate on text. This was used by many collectives to write posts, do translations, or document things as people would do on a wiki. Many of us thought it would become very common over the years to have content-managing systems integrate pads in some form or another to promote real-time cooperation. This mostly did not happen, even though some CMS plugins do just that.
The missing features
Over the years, Etherpad has grown an ecosystem of plugins that augmented its capacities. Some plugins are very well designed and supported, like ep_mypads (maintained by Framasoft) which adds support for user-based and group-based permissions on pads because not everyone should be able to see/edit your content. But many others are at best unmaintained, if not utterly broken. That's the case with most plugins intended to add features to the editor, as we are going to see.
Support for adding images and links (or basically anything that is not plaintext) to a pad was nowhere to be seen, so some people wrote plugins for this purpose.
Links
Links are the foundation of the web. But because of "security reasons", the Etherpad team never implemented proper support for links. They claimed a such feature could be used to trick users into clicking a malicious link, which is of course true of all links on the web. So the only way to link to another page by default is to write out its complete URL. You can link to a page, but you cannot give this link a nice title to fit into your content.
ep_linkify adds the possibility to use internal links with the [[page]] syntax which generates a <a href="page">page</a> link. Still, no custom titles for links. While this is not too bad, ep_linkify is unmaintained. It was updated 2015 to follow some syntax changes, but nothing more. Also, the version found on the NPM repository (the recommended way to install Etherpad plugin) does not include this fix. So if you want to use this plugin, you need to install it from git directly, like so:
$ # From your etherpad-lite folder
$ git clone https://github.com/fourplusone/etherpad-plugins /tmp/etherpad-plugins
$ # Copy the plugin to the node_modules folder where Etherpad looks for plugins
$ cp -r /tmp/etherpad-plugins/ep_linkify node_modules/
ep_embedded_hyperlinks2 (successor to the defunct ep_embedded_hyperlinks) is a more complex plugin that adds a link button in the toolbox, so you can select any block of text and add a link on it just like you would in any WYSIWYG editor. However, the links are not generated in the HTML export.
Images
There are two ways to embed images in an HTML document. The first is to link to the image, and let the browser load it separately. The second uses data: URIs to encode the image directly into the document (making it huge). Of these two approaches, the first is prefered in most situations because it allows to load the HTML document without necessarily having to download images (which can take a long time on slow connections).
ep_previewimages embeds external images in the editor, but they are not added to the HTML export. ep_copy_paste_images only accepts images from the clipboard and encodes them to data URIs. ep_image_upload only accepts images through file upload, not supporting external URLs.
In any case, none of these plugin support a description alongside the image, like <img src="foo.png" alt="description"> in HTML or  in Markdown. So yet again, the situation is far from ideal.
Markdown
Markdown, the simple markup language used almost everywhere on the modern web, also has very poor support on Etherpad.
ep_markdown keeps Etherpad's markup language as the pad content and simply decorates the view with illusions of Markdown markup. It's visually ugly, but it's also far from functional. Links still can't have titles, and the Markdown export generates some weird stuff, like replacing raw URLs with \url{https://...}. Also, this plugin is unmaintained (issues on the bugtracker get no reply).
ep_markdownify works with actual Markdown within the pad. So you write Markdown as the pad content, and then the plugin renders it in the view. Behind the scenes, ep_markdownify doesn't use a proper Markdown parser; it uses the regex/prefix/suffix hacks provided by Etherpad's plugin API, which works surprisingly well! The support for images is far from ideal: they're stacked in the bottom-right corner of the screen. But the links work great! Except when they don't, because Etherpad's linking of raw URLs will take the closing parenthesis (and any text following until whitespace) into the href. So if you write [link](https://thunix.net), clicking link will take you to thunix.net, but clicking thunix.net will take you to thunix.net).
So what's great with ep_markdownify is we can use the plaintext export as a Markdown source for the pad. What's less great is that this plugin is also unmaintained (see here how to patch it for more recent versions of Etherpad).
As we just saw, the situation with Markdown is also not ideal. Still, many people requested Markdown support (1, 2, 3, 4) because it would bring many features that have been missing for years in the Etherpad ecosystem.
When Etherpad came out, many markup languages were emerging. Markdown lacked many features and didn't have a proper specification. Nowadays, there is the Commonmark spec and Markdown is the de-facto standard for writing content on many blogs, forums, commenting systems, bugtrackers and other projects. Etherpad has been experimenting with their own markup language, but Markdown now addresses many of the issues they faced over the years.
The future?
A week ago, i took for granted it would be easy to link pads together and get a Markdown export so i could generate a static site from it. But it all turned out to be wrong, mostly because i've had a consumer relationship with Etherpad. As an end-user, i did not see all of Etherpad's technical debt. However, things are still moving with Etherpad and the latest v1.7.5 release brought support for skins. So maybe there is hope the situation improves.
It would not be hard to replace Etherpad with another solution such as HackMD or Gobby, or even to write one from scratch. But taking a such approach would not address the issues for all the communities who rely Etherpad on a daily basis. So what can we do to change the status quo?
An obvious way of improvement would be to properly decouple and document different parts of Etherpad's codebase, so the synchronization logic can be used as a library for any custom client to use. That would allow easy integration of Markdown (and many other markup formats) as well as custom pad types, like HackMD proposes "templates" for making a slideshow or a docbook.
To some extent, that's already possible using Etherpad's API, but this requires using an API key so you couldn't just plug your 3rd party client to any server. Properly exporting and documenting this internal API would help make initiatives such as etherpad-vim (who don't use an apiKey and therefore works with every server) easier to develop and maintain.
If supporting 3rd-party clients is not going to be a priority for the Etherpad project, we could probably try and improve the existing plugins. Forking ep_markdownify to implement HTML export and better support for images is a reasonable idea. But Etherpad's current plugin API prevents us from working with the whole content of the pad (it works on a node-per-node basis with hardcoded regex and prefix/suffix rules), making the situation for multi-line markup (such as tables) a lot tougher.
This situation with Etherpad is so disconnected from the dreams we had back then. And not because those dreams were technically unachievable, but because we've let them fall into the hands of cyberfascist silos such as Google Docs and Microsoft Office. It's time to reclaim our dreams.
Who knows, maybe in a couple years, federating pad permissions over different Etherpad instances will be a thing?
Annexes
Accessibility concerns
Despite some efforts to make Etherpad more accessible with screen readers (for visually-impaired users), requiring a GUI browser limits usability for people with braille terminals, as well as for people with old hardware incapable of running a modern browser. Supporting 3rd-party clients as proposed above would help a lot in regards to accessility.
Towards a federation?
Etherpad is used by many collectives to provide private pads to their users, with the ep_mypads plugin. Now that we have user management, we could federate those users with other instances. Implementing a federation protocol such as ActivityPub, XMPP, or SSB would allow a user on an instance to give access to a private pad of theirs, to a user on another instance.
Currently, ep_mypads can either use a server-defined external identity provider (through LDAP) or act as one with a hardcoded list of users. In the context of a federation protocol, the instance will in any case act as an identity provider. But the whole protocol doesn't have to be implemented. The point is to not reinvent the wheel. Smart people already have been through the process of writing federation protocols, and we can benefit from their experiences and past mistakes. Concretely, we can use one of the many readily-available software libraries for those federation protocols.
Additionally, and independently from the federation protocol adopted, we could leverage user-supplied identity providers through standard federated authentication protocols: IndieAuth (from the indieweb ecosystem), Id4Me (from the DNS ecosystem), SAML, RFC8414 (from the OAuth 2.0 ecosystem) or even XEP-0070 (from the XMPP ecosystem).
Security concerns
It is worth noting that using external resources raises some security questions. For example, loading external images can allow the hosts of these images to track pad users.To tackle this issue, we can either make image opt-in on the client side (through a clickable « Download image » block), or implement a server side cache of linked images and have the client query the cache instead of the actual source of the image. The second option would play well with Etherpad instances hosted onion services such as Riseup's pads (also available at pad.riseup.net).
Links are less of a threat because users have the possibility to inspect them. Still, someone can maliciously replace a link with the intent to phish people or infect their computer through zero-day or unpatched vulnerabilities, knowing that many people won't bother to inspect them. These issues could be mitigated in various ways.
For example, we could have an instance-wide whitelist of domains anyone could link to. But this approach brings more questions than answers, in that pads are currently mostly a neutral and common tool where - except for some private instances - anyone can write anything. Restricting what users can do in the name of security rarely brings any security, but always empowers some people to exert more control over our tools and our lives.
But there's a less invasive way to deal with this: make sure editing a URL visually changes the color of the displayed link to match the person who did the edit. Just like with any other piece of text, with the subtlety that authorship of the link title would have to change when authorship of the link URL changes. So if Alice adds a link to her public key in the pad ([Alice](https://example.com/alice.asc), when Mallory changes the link to https://example.net/alice.legit.asc, authorship for of the link should be attributed to Mallory. This way, when Bob goes to the pad where they usually hang out with Alice, he will notice instantly that the link doesn't match Alice's color.
Then again, that's what we can do if we think links are a threat. But maybe they're not! Encouraging users to inspect links isn't a waste of energy. If we feel like we need more security on our pads they contain crucial/private data, maybe we should consider setting up some forms of permissions management (or ACL) so only logged-in users can read oredit certain pages.
Links are rather neutral as a tool: they are not themselves vectors of power dynamics, allowing centralization or decentralization of power depending on how links are used in practice. Links are the foundation of the web: restricting the use of links in web software contributes to damaging the World Wide Web in irreversible ways.