Offpunk is now able to display pages, pictures and follow links. A lot
of TODO have been clearly identified. I think that the whole
GeminiItem() object will be removed and URL will be accessed directly
with a dict{url, renderer} to avoid redrawing all the time.
Next challenge: remove GeminiItems!
This seems to be the most common convention among gopher servers (e.g.
Gophernicus, go-gopher).
Added a script to migrate the cache to the new format. Alternatively the
following command can be used for migration in a POSIX system:
find ~/.cache/offpunk/gopher/ -type f -name 'index.txt' -exec sh -c 'mv {} "$(dirname {})/gophermap"' \;
The previous code was using the first path segment as an item type, if
it was present and a single character long. However, the standard
requires the first character after the slash to be used, if present,
regardless of whether if forms a complete path segment. This rewrite
conforms the code to the standards behaviour, and fixes both the
original problem with gopher://alexschroeder.ch/ and the problem created
with other gopherholes in my attempt to fix it.
I've also changed one piece of code that generates a gopher URL to
better conform to the standard, by avoiding the introduction of leading
slashes. This should prevent any gopherholes that don't accept leading
slashes (including gopher://alexschroeder.ch/) from having problems with
requests sent from offpunk.
The previous code would throw KeyError if a key was present in both
os.environ and env, e.g. when LESSHISTFILE is defined in the
environment. The current code ensures that for keys present in both
dictionaries the value from env will be used.
This fixes text encoding detection for gopher requests which was using
the chardet module without importing it.
Here's a URL that causes a crash before the fix:
gopher://sdf.org/0/users/d1337/textfiles/hacker_crackdown.txt
Flit is the simplest of PEP517 build systems so I used it.
Packagers will need to switch from legacy (setup.py) mode to PEP517, if
not already.
Most offpunk.py changes are stripped whitespace. Relevant are:
- Added module docstring (__doc__ variable)
- Added __version__ variable
These two will be used by Flit so you will need to bump version in one
place only.
Previously, offpunk would send the UTF-8 host in the Gemini request.
This changes it to first IDNA-encode the hostname in the URL before
making the request. This bug only existed for Gemini requests, since the
host is already being encoded for Spartan requests and `requests`
handles this for HTTP URLs.
Additionally, the code now treats both UTF-8 hosts and their IDNA
equivalents as the same host for the purposes of identifying
cross-domain redirects.
Not everyone use zathura and feh so they are not removed by default.
To restore previous behaviour, add the following lines to your offpunkrc:
handler image/* feh -. %s
handler application/pdf zathura %s
In order for `readline` to handle up/down arrow presses with ANSI
escape codes in the prompt, all non-printable characters must be
prefaced with \001 and suffixed with \002. Otherwise, long lines
do not get entirely cleared when scrolling up/down.
This should fix https://notabug.org/ploum/offpunk/issues/9 . This
involves a few closely-related changes to subprogram execution:
- If a path, url or file contents were being passed using `cat` or
`echo`, the code was changed to pass the file/string on stdin. This
also makes several pipelines into single programs and should allow for
the removal of `shell=True` in the future.
- For `file`, `xdg-open` and `less`, which either can't accept their
input on stdin or otherwise use the path, the paths are now being
escaped with `shlex.quote()`.
- Finally, the environment variable $LESSHISTFILE is now being set in
python code, where escaping is not necessary.
Notably, the argument to `grep` in `less_cmd()` is not quoted in this
commit, since I was unsure of how it was meant to be used. If the
argument is not already quoted, this should probably be passed through
`shlex.quote()`.
This does not do the following, which may be desired:
- This does not disable `shell=True` anywhere, since `subprocess.run()`
requires the command to be already split into a list of strings. I
think this would just require a `shlex.split()` in `run()` when this
is disabled, but it may require more thought.
- Some of the invoked programs (with the notable exception of `echo` and
`xdg-open`) support the use of "--" to prevent any following arguments
from being treated as program flags if they start with "-". I don't
believe there are any paths that start with "-", but it may make sense
to include this where possible.
I have briefly tested this commit, but it touches quite a few code
paths, so there might be bugs that I missed.
Fixes a crash when trying to look up the contents of the current index
using `ls` command whenever there is no current page loaded by making a
current item a requirement.
This is an an experimental and unstable release. Lot of breakages are expected.
Wait for 2.1 if you are not willing to do testing/bug reporting.
- New command-line tool: "netcache"
- New command-line tool: "ansirenderer"
## 1.10 - unreleased
- IMPORTANT : new optional dependency : python-chardet
- IMPORTANT : Gopher directory index filename changed from "index.txt" to "gophermap". To update the cache to the new format run the `migrate-offpunk-cache` script (Sotiris Papatheodorou)
- "set accept_bad_ssl_certificates True" now also used for Gemini expired certificates
UPGRADE: Users who subscribed to pages before 0.4 should run once the command "list subscribe subscribed". Without that, the subscribed list will be seen as a normal list by sync.
- New list command : "list freeze" and "list suscribe"
- Pictures are now displayed directely in terminal (suggested by kelbot)
- "open" command to open current page/image/file with external handler.
- "set width XX" now works to set the max width. If smaller, terminal width is used (thanks kelbot for reporting the bug)
- RSS feeds are now rendered as Gemlogs to improve consistency while browsing
- "subscribe" will detect feeds in html pages if any
- "less" will restore previous position in a page (requires less 572+)
- Improved syncing performances and multiple bug/crash fixes.
- "version" will now display info about your system installation
- "info" command will display technical information about current page
- "sync" allows you to do the sync from within Offpunk
Offpunk is a fork of the original [AV-98](https://tildegit.org/solderpunk/AV-98) by Solderpunk and was originally called AV-98-offline as an experimental branch.
## How to use
Offpunk is a single python file. Installation is optional, you can simply download and run "./offpunk.py" or "python3 offpunk.py" in a terminal.
You use the `go` command to visit a URL, e.g. `go gemini.circumlunar.space`. (gemini:// is assumed if no protocol is specified. Supported protocols are gemini, gopher, finger, http, https, mailto, spartan and file).
Links in pages are assigned numerical indices. Just type an index to follow that link. If page is too long to fit on your screen, the content is displayed in the less pager (by default). Type `q` to quit and go back to Offpunk prompt. Type `view` or `v` to display it again. (`view full` or `v full` allows to see the full html page instead of the article view. `v feed` try to display the linked RSS feed and `v feeds` displays a list of available feeds. This only applies to html pages)
Use `add` to add a capsule to your bookmarks and `bookmarks` or `bm` to show your bookmarks (you can create multiple bookmarks lists, edit and remove them. See the `list` manual with `help list`).
Use `offline` to only browse cached content and `online` to go back online. While offline, the `reload` command will force a re-fetch during the next synchronisation.
Use the `help` command to learn about additional commands. Some abreviations are available. See `abbrevs`.
When launched with the "--sync" option, offpunk will run non-interactively and fetch content from your bookmarks, lists and ressources tentatively accessed while offline. New content found in your subscriptions (see `help subscribe`) will be automatically added to your tour (use `tour ls` to see your current tour, `tour` without argument to access the next item and `tour X` where X is a link number to add the content of a link to your tour).
With "--sync", one could specify a "--cache validity" in seconds. This option will not refresh content if a cache exists and is less than the specified amount of seconds old.
For example, running
`offpunk --sync --cache-validity 43200`
will refresh your bookmarks if those are at least 12h old. If cache-validity is not set or set to 0, any cache is considered good and only content never cached before will be fetched. `--assume-yes` will automatically accept SSL certificates with errors instead of refusing them.
Offpunk can also be configured as a browser by other tool. If you want to use offpunk directly with a given URL, simply type:
`offpunk URL`
To have offpunk fetch the URL at next sync and close immediately, run:
`offpunk --fetch-later URL`
## More
Important news and releases will be announced on the offpunk-devel mailing list
=> https://lists.sr.ht/~lioploum/offpunk-devel
Questions can be asked on the users mailing list:
=> https://lists.sr.ht/~lioploum/offpunk-users
## Dependencies
AV-98 has no "strict dependencies", i.e. it will run and work without anything
else beyond the Python standard library. However, it will "opportunistically
Offpunk has no "strict dependencies", i.e. it should run and work without anything
else beyond the Python standard library and the "less" pager. However, it will "opportunistically
import" a few other libraries if they are available to offer an improved
experience.
experience or some other features. Python libraries requests, bs4 and readability are required for http/html support. Images are displayed if chafa or timg are presents (python-pil is needed for chafa version before 1.10). When displaying only a picture (not inline), rendering will be pixel perfect in compatible terminals (such as Kitty) if chafa is at least version 1.8 or if timg is used.
* The [ansiwrap library](https://pypi.org/project/ansiwrap/) may result in
neater display of text which makes use of ANSI escape codes to control colour.
* The [cryptography library](https://pypi.org/project/cryptography/) will
provide a better and slightly more secure experience when using the default
TOFU certificate validation mode and is highly recommended.
To avoid using unstable or too recent libraries, the rule of thumb is that a library should be packaged in Debian/Ubuntu. Keep in mind that Offpunk is mainly tested will all libraries installed. If you encounter a crash without one optional dependencies, please report it. Patches and contributions to remove dependencies or support alternatives are highly appreciated.
* PIP: [requirements file to install dependencies with pip](requirements.txt)
* Debian Unstable: [Official Package by Étienne Mollier](https://packages.debian.org/sid/offpunk)
* Ubuntu/Debian: [command to install dependencies on Ubuntu/Debian without pip](ubuntu_dependencies.txt)
* Arch: [AUR package for Arch Linux, maintained by kseistrup](https://aur.archlinux.org/packages/offpunk-git)
* [Nix](https://nixos.org/): [package](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/browsers/offpunk/default.nix), maintained by [DamienCassou](https://github.com/DamienCassou)
* Alpine Linux: [package maintained by mio](https://pkgs.alpinelinux.org/packages?name=offpunk)
* Please contribute packages for other systems, there’s a [mailing-list dedicated to packaging](https://lists.sr.ht/~lioploum/offpunk-packagers).
Run command `version` in offpunk to see if you are missing some dependencies.
Mandatory or highly recommended (packagers should probably make those mandatory):
* [less](http://www.greenwoodsoftware.com/less/): mandatory but is probably already on your system
* [file](https://www.darwinsys.com/file/) is used to get the MIME type of cached objects. Should already be on your system.
* [xdg-utils](https://www.freedesktop.org/wiki/Software/xdg-utils/) provides xdg-open which is highly recommended to open files without a renderer or a handler. It is also used for mailto: command.
* The [cryptography library](https://pypi.org/project/cryptography/) will provide a better and slightly more secure experience when using the default TOFU certificate validation mode and is recommended (apt-get install python3-cryptography).
Dependencies to enable web browsing (packagers may put those in an offpunk-web meta-package but it is recommended to have it for a better offpunk experience)
* [Python-requests](http://python-requests.org) is needed to handle http/https requests natively (apt-get install python3-requests). Without it, http links will be opened in an external browser
* [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup) and [Readability](https://github.com/buriy/python-readability) are both needed to render HTML. Without them, HTML will not be rendered or be sent to an external parser like Lynx. (apt-get install python3-bs4 python3-readability or pip3 install readability-lxml)
* [Python-feedparser](https://github.com/kurtmckee/feedparser) will allow parsing of RSS/Atom feeds and thus subscriptions to them. (apt-get install python3-feedparser)
* [Chafa](https://hpjansson.org/chafa/) allows to display pictures in your console. Install it and browse to an HTML page with picture to see the magic.
* [Timg](https://github.com/hzeller/timg) is a slower alternative to chafa for inline images. But it has better rendering when displaying only the image. Install both to get the best of both world but if you need to choose one, choose Chafa.
* [Python-pil](http://python-pillow.github.io/) is required to only display the first frame of animated gif with chafa if chafa version is lower than 1.10.
Nice to have (packagers should may make those optional):
* [Xsel](http://www.vergenet.net/~conrad/software/xsel/) allows to `go` to the URL copied in the clipboard without having to paste it (both X and traditional clipboards are supported). Also needed to use the `copy` command. (apt-get install xsel)
* [Python-setproctitle](https://github.com/dvarrazzo/py-setproctitle) will change the process name from "python" to "offpunk". Useful to kill it without killing every python service.
* [Python-chardet](https://github.com/chardet/chardet) is used to detect the character encoding on Gopher (and may be used more in the future)
## Features
* TOFU or CA server certificate validation
* Extensive client certificate support if an `openssl` binary is available
* Ability to specify external handler programs for different MIME types
* Gopher proxy support (e.g. for use with
[Agena](https://tildegit.org/solderpunk/agena))
* Advanced navigation tools like `tour` and `mark` (as per VF-1)
* Bookmarks
* Browse https/gemini/gopher/spartan without leaving your keyboard and without distractions
* Built-in documentation: type `help` to get the list of command or a specific help about a command.
* Offline mode to browse cached content without a connection. Requested elements are automatically fetched during the next synchronization and are added to your tour.
* HTML pages are prettified to focus on content. Read without being disturbed or see the full page with `view full`.
* RSS/Atom feeds are automatically discovered by `subscribe` and rendered as gemlogs. They can be explored with `view feed` and `view feeds`.
* Support "subscriptions" to a page. New content seen in subscribed pages are automatically added to your next tour.
* Complex bookmarks management through multiple lists, built-in edition, subscribing/freezing lists and archiving content.
* Advanced navigation tools like `tour` and `mark` (as per VF-1). Unlike AV-98, tour is saved on disk accross sessions.
* Ability to specify external handler programs for different MIME types (use `handler`)
* Enhanced privacy with `redirect` which allows to block a http domain or to redirect all request to a privacy friendly frontent (such as nitter for twitter).
* Non-interactive cache-building with configurable depth through the --sync command. The cache can easily be used by other software.
* IPv6 support
* Supports any character encoding recognised by Python
## Lightning introduction
You use the `go` command to visit a URL, e.g. `go gemini.circumlunar.space`.
Links in Gemini documents are assigned numerical indices. Just type an index to
follow that link.
If a Gemini document is too long to fit on your screen, use the `less` command
to pipe it to the `less` pager.
Use the `help` command to learn about additional commands.
* Cryptography : TOFU or CA server certificate validation
* Cryptography : Extensive client certificate support if an `openssl` binary is available
## RC files
You can use an RC file to automatically run any sequence of valid AV-98
You can use an RC file to automatically run any sequence of valid Offpunk
commands upon start up. This can be used to make settings controlled with the
`set` or `handler` commanders persistent. You can also put a `go` command in
your RC file to visit a "homepage" automatically on startup, or to pre-prepare
a `tour` of your favourite Gemini sites.
a `tour` of your favourite Gemini sites or `offline` to go offline by default.
The RC file should be called `offpunkrc` and goes in $XDG_CONFIG_DIR/offpunk (or .config/offpunk or .offpunk if xdg not available). In that file, simply write one command per line, just like you would type them in offpunk.
## Cache design
The offline content is stored in ~/.cache/offpunk/ as plain .gmi/.html files. The structure of the Gemini-space is tentatively recreated. One key element of the design is to avoid any database. The cache can thus be modified by hand, content can be removed, used or added by software other than offpunk.
There’s no feature to automatically trim the cache. But part of the cache can safely be removed manually.
The RC file should be called `av98rc`. AV-98 will look for it first in
`~/.av98/` and second in `~/.config/av98/`. Note that either directory might
already exist even if you haven't created it manually, as AV-98 will, if
necessary, create the directory itself the first time you save a bookmark (the
bookmark file is saved in the same location). AV-98 will create
`~/.config/av98` only if `~/.config/` already exists on your system, otherwise
# OFFPUNK - An Offline-First Browser for the Smolnet
Offpunk is a command-line browser and feed reader dedicated to browsing the Web, Gemini, Gopher and Spartan. Thanks to its permanent cache, it is optimised to be used offline with rare connections but works as well when connected.
Offpunk is optimised for reading and supports readability mode, displaying pictures, subscribing to pages or RSS feeds, managing complex lists of bookmarks. Its integrated help and easy commands make it a perfect tool for command-line novices while power-users will be amazed by its shell integration.
Offpunk is written in Python 3 by Ploum. It aims to be portable and minimise dependencies, making them optional. It supports http/https/gopher/gemini/spartan on both IPv4 and IPv6.
=> install.gmi Installing Offpunk and dependencies
Lionel Dricot
Sotiris Papatheodorou - sotiris at papatheodorou.xyz
Maeve Sproule - code at sprock.dev
Anna “CyberTailor” - cyber+misc at sysrq.in
Anna “CyberTailor”
Anna “CyberTailor”
Ben Winston - ben at benwinston.us
Marty Oehme - marty.oehme at gmail.com
ploum
mio
Damien Cassou
Iván Ruvalcaba
Björn Wärmedal
