lucidiot
/
objtools
1
0
Fork 0
Code Issues Pull Requests Releases Activity

Add documentation

This commit is contained in:
Lucidiot 2019-10-03 08:39:35 +02:00
parent e5dcb746ad
commit 269e578dd4
Signed by: lucidiot
GPG Key ID: 3358C1CA6906FB8D
9 changed files with 585 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
docs/Makefile Normal file
View File

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

6
docs/collections.rst Normal file
View File

@ -0,0 +1,6 @@
Collections
===========
.. automodule:: objtools.collections
:members:
:undoc-members:

72
docs/conf.py Normal file
View File

@ -0,0 +1,72 @@
from pathlib import Path
import sys
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
BASE_DIR = Path(__file__).resolve().parent
sys.path.insert(0, BASE_DIR / '..')
# -- Project information -----------------------------------------------------
project = 'objtools'
copyright = '2019, Lucidiot'
author = 'Lucidiot'
# The full version, including alpha/beta/rc tags
with (BASE_DIR / '..' / 'VERSION').open() as f:
release = f.read().strip()
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.coverage',
'sphinx.ext.viewcode',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
html_theme_options = {
'description': 'Common patterns for my Python packages',
'fixed_sidebar': 'true',
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
# -- Extension configuration -------------------------------------------------
# Concatenate the class' and __init__'s docstrings when documenting a class
autoclass_content = 'both'

98
docs/contributing.rst Normal file
View File

@ -0,0 +1,98 @@
Contributing
============
Contributions to the project are greatly appreciated.
Bugs and suggestions
--------------------
You may `submit an issue`_ to GitLab to warn of any bugs, ask for new features,
or ask any questions that are not answered in this documentation.
When reporting a bug, do not forget to put in your version of Python and your
version of *objtools*. This will greatly help when troubleshooting, as most
errors often come from version incompatibilities.
Development
-----------
Setup
^^^^^
You will need a virtual envionment to work properly. `virtualenvwrapper`_ is
recommended::
git clone https://gitlab.com/Lucidiot/objtools
cd objtools
mkvirtualenv -a . objtools
pip install -e .[dev]
This will clone the repository, create a virtual environment named
``objtools``, then tell pip to let the package be editable (``-e``).
The ``[dev]`` suffix adds the extra requirements useful for development.
Unit tests
^^^^^^^^^^
Unit tests use the standard ``unittest`` package; you may run them using the
standard ``setup.py`` command::
./setup.py test
Tests coverage
^^^^^^^^^^^^^^
I aim for 100% coverage on all of my Python packages whenever I add unit
tests to them; this package is no exception. CI checks use the `coverage`_
Python package and `codecov`_ to check for test coverage. To get test coverage
data locally, run::
coverage run setup.py test
You may then get a short coverage summary in your terminal::
coverage report
Or generate an HTML report in a ``htmlcov`` folder, which can be browsed
offline using your favorite web browser and shows line by line coverage::
coverage html
If you are having issues reaching 100% coverage, try to still add some tests,
and mention your issues when creating a pull request to the
`GitLab repository`_.
Linting
^^^^^^^
The source code follows the PEP 8 code style and performs CI checks using the
``flake8`` tool. To perform the same checks locally, run ``flake8`` on the root
directory of this repository.
Static typing
^^^^^^^^^^^^^
This package makes use of the standard `typing`_ module to include PEP 484
type annotations. Type checking is done using the ``mypy`` tool and everything
in this package should be typed; this allows other packages to use *objtools*
and use static typing themselves or benefit from the enhanced documentations
or IDE warnings. To run type checking locally, run ``mypy`` on the root
directory of the repository.
Documentation
-------------
The documentation you are reading is generated by the `Sphinx`_ tool.
The text files that hold the documentation's contents are written in
`reStructuredText`_ and are available under the ``/docs`` folder of the
`GitLab repository`_.
They are also subject to linting using the ``doc8`` tool.
.. _submit an issue: https://gitlab.com/Lucidiot/objtools/issues/new
.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.io
.. _coverage: https://coverage.readthedocs.io/
.. _codecov: https://codecov.io/gl/Lucidiot/objtools
.. _GitLab repository: https://gitlab.com/Lucidiot/objtools
.. _typing: https://docs.python.org/3/library/typing.html
.. _Sphinx: http://www.sphinx-doc.org/
.. _reStructuredText: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

50
docs/index.rst Normal file
View File

@ -0,0 +1,50 @@
objtools
========
:ref:`genindex` - :ref:`modindex` - :ref:`search`
.. image:: https://img.shields.io/pypi/v/objtools.svg
:target: https://pypi.org/project/objtools
.. image:: https://img.shields.io/pypi/l/objtools.svg
:target: https://pypi.org/project/objtools
.. image:: https://img.shields.io/pypi/format/objtools.svg
:target: https://pypi.org/project/objtools
.. image:: https://img.shields.io/pypi/pyversions/objtools.svg
:target: https://pypi.org/project/objtools
.. image:: https://img.shields.io/pypi/status/objtools.svg
:target: https://pypi.org/project/objtools
.. image:: https://gitlab.com/Lucidiot/objtools/badges/master/pipeline.svg
:target: https://gitlab.com/Lucidiot/objtools/pipelines
.. image:: https://requires.io/github/Lucidiot/objtools/requirements.svg?branch=master
:target: https://requires.io/github/Lucidiot/objtools/requirements/?branch=master
.. image:: https://img.shields.io/github/last-commit/Lucidiot/objtools.svg
:target: https://gitlab.com/Lucidiot/objtools/commits
.. image:: https://img.shields.io/badge/badge%20count-9-brightgreen.svg
:target: https://gitlab.com/Lucidiot/objtools
A Python library for common patterns found throughout my other packages.
Installation
------------
This package has a very standard Python setup::
pip install objtools
That's it, nothing more.
Table of contents
-----------------
.. toctree::
collections
registry
contributing

35
docs/make.bat Normal file
View File

@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd

208
docs/registry.rst Normal file
View File

@ -0,0 +1,208 @@
Class registry
==============
Background
----------
I initially developed the class registry for `stdqs`_, a Python package
aiming to add Django QuerySet-like features to any Python iterable. Django
has a notion of *Transforms* and *Lookups*, classes which allow to
transform data (using for example an SQL function like ``LOWER()``) perform
boolean operations on them, respectively. One may create custom transforms
and lookups on a given field using the ``@TheField.register_lookup``
decorator. See the `Django docs`_ to learn more.
My issue with this system is that it requires registering into a *Field*,
which is a notion that does not exist inside *stdqs* as I do not need to
handle persistence or really know much about the type of an attribute myself;
dealing with types is left to the users as most of the standard lookups and
transforms in *stdqs* will simply apply standard Python operators and methods.
Simply using a ``dict`` linking classes to lookup/transform names would
require using ``global`` variables, which is a pattern I really wanted to
avoid, so using a module-level decorator alone would not be enough.
After some thinking, I had build a ``stdqs.lookups.LookupRegistry`` class,
which included a ``register`` method which could be used as a decorator on
classes. I wanted to cleanly separate lookups and transforms, so I also made
a ``stdqs.transforms.TransformRegistry``, which was way too similar to the
first registry to not make them use a common class,
``stdqs.helpers.ClassRegistry``. Both registries included ways to perform
various checks on keys and values of the dictionary, for example to ensure
that a lookup has a valid name so it can be used inside of *stdqs* QuerySets.
Since both lookups and transforms in my project had abstract base classes,
I thought it would be possible to use a custom metaclass to allow for
automatic registration simply by subclassing. I had done something similar
in a very complex way using a metaclass in `PseudoScience`_ for the unit
system a while ago.
This, and other common patterns explained elsewhere, led me to create this
Python library.
Usage
-----
The :class:`ClassRegistry <objtools.registry.ClassRegistry>` provides a
generic answer to the problem of automatic registration of classes with names
in a way that allows easy extension by third parties: merely importing a
module which uses the registry's metaclass will auto-register everything in
this module. The registry itself subclasses ``dict``, therefore has all of its
usual methods.
The most basic usage of the registry is as follows:
.. code:: python
>>> from objtools.registry import ClassRegistry
>>> registry = ClassRegistry()
>>> class RegisteredClass(metaclass=registry.metaclass): ...
>>> dict(registry)
{'RegisteredClass': __main__.RegisteredClass}
The :meth:`metaclass <objtools.registry.ClassRegistry.metaclass>` property
provides a metaclass which performs auto-registration on the current
registry instance. Two keyword arguments may optionally be sent to this
metaclass:
**register** (bool)
When True, will perform auto-registration. Defaults to True.
**key** (str or None)
If defined, will override the default name when registering.
If left unspecified, will use the class' ``__name__``.
.. code:: python
>>> class NotRegisteredClass(
... metaclass=registry.metaclass,
... register=False): ...
>>> class RenamedClass(metaclass=registry.metaclass, key='foo'): ...
>>> dict(registry)
{'RegisteredClass': __main__.RegisteredClass, 'foo': __main__.RenamedClass}
Manual registration and unregistration can also be performed using the
:meth:`register <objtools.registry.ClassRegistry.register>` and
:meth:`unregister <objtools.registry.ClassRegistry.unregister>` methods.
These are aliases to the standard ``__setitem__`` and ``__delitem__`` dict
methods.
Custom checks
^^^^^^^^^^^^^
The default implementation allows registering any class with any name, but
provides overridable methods to perform custom checks upon registration:
:meth:`check_key <objtools.registry.ClassRegistry.check_key>` and
:meth:`check_value <objtools.registry.ClassRegistry.check_value>`. Any
exception raised in those methods will be turned into :exc:`KeyError` or
:exc:`ValueError`, respectively. This allows, for example, to check for
specific attributes, or subclassing an abstract class
.. code:: python
>>> class MyClass(object): ...
>>> class CustomRegistry(ClassRegistry):
... def check_key(self, key):
... assert key.lower(), 'Key must be lowercase'
...
... def check_value(self, value):
... assert issubclass(value, MyClass), 'Must subclass MyClass'
...
>>> registry = CustomRegistry()
>>> class UppercasedKeyClass(metaclass=registry.metaclass, key='FOO'): ...
Traceback (most recent call last):
...
KeyError: Key must be lowercase
>>> class NotMyClass(metaclass=registry.metaclass): ...
Traceback (most recent call last):
...
ValueError: Must subclass MyClass
Cleaner code
^^^^^^^^^^^^
I suggest using the following pattern to make using auto-registration cleaner
in other parts of your code:
.. code:: python
from objtools.registry import ClassRegistry
class CustomRegistry(ClassRegistry):
pass
registry = CustomRegistry()
register = registry.register
unregister = registry.unregister
class BaseClass(metaclass=registry.metaclass, register=False):
pass
This will then allow other modules to import ``yourpkg.yourmodule.BaseClass``
and use it in a perfectly normal way, making users who do not have to use the
registry directly not have to deal with anything relating to it. Note that
you may need to explain in a documentation of some sort the ``register`` and
``key`` keyword arguments.
Working with ABCs or another metaclass
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python's ``abc`` module uses the ``abc.ABCMeta`` metaclass to perform checks
on subclasses and prevent creating instances of subclasses which do not
implement all abstract methods or properties, and has an ``abc.ABC`` class
which does not do much more than having ``ABCMeta`` as its metaclass.
Using the above suggested pattern, a class which uses the metaclass but also
needs to be an abstract base class, combined with a custom check on the
registry itself to ensure all registered classes subclass it, will help
ensure all registered classes provde a common set of methods and properties.
.. code:: python
from abc import ABC
from objtools.registry import ClassRegistry
class CustomRegistry(ClassRegistry):
# ... custom checks etc. ...
pass
registry = CustomRegistry()
class BaseClass(ABC, metaclass=registry.metaclass, register=False):
pass
However, since ``ABC`` uses a custom metaclass itself, creating such an
abstract class will result in a cryptic exception::
TypeError: metaclass conflict: the metaclass of a derived class must be
a (non-strict) subclass of the metaclasses of all its bases
The root cause of this issue is that
:meth:`ClassRegistry.metaclass <objtools.registry.ClassRegistry.metaclass>`
does not inherit from ``abc.ABCMeta``. One could override this property to
provide a custom metaclass which inherits from it, but having to rewrite the
whole metaclass property — which is probably the strangest code of the
:class:`ClassRegistry <objtools.registry.ClassRegistry>` — is not easy or
future-proof. Instead, creating a class which subclasses both metaclasses
solves this issue easily:
.. code:: python
from abc import ABCMeta
# Registry code here
class BaseClassMetaclass(registry.metaclass, ABCMeta):
# Note that we do not use metaclass=... here
pass
class BaseClass(metaclass=BaseClassMetaclass, register=False):
pass
Reference
---------
.. automodule:: objtools.registry
:members:
.. _stdqs: https://gitlab.com/Lucidiot/stdqs
.. _Django docs: https://docs.djangoproject.com/en/dev/howto/custom-lookups/
.. _PseudoScience: https://gitlab.com/Lucidiot/PseudoScience

45
objtools/collections.py
View File

@ -1,3 +1,8 @@
"""
This module provides helpers for working with collections (as in abstract
base classes defined in the ``collections.abc`` standard Python module:
iterables, iterators, mappings, etc.).
"""
from typing import (
Mapping, MutableMapping, Sequence, Set,
Iterable, Iterator, ByteString, Any,
@ -5,6 +10,22 @@ from typing import (
def namespacify(value: Any) -> Any:
"""
Turn all mappings into :class:`Namespace` instances as needed.
:param value: Anything that may be a mapping or may be an iterable
that may hold a mapping.
:returns:
* The value itself, if it is a Namespace instance or is neither
a mapping, a sequence, a set, or an iterable;
* A :class:`Namespace` instance for a mapping;
* A list holding the result of calling :meth:`namespacify` on
every item for a sequence;
* A set holding the result of calling :meth:`namespacify` on
every item for a set;
* A ``map`` object applying :meth:`namespacify` on every item
for other iterables.
"""
if isinstance(value, Namespace):
return value
if isinstance(value, Mapping):
@ -22,8 +43,29 @@ def namespacify(value: Any) -> Any:
class Namespace(MutableMapping):
"""
A class that maps items of a mapping to attributes. Takes the same
arguments as a classic ``dict``. Attributes and items are kept in sync;
deleting an item deletes the attribute, and updating an item updates the
attribute, and vice-versa.
Getting an attribute will call :meth:`namespacify` on the returned value,
to allow easier manipulation of complex structures such as JSON documents:
.. code:: python
>>> from objtools.collections import Namespace
>>> ns = Namespace({'foo': [{'bar': {'baz': 42}}]})
>>> ns.foo[0].bar.baz
42
"""
def __init__(self, *args: Iterable, **kwargs: Any):
"""
:param \\*args: Mappings or iterables yielding
``(key, value)`` two-tuples.
:param \\**kwargs: Key-value pairs set as attributes.
"""
for iterable in (*args, kwargs):
if isinstance(iterable, Mapping):
iterable = iterable.items()
@ -68,4 +110,7 @@ class Namespace(MutableMapping):
p.text(')')
def copy(self) -> 'Namespace':
"""
Create a shallow copy of this namespace.
"""
return self.__class__(self.__dict__)

53
objtools/registry.py
View File

@ -2,14 +2,49 @@ from typing import Dict, Optional, Any, Callable
class ClassRegistry(Dict[str, Callable]):
"""
Subclass of ``dict`` to map names to classes, with customizable checks
and a custom metaclass for easy auto-registration of classes.
"""
def check_key(self, key: str) -> None:
pass
"""
Perform checks on a given key. Does nothing by default;
is available for easy customization of the registry.
Does not return any meaningful value; not raising an exception should
be considered a successful check.
:param str key: Key of a class being registered.
"""
def check_value(self, value: Callable) -> None:
pass
"""
Perform checks on a given class. Does nothing by default;
is available for easy customization of the registry.
Does not return any meaningful value; not raising an exception should
be considered a successful check.
:param Callable value: A class being registered.
"""
def check(self, key: str, value: Callable) -> None:
"""
Called before any registration; performs optional checks and raises
exceptions. The default implementation calls :meth:`check_key`,
raising :exc:`KeyError` for any exceptions raised by this method,
then calls :meth:`check_value`, raising :exc:`ValueError` for any
exceptions raised by this method.
Does not return any meaningful value; not raising an exception should
be considered a successful check.
:param str key: Key of a class being registered.
:param Callable value: Class being registered.
:raises KeyError: When :meth:`check_key` raises any exception.
:raises ValueError: When :meth:`check_value` raises any exception.
"""
try:
self.check_key(key)
except KeyError:
@ -25,9 +60,20 @@ class ClassRegistry(Dict[str, Callable]):
raise ValueError(str(e))
def register(self, key: str, value: Callable) -> None:
"""
Register a new class. Alias to ``registry[key] = value``.
:param str key: Key to register the class under.
:param Callable value: Class to register.
"""
self[key] = value
def unregister(self, key: str) -> None:
"""
Unregister a class. Alias to ``del registry[key]``.
:param str key: Key of a registered class.
"""
del self[key]
def __setitem__(self, key: str, value: Callable) -> None:
@ -45,6 +91,9 @@ class ClassRegistry(Dict[str, Callable]):
@property
def metaclass(self) -> type:
"""
A custom metaclass which performs auto-registration of any subclass.
"""
class RegistryMetaclass(type):
def __new__(cls,