API Reference

indicium.base

class indicium.base.BytesSerializer(store: indicium.base.Store)

A simple serializer which ensures values are of type bytes.

This serializer is mostly useful to ensure that values passed down to a FSStore are of type bytes.

class indicium.base.Cache(store: indicium.base.Store, cache: indicium.base.Store)

Wraps a backend store with a caching store.

Fetching an element will try to fetch the element from the cache first. In case of a cache miss (the element is not found in the cache), the element is searched for in the backend and, if found, added to the cache.

Storing or deleting an element will perform the operation in both the cache and backend stores.

Queries are always delegated to the backend store, as it is assumed that cache stores may not contain the all the information needed to perform queries—or they may not support querying at all.

Parameters:
  • store (Store) – Store used as backend.
  • cache (Store) – Store used as cache.

Properties:

cache

Store used as cache.

child

Store used as backend.

class indicium.base.DictStore

Simple in-memory store using a dictionary as backend.

All the elements are kept in memory, and are naïvely stored in a single dictionary. This store should be used for testing and when handling small amounts of transient data.

class indicium.base.NullStore

A sinkhole key-value store.

Stored values are discarded by this store. Mostly useful for testing.

class indicium.base.Serializer(store, serializer)

Generic (de)serializer for pickle-style (de)serializer.

This can be used to wrap a store and perform (de)serialization using any object which has a pair of loads and dumps. function attributes—like the pickle module from the Python standard librart does:

>>> import pickle
>>> store = Serializer(DictStore(), pickle)

The following are known to work with Serializer are:

  • JSON using Python’s built-in json module.
  • HiPack, using the hipack module from hipack-python.
Parameters:
  • store (Store) – A store to be wrapped.
  • serializer – An object with attributes loads and dumps with the same API as those from the pickle module.
class indicium.base.SerializerShim(store: indicium.base.Store)

Abstract class which defines the (de)serialization protocol.

dumps(value)

Serializes a value.

Parameters:value – Deserialized value.
Returns:Serialized value.
loads(value)

Deserializes a value.

Parameters:value – Serialized value.
Returns:Deserialized value.
class indicium.base.Shim(store: indicium.base.Store)

Wraps a Store and delegates all operations to it.

This can be used as a base class for more complex wrappers, without needing to reimplement all the abstract methods of Store.

Parameters:store (Store) – The wrapped store object.
child

The wrapped Store instance. Changing the value of the attribute after instantiation is discouraged.

class indicium.base.Store

A Store is the interface to a key-value storage system.

contains(key)

Returns whether the object named by key exists.

The default implementation uses get() to determine the existence of values, therefore store implementations may want to provide a specialized version of this method.

Parameters:key (str) – A key.
delete(key)

Removes the object named by key.

Store implementations must implement this method.

Parameters:key (str) – A key.
get(key)

Return the object named by key or None if it does not exist.

Store implementations must implement this method.

Parameters:key (str) – A key.
put(key, value)

Stores the object value named by key.

Store implementations must implement this method.

Parameters:
  • key (str) – A key.
  • value – Value associated to the key.
query(pattern, limit=None, offset=0)

Iterate over the key-value pairs of keys matching a fnmatch pattern.

The pattern is a fnmatch-style pattern, and it is used to filter elements by matching their keys. specifiers:

The main use-case for pattern specifiers is enumerating and retrieving sets of values which share a common key prefix. This is a common pattern: related values are all stored using keys with a common prefix, for example where each user of a system has /user/ at the beginning of its key, then one can enumerate all the users with the query pattern /user/*.

Parameters:
  • pattern (str) – A fnmatch-style pattern, as a string.
  • limit (int) – Maximum number of elements returned by the query. Using None returns all the matched elements.
  • offset (int) – Index of the first element to return.
Returns:

Iterable which yields (key, value) pairs.

indicium.base.query_iterable(iterable, pattern, limit=None, offset=0)

Takes an iterable over key-value pairs and applies query filtering.

Parameters:
  • iterable – An iterable which yields (key, value) pairs.
  • pattern (str) – A fnmatch-style pattern.
  • limit (int) – Maximum number of result elements.
  • offset (int) – Index of the first result element.

indicium.cache

class indicium.cache.LRUCache(store: indicium.base.Store, size: int)

Provides a LRU cache over a store.

This class is provided for convenience, as:

>>> store = LRUCache(FSStore("./data"), size=1000)

is equivalent to:

>>> store = Cache(FSStore("./data"), LRUCacheStore(size=1000))
Parameters:
  • store (Store) – Backend store.
  • size (int) – Maximum number of elements held by the LRU cache.
class indicium.cache.LRUCacheStore(size)

A least-recently-used (LRU) cache store.

This store holds in memory up to a maximum amount of (key, value) pairs (the size of the cache). When then store is full an more values are to be stored, the elements which were accessed most recently (using either get() or put()) are preserved, and the least-recently used (LRU) element is deleted.

Typically this store is used in a base.Cache() combined with a slower store: the LRUCacheStore provides fast access to the elements used more often, while the slower store ensures that all data is persisted. For example, to put a cache in front of a filesystem-based store (FSStore):

>>> store = Cache(FSStore("./data"), LRUCacheStore(size=100))

Note that LRUCache can be used for the above use-case.

Parameters:size (int) – Maximum number of elements held by the store.

Properties:

size

Size of the cache. The maximum number of elements that the cache can hold can be changed by setting this property.

indicium.fs

class indicium.fs.FSStore(path='.', extension='.data')

File system storage using directories and flat files.

Each value is stored in its own file, in a subdirectory of the store’s root path derived from the key. The extension of Data files can be configured as well.

Note that values are written directly to files, so they must be bytes objects. Most of the time a SerializerShim subclass will be used to wrap the store, allowing for storing Python values directly. For example, the following creates a store which saves values as JSON in plain text files:

>>> from indicium import base, fs
>>> import json, tempfile, shutil
>>> datadir = tempfile.mkdtemp()
>>> store = base.Serializer(fs.FSStore(datadir, ".json"), json)

Values can then be stored directly:

>>> store.put("/user/jdoe", { "name": "John Doe" })
>>> store.contains("/user/jdoe")
True
>>> store.get("/user/jdoe")
{"name": "John Doe"}
>>> store = None; shutil.rmtree(datadir)
Parameters:
  • path (str) – Path to a directory to be used as root.
  • extension (str) – File suffix used for data files.

Properties:

extension

Extension of data files.

path

Path of root directory.

indicium.key

Utility functions to deal with store keys.

indicium.key.join(components, *arg)

Joins path components to form a normalized key.

>>> join("foo", "bar", "baz")
"/foo/bar/baz"
>>> join(["foo", "bar"], "baz")
"/foo/bar/baz"
Parameters:
  • components – An iterable value of path components, or a string (which will be passed to split() to turn it into an iterable value).
  • arg – Additional path components to be appended to the generated key.
Returns:

Normalized key, as a string.

indicium.key.normalize(key)

Normalizes a key, removing duplicate slashes.

This function is equivalent to join(split(key)), just slightly faster.

>>> normalize("foo//bar///baz/")
"/foo/bar/baz"
Parameters:key – A key, as a string.
Returns:Normalized key, as a string.
indicium.key.split(key)

Splits a key into its path components.

Note that this function removes duplicate slashes.

>>> split("/foo//bar////baz/")
["foo", "bar", "baz"]
Parameters:key – Key as a string.
Returns:List of strings, each one being a path component.