manardb

Introduction

Manardb provides persistent classes (integrated into the object system via the meta-object protocol), stored efficiently in memory mapped regions. It features transactions, two different garbage collection mechanisms (to clean up unreferenced objects in the datastore), in-place modification of arrays, efficient in-place numeric typed slots, and the ability to transparently serialise Lisp objects of many types (lists, vectors, floats, integers, symbols, strings).

It allows Common Lisp programs to efficiently access in-memory representations of large numbers of persistent objects without putting pressure on the Lisp garbage collector.

Concurrency

Multiple concurrent readers and writers can access the database without problems; both multiple threads and multiple processes work fine. However, they have to either arrange not allocate new objects (by using fixed-size numeric-typed slots, and fixed-length strings) or ensure via some locking mechanism that only one thread can allocate at a time, and all processes update their mappings after an allocation. Additionally, they must ensure that writers do not conflict together or confuse readers.

Alternatively, they could use the race free transaction support. (Readers continue to refer to an old snapshot until they explicitly reload the database.)

Efficiency

According to our benchmarks, when instantiating a object with two slots one million times, manardb is about seven times faster than AllegroCache 2.1.11 on Allegro 8.1, and about fifteen times faster than AllegroCache on SBCL 1.0.31. It is more than ten times faster to iterate over the 1M objects created and sum the values of one numeric slot on Allegro Lisp, and about one hundred times faster on SBCL.

Portability

Manardb works on Common Lisp implementations on Linux. We have tested SBCL 1.0.31 and Allegro 8.1, and lightly tested Lispworks and ClozureCL.

It uses the mremap system call so is restricted to Linux. It would be trivial to change the mremap to a munmap, followed by mmap on other UN*X like platforms, as manardb is already prepared for the base address of the mapping to change.

Using it

Start Lisp. Get asdf-install.

      CL-USER> (asdf-install:install 'manardb)
      CL-USER> (asdf:operate 'asdf:load-op 'manardb)

CL-USER> (defclass foo ()
  ((next :initform nil)
   (val :initarg :val :accessor foo-val :type (unsigned-byte 32)))
  (:metaclass manardb:mm-metaclass))
#<MM-METACLASS FOO>

The API is documented.

Downloads

Git development tree access

git clone http://cl-www.msi.co.jp/projects/manardb/manardb.git

Future directions

manardb is just starting out. It hasn't tapped the full potential of this approach. There are many ways to improve it.

Indexing
Storage of more complex types: union types, enumerated types, variable length objects, interface to the ability to store directly store arrays of instances of user-defined class (instead of an array of pointers to such instances), etc.
Storage of more Lisp types: multidimensional arrays, complex numbers, bignums, characters, etc.
More transparent shared access to arrays of unboxed types.
Many MOP optimisations: generating custom accessor methods for memory mapped slots.
Improve the whole-datastore garbage collector to use more memory efficient datastructures.