5. New Backends

5.1 About

You can define your own backends in cl-store to do custom object I/O. Theoretically one can add a backend that can do socket based communication with any language provided you know the correct format to output objects in. If the framework is not sufficient to add your own backend just drop me a line and we will see what we can do about it.

5.2 The Process

5.2.1 Add the backend

Use defbackend to define the new backend choosing the output format, an optional magic number, extra fields for the backend and a backend to extend which defaults to the base backend. eg. (from the cl-store-backend)

(defbackend cl-store :magic-number 1347643724
                :stream-type '(unsigned-byte 8)
                :old-magic-numbers (1912923 1886611788 1347635532)
                :extends resolving-backend
                :fields ((restorers :accessor restorers :initform (make-hash-table))))

5.2.2 Recognizing Objects.

Decide how to recognize objects on restoration. When restoring objects the backend has a responsibility to return a symbol identifying the defrestore method to call by overriding the get-next-reader method. In the cl-store backend this is done by keeping a mapping of type codes to symbols. When storing an object the type code is written down the stream first and then the restoring details for that particular object. The get-next-reader method is then specialized to read the type code and look up the symbol in a hash-table kept on the backend.

eg. (from the cl-store-backend)

(defvar *cl-store-backend* (find-backend 'cl-store))
;; This is a util method to register the code with a symbol
(defun register-code (code name &optional (errorp t))
  (aif (and (gethash code (restorers *cl-store-backend*)) errorp)
       (error "Code ~A is already defined for ~A." code name)
       (setf (gethash code (restorers *cl-store-backend*))
             name))
  code)
;; An example of registering the code 7 with ratio
(defconstant +ratio-code+ (register-code 7 'ratio))

;; Extending the get-next-reader method
(defmethod get-next-reader ((backend cl-store) (stream stream))
  (let ((type-code (read-type-code stream)))
    (or (gethash type-code (restorers backend))
        (values nil (format nil "Type ~A" type-code)))))

(defstore-cl-store (obj ratio stream)
  (output-type-code +ratio-code+ stream) ;; output the type code
  (store-object (numerator obj) stream)
  (store-object (denominator obj) stream))

5.2.3 Extending the Resolving backend

If you are extending the resolving-backend you have a couple of extra responsibilities to ensure that circular references are resolved correctly. Store-referrer must be extended for your backend to output the referrer code. This must be done as if it were a defstore for a referrer. A defrestore-<backend-name> must also be defined for the referrer which must return a referrer created with make-referrer. Once that is done you can use resolving-object and setting to resolve circularities in objects.

eg (from the cl-store backend)

(defconstant +referrer-code+ (register-code 1 'referrer nil))
(defmethod store-referrer (ref stream (backend cl-store))
  (output-type-code +referrer-code+ stream)
  (store-32-bit ref stream))

(defrestore-cl-store (referrer stream)
  (make-referrer :val (read-32-bit stream nil)))

5.3 Example: Simple Pickle Format

As a short example we will define a backend that can handle simple objects using the python pickle format.

5.3.1 Define the backend

(in-package :cl-user)
(use-package :cl-store)

(defbackend pickle :stream-type 'character)

5.3.2 Recognize Objects

This is just a simple example to be able to handle single strings stored with Python's pickle module.

(defvar *pickle-mapping*
  '((#\S . string)))

(defmethod get-next-reader ((backend pickle) (stream stream))
  (let ((type-code (read-char stream)))
    (or (cdr (assoc type-code *pickle-mapping*))
        (values nil (format nil "Type ~A" type-code)))))

(defrestore-pickle (noop stream))

(defstore-pickle (obj string stream)
  (format stream "S'~A'~%p0~%." obj))

(defrestore-pickle (string stream)
  (let ((val (read-line stream)))
    (read-line stream) ;; remove the PUSH op
    (read-line stream) ;; remove the END op
    (subseq val 1 (1- (length val)))))

5.3.3 Test the new Backend.

This can be tested with the code

Python
>>> import pickle
>>> pickle.dump('Foobar', open('/tmp/foo.p', 'w'))

Lisp
* (cl-store:restore "/tmp/foo.p" 'pickle)
=> "Foobar"
And 

Lisp 
* (cl-store:store "BarFoo" "/tmp/foo.p" 'pickle)

Python
>>> pickle.load(open('/tmp/foo.p'))
'BarFoo'

5.4 API

5.4.1 Functions

Generic: backend-restore backend place

Restore the object found in stream place using backend backend. Checks the magic-number and invokes backend-restore-object. Called by restore, override for custom restoring.

Generic: backend-restore backend place

Find the next function to call to restore the next object with backend and invoke it with place. Called by restore-object, override this method to do custom restoring (see `circularities.lisp' for an example).

Generic: backend-store backend place obj

Stores the backend code and calls store-object. This is called by store. Override for custom storing.

Generic: backend-store-object backend obj place

Called by store-object, override this to do custom storing (see `circularities.lisp' for an example).

Generic: get-next-reader backend place

Method which must be specialized for backend to return the next symbol designating a defrestore instance to restore an object from place. If no reader is found return a second value which will be included in the error.

5.4.2 Macros

Macro: defbackend name &key (stream-type (required-arg "stream-type")) magic-number fields (extends 'backend) old-magic-numbers

eg. (defbackend pickle :stream-type 'character) This creates a new backend called name, stream-type describes the type of stream that the backend will serialize to which must be suitable as an argument to open. Magic-number, when present, must be of type (unsigned-byte 32) which will be written as a verifier for the backend. Fields are extra fields to be added to the new class which will be created. By default the extends keyword is backend,the root backend, but this can be any legal backend. Old-magic-numbers holds previous magic-numbers that have been used by the backend to identify incompatible versions of objects stored.

This document was generated by Sean on September, 1 2005 using texi2html 1.76.