Clojure frustrations

One of the nice things about Clojure is that it comes with a huge set of data structures, all of which can be handled using normal Lisp commands. However things are not as simple as they first seem, and it’s a bit of a dark art as there is limited information online. I’ve just spent a day or so on a frustrating voyage of discovery, so I post this here in the hope of saving someone some time in the future.

Lisp generally allows you to read and write data very simply, there is no need to use a separate parser as you store information in lists – the same as the code, for which you already have a parser.

For example if you have a text file called data.txt containing ‘(“one” “two (“three” 4) 5) all you need to do is call (define d (load “data.txt”)) and it gets parsed for you from it’s text form: (first d) returns “one”. This is more efficient than using another kind of format such as XML which you’d need to run a separate chunk of code for.

One of the clojure data types I decided to use was StructMaps, which you define like this (defstruct mystructfieldone :fieldtwo :fieldthree) : and gets created like this:
(struct mystruct 1 2 “three”) => {:fieldone 1, :fieldtwo 2, :fieldthree “three”}

Clojure also comes with some handy io calls, so you can save the StructMap with (spit (struct mystruct 1 2 “three”) “data.txt”) and you get a file created containing {:fieldone 1, :fieldtwo 2, :fieldthree “three”}. This can be read back in simply by using (read-string (slurp “data.txt”)).

The catch is that the result of read-string is a different type to the object we saved:
(type (read-string (slurp “data.txt”))) => clojure.lang.PersistentArrayMap whereas: (type (struct mystruct 1 2 “three”)) => clojure.lang.PersistentStructMap. Presumably the reader interprets the curly braces as a different type. Annoying, but I then got reading about print-dup, which provides more serialisation information for recreating the data:

(print-dup (struct mystruct 1 2 “three”) *out*) => #=(clojure.lang.PersistentStructMap/create {:fieldone 1, :fieldtwo 2, :fieldthree “three”})nil

This seems to output code that should be able to recreate the StructMap. However, to actually write and read this from disk, we have to call it a bit differently:

(use ‘clojure.contrib.io)

(defn serialise [data-structure filename]
    (with-out-writer
        (java.io.File. filename)
        (binding [*print-dup* true] (prn data-structure))))

(defn deserialise [filename]
    (with-open [r (java.io.PushbackReader. (java.io.FileReader. filename))]
        (read r)))

At this point I was getting a bit frustrated, but felt I was close. Unfortunately, deserialising the file created – I got this message:

java.lang.IllegalArgumentException: No matching method found: create (NO_SOURCE_FILE:1)

Great, so the clojure.lang.PersistentStructMap/create doesn’t actually exist? More searching revealed that this is indeed the case, and is not going to be fixed in the short or medium term. Perhaps we can bump down a level and use some serialisation stuff from java – some googling provided these replacements:

(defn serialise [o filename]
    (with-open [outp (-> (java.io.File. filename) java.io.FileOutputStream. java.io.ObjectOutputStream.)]
        (.writeObject outp o)))

(defn deserialise [filename]
    (with-open [inp (-> (java.io.File. filename) java.io.FileInputStream. java.io.ObjectInputStream.)]
        (.readObject inp)))

These seem to do the trick: (type (deserialise “data.txt”)) => clojure.lang.PersistentStructMap. However data.txt is now a (relatively) large binary file, and not human readable. Not a showstopper, but all the same, not very Lisp like. I’ve been aware all this time that the documentation suggests using Records instead of StructMaps, so I assume these will work better – you create and use them in a similar way:

(defrecord myrecord [fieldone fieldtwo fieldthree])
(myrecord. 1 2 “three”) => #:user.myrecord{:fieldone 1, :fieldtwo 2, :fieldthree “three”}

However, they exhibit all the same problems as ArrayMaps when it comes to serialisation – and although they do work with the last method, they exhibit a further problem. If I change the definition of myrecord and then stream in the file, I get this:

java.io.InvalidClassException: user.myrecord; local class incompatible: stream classdesc serialVersionUID = -791607609539721003, local class serialVersionUID = 4065564404893761810 (NO_SOURCE_FILE:1)

The problem here is that I need to be able to change my structures and then convert older saved data using versioning. I can do this with StructMaps as they are not statically typed, so I can write code to check them on load and modify them to the current version (this was used a lot on NoP to keep the game running all the time with the same data while changing the fundamental types). I think a lot of my problems stem from coming from Scheme, so I’m not really aware of the “Java way”.

The other approach is to just abandon looking for an existing solution and go back to writing per-object versioned serialisation/deserialisation parsing like I do in C++.