Fast serialization library for Clojure
Clojure's rich data types are awesome. And its reader allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
Nippy is a mature, high-performance drop-in alternative to the reader.
It is used at scale by Carmine, Faraday, PigPen, Onyx, XTDB, Datalevin, and others.
Why Nippy?
- Small, simple pure-Clojure library
- Terrific performance: the best for Clojure that I'm aware of
- Comprehensive support for all standard data types
- Easily extendable to custom data types
- Robust test suite incl. coverage of every supported type
- Mature and widely used in production for 12+ years
- Optional auto fallback to Java Serializable for safe types
- Optional auto fallback to Clojure Reader (including tagged literals)
- Optional smart compression with LZ4 or Zstandard
- Optional encryption with AES128
- Tools for easy + robust integration into 3rd-party libraries, etc.
- Powerful thaw transducer for flexible data inspection and transformation
This story begins, as they so often do, when I noticed that my machine was behaving poorly. My Windows 10 work machine has 24 cores (48 hyper-threads) and they were 50% idle. It has 64 GB of RAM and that was less than half used. It has a fast SSD that was mostly idle. And yet, as I moved the mouse around it kept hitching – sometimes locking up for seconds at a time.
So I did what I always do – I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10.