Acunu is reengineering the Linux storage stack from the ground-up for Big Data applications. Since our inception, we've been users of OCaml for research, prototyping and production code. We've use OCaml when implementing brand new, cutting edge algorithms[1] in an prototype storage engine, and when producing an enterprise grade, multi-master management stack. I'd like to talk about our experiences (both good and bad). I'd like to discuss why we've moved away from OCaml as our primary prototyping language. We've interfaced with a lot of C code, and I'd like to share with the audience the unwritten rules and techniques we've used when when we encounter hard to debug segfaults. And finally we are just about to launch a product containing 50k+ lines of OCaml in the management stack, so something must have gone well...
The Acunu Management Stack is a disaggregated collection of services running on every host in a cluster, implementing role such as deployment assistance, configuration and monitoring. Each service has a specific role and exposes (a subset of) the same, autogenerated API - for instance, we have a clustering service that continually gossips to discover other hosts and keep an (eventually) consistent view of the cluster. Our JavaScript/HTML UI communicates with these service via a routing service, which exposes the unified API and can route events and forward RPC calls between services on different hosts, based on various policies.
I’d like to talk about our experiences developing this system in OCaml and what specific features of the language we have used. I also like to discuss some of the common patterns which have fallen out of our development, and what libraries we have used (or wish we had used!) Finally (if I get time), I’d like quickly to demonstrate how cool the Acunu Storage Platform is, and what kind of performance improvements it can bring to massive, distributed storage systems.
[1] Byde, A. et al (2011). Stratified B-trees and Versioned Dictionaries. Appearing in HotStorage ’11