While there’s no shortage of libraries that let one interact with Spark using Clojure they treat the problem as providing a more fluid experience of using Java interfaces. In doing so they overlook the REPL experience (by relying on AOT) and create code duplication (having their own fn, map, reduce, filter, mapcat…). But there is another way: ignore most of the Java/Scala interfaces and consider Spark artifacts (RDDs and streams) as a transducing context. Through doing this we have achieved a true REPL experience, a small API and overall made interactions with Spark more Clojuresque.
Igor, has been trying to bring better technology to healthcare for almost a decade; currently using Clojure to deliver data and analytics products at HCA.
Christophe, an independent software engineer and trainer, is a clojurist of the early days. He coauthored Clojure Programming.