“If you’ve collected a large amount of data that you want to analyze, the go-to method for years has been to follow a programming paradigm called MapReduce, typically using Apache’s Hadoop framework. It’s a tried and true process, but it isn’t simple: Hadoop, which is mostly written in Java, has a reputation for being difficult.
Companies that want to get serious about data analysis often have to hire elite programmers who specialize in writing Hadoop MapReduce jobs. Or, they could contract a third-party company such as Cloudera to facilitate this kind of analysis. Neither of these options are an easy or inexpensive undertaking. This all means that early stage companies or projects often just don’t have the resources or know-how to take advantage of “big data.”
Pachyderm is a new startup launching out of the Winter 2015 class of Y Combinator that aims to make big data analysis much simpler and accessible. Claiming to provide the power of MapReduce without the complexity of Hadoop, Pachyderm is an open source tool that purports to enable programmers to run analysis of large amounts of data without writing a line of Java or knowing a thing about how MapReduce works.”
