Cascading through Hadoop: A DSL for Simpler MapReduce

Hadoop is a MapReduce framework that has literally sprung into the vernacular of “big data” developers everywhere. But coding to the raw Hadoop APIs can be a real chore. Data analysts can express what they want in more English-like vocabularies, but it seems the Hadoop APIs require us to be the translator to a less comprehensible functional and data-centric DSL.

The Cascading framework gives developers a convenient higher level abstraction for querying and scheduling complex jobs on a Hadoop cluster. Programmers can think more holistically about the questions being asked of the data and the flow that such data will take without concern for the minutia.

We'll explore how to set up, code to, and leverage the Cascading API on top of a Hadoop sample or production cluster for a more effective way to code MapReduce applications all while being able to think in a more natural (less than fully MapReduce) way.

During this presentation, we'll also explore Cascading's Clojure-based derivative, Cascalog, and how functional programming paradigms and language syntax are emerging as the next important step in big-data thinking and processing.

About Matthew McCullough

Matthew McCullough is an energetic 15 year veteran of enterprise software development, open source education, and co-founder of Ambient Ideas, LLC, a Denver consultancy. Matthew currently is VP of Training at GitHub.com, author of the Git Master Class series for O'Reilly, speaker at over 30 national and international conferences, author of three of the top 10 DZone RefCards, and President of the Denver Open Source Users Group. His current topics of research center around project automation: build tools (Gradle), distributed version control (Git, GitHub), Continuous Integration (Jenkins, Travis) and Quality Metrics (Sonar). Matthew resides in Denver, Colorado with his beautiful wife and two young daughters, who are active in nearly every outdoor activity Colorado has to offer.

More About Matthew »