Hadoop: Divide and Conquer Gigantic Datasets (Intro)

Moore's law has finally hit the wall and CPU speeds have actually decreased in the last few years. The industry is reacting with hardware with an ever-growing number of cores and software that can leverage “grids” of distributed, often commodity, computing resources. But how is a traditional Java developer supposed to easily take advantage of this revolution? The answer is the Apache Hadoop family of projects. Hadoop is a suite of Open Source APIs at the forefront of this grid computing revolution and is considered the absolute gold standard for the divide-and-conquer model of distributed problem crunching. The well-travelled Apache Hadoop framework is curently being leveraged in production by prominent names such as Yahoo, IBM, Amazon, Adobe, AOL, Facebook and Hulu just to name a few.

In this session, you'll start by learning the vocabulary unique to the distributed computing space. Next, we'll discover how to shape a problem and processing to fit the Hadoop MapReduce framework. We'll then examine the incredible auto-replicating, redundant and self-healing HDFS filesystem. Finally, we'll fire up several Hadoop nodes and watch our calculation process get devoured live by our Hadoop cluster. At this talk's conclusion, you'll understand the suite of Hadoop tools and where each one fits in the aim of conquering large data sets.

About Matthew McCullough

Matthew McCullough is an energetic 15 year veteran of enterprise software development, open source education, and co-founder of Ambient Ideas, LLC, a Denver consultancy. Matthew currently is VP of Training at GitHub.com, author of the Git Master Class series for O'Reilly, speaker at over 30 national and international conferences, author of three of the top 10 DZone RefCards, and President of the Denver Open Source Users Group. His current topics of research center around project automation: build tools (Gradle), distributed version control (Git, GitHub), Continuous Integration (Jenkins, Travis) and Quality Metrics (Sonar). Matthew resides in Denver, Colorado with his beautiful wife and two young daughters, who are active in nearly every outdoor activity Colorado has to offer.

More About Matthew »