Converging Big Data And Web Applications

About this Presentation

For most of my lifetime in the computing world, data crunching and web serving were two very separate worlds. If a web app wanted access to the analysis there was a long process of ETL, DB work, imports and exports, and bribing various network and storage people for the resources you needed. With the rise of containers, orchestration, cheap computing and networking, and over 10 years of people tackling large problems at new scales we have finally come to a convergence.

It is now possible for us to actually use the same hardware, and more importantly, clustering software to converge both types of workloads. We are going to lay out a high level vision of how this can look with Containers, Kubernetes, web servers, and Apache Spark. This can be considered a germ of what we can look to build in the future. We will demo this in action and show this is actually now achievable for mere mortals such as us. This convergence allows data analysis move from custom R or Python scripts on an analyst’s desktop to a front-line accessible web app. At the same time, the analyst can “constrain” the analysis to prevent “statistical overreach”.