Topic: This post describes a data pipeline for a machine learning task of interest in high energy physics: building a particle cla

Topic: this post is about a simple implementation with examples of IPython custom magic functions for running SQL in Apache Spark usin

Introduction

In the part 2 of 'Integrating Hadoop and Elasticsearch' blogpost series we look at bridging Apache Spark and Elasticsearch. I assume that you have access to Hadoop and Elasticsearch clusters and you are faced with the challenge of bridging these two distributed systems. As spark code can be written in scala, python and java, we look at the setup, configuration and code snippets across all these three languages both in batch and interactively.