Chapter 1: Introduction to SparkChapter Goal: Introduce the reader to Spark in general. This book does not assume that the reader is already familiar with Spark. Sub -Topics Introduction to Spark and its key selling points The programming model Architecture Introduction to other systems within the ecosystem, such as MLlib, GraphX, SparkSQL, and SparkR Chapter 2: Spark StreamingChapter Goal: Introduces Spark Streaming and the concept of micro batch processing (DStreams) Sub - Topics Introduction to Spark Streaming/DStreams Comparison with traditional stream processing How Spark Streaming works under the hood Programming API and how it relates to the general Spark API First sample application using FileInputDStream Chapter 3: Best PracticesChapter Goal: To transfer best practices in terms of application development Sub - Topics: Maintaining state in an application Data caching to reduce redundant work Offloading RDD maintenance to Tachyon Fault-tolerance and check-pointing Chapter 4: Ingesting data from external data sourcesChapter Goal: To enable the reader to understand the various data ingestion options, their pros and cons, and their integration with Spark Streaming Sub - Topics: 1. Introduction to Receivers 2. Kafka 3. Twitter 4. Flume 5. Other sources 6. Writing your own connector. Example Apache Qpid Chapter 5: Optimizing and maintaining a Spark Streaming application/deploymentChapter Goal: To help the user in optimizing an application and how it can be maintained in production Sub - Topics Different configuration parameters and how they affect the application Parallelism Serialization, memory, etc. enhancements Various monitoring and instrumentation options Chapter 6: Spark Streaming, SQL, and RChapter Goal: To illustrate how a SQL/Dataframe interface can simplify common transforms Introduction to SparkSQL and SQLContext Various SQL constructs Integration with R Design of a few real-world applications Chapter 7: Streaming Machine LearningChapter Goal: Employ MLlib to implement streaming machine learning applications Introduction to streaming algorithms in MLlib Real-world applications using streaming MLlib Chapter 8: Lambda Architecture using SparkChapter Goal: Blending data at rest with data in motion Introduction to the Lambda Architecture Design of Lambda Architecture using Spark Chapter 9: Java and Python APIs for Spark StreamingChapter Goal: Introduction to Spark Streaming in Java and Python Java API Python API Chapter 10: Spark Streaming and BeyondChapter Goal: Overview of some of the future plans for Spark Streaming from the ope n source community Project Tungsten and how its CPU and memory improvements can benefit streaming applications Links to useful resources
Zubair Nabi is one of the very few computer scientists who have solved Big Data problems in all three domains: academia, research, and industry. He currently works at Qubit, a London-based start up backed by Goldman Sachs, Accel Partners, Salesforce Ventures, and Balderton Capital. Qubit helps retailers understand their customers and provide personalized customer experience, and which has a rapidly growing client base that includes Staples, Emirates, Thomas Cook, and Topshop. Prior to Qubit, he was a researcher at IBM Research, where he worked at the intersection of Big Data systems and analytics to solve real-world problems in the telecommunication, electricity, and urban dynamics space. Zubair's work has been featured in MIT Technology Review, SciDev, CNET, and Asian Scientist, and on Swedish National Radio, among others. He has authored more than 20 research papers, published by some of the top publication venues in computer science including USENIX Middleware, ECML PKDD, and IEEE BigData; and he also has a number of patents to his credit. Zubair has an MPhil in computer science with distinction from Cambridge.