Kafka for Application Developers

Chapter 1 - Introduction to Kafka

• Messaging Architectures – What is Messaging?
• Messaging Architectures – Steps to Messaging
• Messaging Architectures – Messaging Models
• What is Kafka?
• What is Kafka? (Contd.)
• Kafka Overview
• Kafka Overview (Contd.)
• Need for Kafka
• When to Use Kafka?
• Kafka Architecture
• Core concepts in Kafka
• Kafka Topic
• Kafka Partitions
• Kafka Producer
• Kafka Consumer
• Kafka Broker
• Kafka Cluster
• Why Kafka Cluster?
• Sample Multi-Broker Cluster
• Overview of ZooKeeper
• Kafka Cluster & ZooKeeper
• Who Uses Kafka?
• Summary

Chapter 2 - Using Apache Kafka

• Installing Apache Kafka
• Configuration Files
• Starting Kafka
• Using Kafka Command Line Client Tools
• Setting up a Multi-Broker Cluster
• Using Multi-Broker Cluster
• Kafka Cluster Planning
• Kafka Cluster Planning – Producer/Consumer Throughput
• Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
• Kafka Cluster Planning – Sizing for Topics and Partitions
• Kafka Cluster Planning – Sizing for Storage
• Kafka Connect
• Kafka Connect – Configuration Files
• Using Kafka Connect to Import/Export Data
• Creating a Spring Boot Producer
• Adding Kafka dependency to pom.xml
• Defining a Spring Boot Service to Send Message(s)
• Defining a Spring Boot Controller
• Testing the Spring Boot Producer
• Creating a Nodejs Consumer
• Summary

Chapter 3 - Building Data Pipelines

• Building Data Pipelines
• What to Consider When Building Data Pipelines
• Timeliness
• Reliability
• High and Varying Throughput
• High and Varying Throughput (Contd.)
• Data Formats
• Data Formats (Contd.)
• Transformations
• Transformations - ELT
• Security
• Failure Handling
• Agility and Coupling
• Ad-hoc Pipelines
• Metadata Loss
• Extreme Processing
• Kafka Connect vs. Producer and Consumer
• Kafka Connect vs. Producer and Consumer (Contd.)
• Summary

Chapter 4 - Integrating Kafka with Other Systems

• Introduction to Kafka Integration
• Kafka Connect
• Kafka Connect (Contd.)
• Running Kafka Connect Operating Modes
• Key Configurations for Connect workers:
• Kafka Connect API
• Kafka Connect Example – File Source
• Kafka Connect Example – File Sink
• Kafka Connector Example – MySQL to Elasticsearch
• Kafka Connector Example – MySQL to Elasticsearch (Contd.)
• Write the data to Elasticsearch
• Building Custom Connectors
• Kafka Connect – Connectors
• Kafka Connect - Tasks
• Kafka Connect - Workers
• Kafka Connect - Offset management
• Alternatives to Kafka Connect
• Introduction to Storm
• Integrating Storm with Kafka
• Integrating Storm with Kafka – Sample Code
• Integrating Storm with Kafka
• Integrating Hadoop with Kafka
• Hadoop Consumers
• Hadoop Consumers (Contd.)
• Hadoop Consumers – Produce Topic
• Hadoop Consumers – Fetch Generated Topic
• Kafka at Uber
• Kafka at Uber (Contd.)
• Kafka at LinkedIn
•  
• Kafka at LinkedIn – Core Kafka Services
• Kafka at LinkedIn – Core Kafka Services (Contd.)
• Kafka at LinkedIn – Libraries
• Kafka at LinkedIn – Monitoring and Stream Processing
• Summary

Chapter 5 - Kafka and Schema Management

• Evolving Schema
• Protobuf (Protocol Buffers) Overview
• Avro Overview
• Managing Data Evolution Using Schemas
• Confluent Platform
• Confluent Schema Registry
• Schema Change and Backward Compatibility
• Collaborating over Schema Change
• Handling Unreadable Messages
• Deleting Data
• Segregating Public and Private Topics
• Summary

Chapter 6 - Kafka Streams and KSQL

• What Kafka can be used for?
• What Kafka can be used for? (Contd.)
• What Exactly is Kafka?
• The APIs for Stream Processing
• Kafka: A Streaming Platform
• What is KSQL?
• What is KSQL? (Contd.)
• Starting KSQL
• Using the KSQL CLI
• KSQL Data Types
• Review the Structure of an Existing STREAM
• Query the STREAM
• KSQL Functions
• Writing to a Topic
• KSQL Table vs. Stream
• KSQL JOIN
• Windows in KSQL Queries
• Miscellaneous KSQL Commands
• Summary

Chapter 7 - KSQL UDF and Deployment

• KSQL Custom Functions
• KSQL UDF/UDAF
• Implement a Custom Function
• Creating UDF and UDAF
• Creating UDF and UDAF (Contd.)
• UDFs and Null Handling
• UDFs and Null Handling (Contd.)
• Sample UDF Class
• Build Engine
• UDAF
• UDAF Sample Class
• Supported Types
• Deploying Custom Functions
• Using Custom Functions
• Summary