Course #:WA2920

Confluent Kafka for System Administrators Training

In this course, you will learn how to use Kafka to modernize your applications. Upon completion of the Kafka course, participants will be able to:

  • Understand the use of Kafka for high performance messaging
  • Identify the usages for Kafka in Microservices
  • Explain the benefits of Kafka patterns
  • Differentiate between messaging and message brokers
  • Summarize characteristics and architecture for Kafka
  • Install and configure a Kafka cluster
  • Design distributed high throughput systems based on Kafka
  • Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka

Audience

Systems administrator who are responsible for planning, installing, operating and troubleshooting a Confluent Kafka installation.

Prerequisites

Basic understanding of messaging, cloud, Linux systems operation, architecture and virtualization would be beneficial

Duration

Three days

Outline of Confluent Kafka for System Administrators Training

Chapter 1. Introduction to Confluent Kafka

  • Messaging Architectures – What is Messaging?
  • Messaging Architectures – Steps to Messaging
  • Messaging Architectures – Messaging Models
  • What is Kafka?
  • What is Kafka? (Contd.)
  • Kafka Overview
  • Need for Kafka
  • When to Use Kafka?
  • Kafka Architecture
  • Core concepts in Kafka
  • Kafka Topic
  • Architecting Topics
  • Kafka Partitions
  • Kafka Producer
  • Kafka Consumer
  • Kafka Broker
  • Kafka Cluster
  • Why Kafka Cluster?
  • Sample Multi-Broker Cluster
  • Overview of ZooKeeper
  • Kafka Cluster & ZooKeeper
  • Who Uses Kafka?
  • Summary

Chapter 2. Installing and Configuring Confluent Kafka

  • Planning for Kafka - Platform
  • Planning for Kafka - OS
  • Planning for Kafka - Java
  • Planning for Kafka - System Requirements
  • Installing Confluent Platform (Kafka)
  • Downloading Confluent Platform
  • Setting up for Development and Testing
  • Setting up for Production
  • Running in Docker Containers
  • Configuration Files
  • Starting Kafka
  • Using Kafka Command Line Client Tools
  • Setting up a Multi-Broker Cluster
  • Using Multi-Broker Cluster
  • Kafka Cluster Planning
  • Kafka Cluster Planning – Producer/Consumer Throughput
  • Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
  • Kafka Cluster Planning – Sizing for Topics and Partitions
  • Kafka Cluster Planning – Sizing for Storage
  • Kafka Connect
  • Kafka Connect – Configuration Files
  • Using Kafka Connect to Import/Export Data
  • Summary

Chapter 3. Data Pipelines Overview

  • Building Data Pipelines
  • Best Practices for Designing Pipelines
  • Considerations When Building Data Pipelines
  • Timeliness
  • Reliability
  • High and Varying Throughput
  • High and Varying Throughput (Contd.)
  • Data Formats
  • Data Formats (Contd.)
  • Transformations
  • Transformations (Contd.)
  • Security
  • Security Best Practices
  • Failure Handling
  • Coupling and Agility
  • Ad-hoc Pipelines
  • Loss of Metadata
  • Extreme Processing
  • Kafka Connect Versus Producer and Consumer
  • Kafka Connect Versus Producer and Consumer (Contd.)
  • Summary

Chapter 4. Integrating Kafka with Other Systems

  • Introduction to Kafka Integration
  • Kafka Connect
  • Kafka Connect (Contd.)
  • Running Kafka Connect
  • Key Configurations for Connect workers:
  • Kafka Connect Example – File Source
  • Kafka Connect Example – File Sink
  • Kafka Connector Example – PostgresSQL to Elasticsearch
  • PostgresSQL to Elasticsearch Process
  • Write Data to Elasticsearch
  • Kafka Connector Scenario – SOAP Web Service to REST
  • Building Custom Connectors
  • Kafka Connect – Connectors
  • Kafka Connect - Tasks
  • Kafka Connect - Workers
  • Kafka Connect – Workers (Contd.)
  • Kafka Connect - Converters and Connect’s data model
  • Kafka Connect - Offset management
  • Alternatives to Kafka Connect
  • Alternatives to Kafka Connect (Contd.)
  • Introduction to Hadoop
  • Hadoop Components
  • Integrating Hadoop with Kafka
  • Hadoop Consumers
  • Hadoop Consumers (Contd.)
  • Hadoop Consumers (Contd.)
  • Hadoop Consumers – Produce Topic
  • Hadoop Consumers – Fetch Generated Topic
  • Kafka at Uber
  • Kafka at Uber (Contd.)
  • Kafka at LinkedIn
  • Kafka at LinkedIn – Core Kafka Services
  • Kafka at LinkedIn – Core Kafka Services (Contd.)
  • Kafka at LinkedIn – Libraries
  • Kafka at LinkedIn – Monitoring and Stream Processing
  • Conclusion
  • Summary

Chapter 5. Schema Management, Streams and KSQL

  • Evolving Schema
  • Protobuf (Protocol Buffers) Overview
  • Avro Overview
  • Managing Data Evolution Using Schemas
  • Confluent Platform
  • Confluent Schema Registry
  • Schema Change and Backward Compatibility
  • Collaborating over Schema Change
  • Handling Unreadable Messages
  • Deleting Data
  • Segregating Public and Private Topics
  • What is KSQL?
  • What is KSQL? (Contd.)
  • Starting KSQL
  • Using the KSQL CLI
  • KSQL Data Types
  • Summary
  • Conclusion

Chapter 6. Managing Kafka

  • Monitoring Kafka
  • Confluent Control Center
  • Control Center - Screen Shot
  • Control Center - Running
  • Control Center - Monitoring
  • Control Center - Management
  • Control Center - Alerts
  • Control Center - Development
  • Troubleshooting basics
  • Zookeeper CLI Shell
  • kafka-topics Command Line Script
  • console-consumer Command Line Script
  • Viewing logs
  • Inspecting Message Queue
  • Viewing Messages with Control Center
  • Viewing messages with REST Proxy
  • Maintaining Kafka Cluster
  • Updating Kafka version
  • Backup and Restore
  • Disaster recovery
  • Summary

Lab Exercises

Lab 1. Confluent Kafka Basics
Lab 2. Confluent Kafka Multiple Brokers and Import/Export Messages
Lab 3. Securing Confluent Kafka
Lab 4. Integrating Confluent Kafka with PostgreSQL
Lab 5. Confluent Kafka Monitoring using Graphite
Lab 6. KSQL Basics
Lab 7. Troubleshoot a Kafka Application

We regularly offer classes in these and other cities. Atlanta, Austin, Baltimore, Calgary, Chicago, Cleveland, Dallas, Denver, Detroit, Houston, Jacksonville, Miami, Montreal, New York City, Orlando, Ottawa, Philadelphia, Phoenix, Pittsburgh, Seattle, Toronto, Vancouver, Washington DC.