WA2920
Confluent Kafka for System Administrators Training
This course teaches system administrators how to manage Confluent Kafka, a powerful platform for real-time data processing. Attendees gain expertise in installing, configuring, and optimizing Kafka clusters, ensuring high-performance messaging for your microservices and cloud-native applications.
Course Details
Duration
3 days
Prerequisites
Basic understanding of messaging, cloud, Linux systems operation, architecture and virtualization would be beneficial.
Target Audience
Systems Administrator who are responsible for planning, installing, operating and troubleshooting a Confluent Kafka installation.
Skills Gained
- Understand the use of Kafka for high performance messaging
- Identify the usages for Kafka in Microservices
- Explain the benefits of Kafka patterns
- Differentiate between messaging and message brokers
- Summarize characteristics and architecture for Kafka
- Install and configure a Kafka cluster
- Design distributed high throughput systems based on Kafka
- Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka
Course Outline
- Introduction to Confluent Kafka
- Messaging Architectures – What is Messaging?
- Messaging Architectures – Steps to Messaging
- Messaging Architectures – Messaging Models
- What is Kafka?
- Kafka Overview
- Need for Kafka
- When to Use Kafka?
- Kafka Architecture
- Core concepts in Kafka
- Kafka Topic
- Architecting Topics
- Kafka Partitions
- Kafka Producer
- Kafka Consumer
- Kafka Broker
- Kafka Cluster
- Why Kafka Cluster?
- Sample Multi-Broker Cluster
- Overview of ZooKeeper
- Kafka Cluster & ZooKeeper
- Who Uses Kafka?
- Installing and Configuring Confluent Kafka
- Planning for Kafka - Platform
- Planning for Kafka - OS
- Planning for Kafka - Java
- Planning for Kafka - System Requirements
- Installing Confluent Platform (Kafka)
- Downloading Confluent Platform
- Setting up for Development and Testing
- Setting up for Production
- Running in Docker Containers
- Configuration Files
- Starting Kafka
- Using Kafka Command Line Client Tools
- Setting up a Multi-Broker Cluster
- Using Multi-Broker Cluster
- Kafka Cluster Planning
- Kafka Cluster Planning – Producer/Consumer Throughput
- Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
- Kafka Cluster Planning – Sizing for Topics and Partitions
- Kafka Cluster Planning – Sizing for Storage
- Kafka Connect
- Kafka Connect – Configuration Files
- Using Kafka Connect to Import/Export Data
- Data Pipelines Overview
- Building Data Pipelines
- Best Practices for Designing Pipelines
- Considerations When Building Data Pipelines
- Timeliness
- Reliability
- High and Varying Throughput
- Data Formats
- Transformations
- Security
- Security Best Practices
- Failure Handling
- Coupling and Agility
- Ad-hoc Pipelines
- Loss of Metadata
- Extreme Processing
- Kafka Connect Versus Producer and Consumer
- Integrating Kafka with Other Systems
- Introduction to Kafka Integration
- Kafka Connect
- Running Kafka Connect
- Key Configurations for Connect workers:
- Kafka Connect Example – File Source
- Kafka Connect Example – File Sink
- Kafka Connector Example – PostgresSQL to Elasticsearch
- PostgresSQL to Elasticsearch Process
- Write Data to Elasticsearch
- Kafka Connector Scenario – SOAP Web Service to REST
- Building Custom Connectors
- Kafka Connect – Connectors
- Kafka Connect - Tasks
- Kafka Connect - Workers
- Kafka Connect - Converters and Connect’s data model
- Kafka Connect - Offset management
- Alternatives to Kafka Connect
- Introduction to Hadoop
- Hadoop Components
- Integrating Hadoop with Kafka
- Hadoop Consumers
- Hadoop Consumers – Produce Topic
- Hadoop Consumers – Fetch Generated Topic
- Kafka at Uber
- Kafka at LinkedIn
- Kafka at LinkedIn – Core Kafka Services
- Kafka at LinkedIn – Libraries
- Kafka at LinkedIn – Monitoring and Stream Processing
- Schema Management, Streams and KSQL
- Evolving Schema
- Protobuf (Protocol Buffers) Overview
- Avro Overview
- Managing Data Evolution Using Schemas
- Confluent Platform
- Confluent Schema Registry
- Schema Change and Backward Compatibility
- Collaborating over Schema Change
- Handling Unreadable Messages
- Deleting Data
- Segregating Public and Private Topics
- What is KSQL?
- Starting KSQL
- Using the KSQL CLI
- KSQL Data Types
- Managing Kafka
- Monitoring Kafka
- Confluent Control Center
- Control Center - Screen Shot
- Control Center - Running
- Control Center - Monitoring
- Control Center - Management
- Control Center - Alerts
- Control Center - Development
- Troubleshooting basics
- Zookeeper CLI Shell
- kafka-topics Command Line Script
- console-consumer Command Line Script
- Viewing logs
- Inspecting Message Queue
- Viewing Messages with Control Center
- Viewing messages with REST Proxy
- Maintaining Kafka Cluster
- Updating Kafka version
- Backup and Restore
- Disaster recovery
- Lab Exercises
- Lab 1. Confluent Kafka Basics
- Lab 2. Confluent Kafka Multiple Brokers and Import/Export Messages
- Lab 3. Securing Confluent Kafka
- Lab 4. Integrating Confluent Kafka with PostgreSQL
- Lab 5. Confluent Kafka Monitoring using Graphite
- Lab 6. KSQL Basics
- Lab 7. Troubleshoot a Kafka Application
Upcoming Course Dates
