How to Scale End-to-End Observability in AWS Environments

Amazon MSK Connect Launched to Better Apache Kafka UX

Amazon follows up on its 2018 data streaming software, Amazon Managed Streaming for Apache Kafka, with a new offering optimized with managed connectors.

Sept. 21, 2021, 5:38 p.m.

TL;DR

On 16 September 2021, AWS announced the launch of Amazon MSK Connect, an offering with extended compatibility to facilitate Kafka Connect efficiency.

It is automated to patch and manage the latent hardware, auto-scale connectors to match throughput changes.

Key Facts

1
Amazon MSK Connect supports inter-platform migration with supported platforms.
2
Amazon MSK Connect mitigates Kafka Connect setup processes with just a few clicks.
3
It eliminates the burden of managing infrastructure, allowing users to focus on important events.

Details

Since its announcement at re: Invent 2018, the real-time data pipeline streaming platform has been optimized by many in the DevOps community.

Apache Kafka is a stream-processing framework that implements a software bus. It is a Scala and Java-based open source development platform developed by the Apache Software Foundation. It is oriented at providing a single, maximum-throughput, low-latency platform for real-time data flows. In 2018, Amazon launched an extension to facilitate real-time data streaming, Amazon Managed Streaming for Apache Kafka, which leverages Apache Kafka's capabilities to build and run applications and data pipelines with ease.

Kafka Connect is an Apache Kafka open source accessory that provides a framework for connecting to external systems such as databases, search indexes, file systems, and key-value stores. Manually running Kafka Connect clusters, on the other hand, necessitates users planning and provisioning the necessary infrastructure, managing cluster operations, and scaling the cluster in response to load changes.

Amazon's new offering is a game-changer in that instance. Amazon MSK Connect makes MSK Connect more soothing to operate. MSK Connect is an escape from the voluminous setup requirements associated with Kafka Connect. Users can now connect and deploy a connector with just a few clicks. It assumes the role of a virtual assistant, perennially monitoring the condition of the connectors to detect frailties that might cause any vacillation. It is automated to patch and manage the latent hardware, auto-scale connectors to match throughput changes. This way, users can implement this on large batches of data without the need for a heavy workforce or constant monitoring; therefore, they can focus on building applications.

MSK Connect and Kafka Connect are entirely compatible; customers may move their current connectors without changing any code. MSK Connect does not require an MSK cluster. It supports Amazon MSK, Apache Kafka, and Apache Kafka compatible clusters. As long as MSK Connect can privately connect to the clusters, these clusters can be self-managed or managed by AWS partners and third parties.