Native C# client for Apache Kafka with full async support.

Kafka versions [0.9, 0.10.1]

OS	Framework	Status
Windows	.net 4.6.2
Windows	.net standard 1.6
Linux	.net standard 1.6

Copyright 2016, Nudge Software Inc under Apache License, V2.0. See LICENSE file.

This library is a fork of gigya's KafkaNetClient, itself a fork of jroland's kafka-net library. There are significant API changes.

The original .Net project is a port of the Apache Kafka protocol. The wire protocol portion is based on the kafka-python library writen by David Arthur and the general class layout attempts to follow a similar pattern as his project. To that end, this project builds up from the low level Connection object for handling async requests to/from the kafka server, all the way up to a higher level Producer/Consumer classes.

A big thank you to Nudge Software for backing this project.

var options = new KafkaOptions(new Uri("tcp://SERVER1:9092"), new Uri("tcp://SERVER2:9092"));

using(var client = new Producer(options)) {
     await client.SendMessageAsync("TestTopic", new Message("hello world"));
}

var options = new KafkaOptions(new Uri("tcp://SERVER1:9092"), new Uri("tcp://SERVER2:9092"));

using (var router = new BrokerRouter(options)) {
	var offset = await router.GetTopicOffsetAsync("TestTopic", 0, CancellationToken.None);

	using (var consumer = new Consumer(router)) {
	    await consumer.FetchAsync(offset, 10, message => Console.WriteLine("Response: P{0},O{1} : {2}", message.Meta.PartitionId, message.Meta.Offset, message.Value), cancellationToken);
	}
}

Provides async methods on a persistent connection to a kafka broker (server). The send method uses the TcpClient sendAsync function and the read stream has a dedicated thread which uses the correlation Id to match send responses to the correct request.

Provides a high level abstraction for sending batches of messages to a Kafka cluster. Internally, it uses the IBrokerRouter and IConnection.

Provides a mechanism for fetching messages from a Kafka cluster.

Provides an extesible mechanism for assigning partitions to members of consumer groups, enabling complex coordination across multiple consumers of a set of Kafka topics. The default assignor (ConsumerAssignor) will round robin partition selection across topics.

Provides the logic for routing requests to a particular topic to a partition. The default selector (PartitionSelector) will use round robin partition selection if the key property on the message is null and a mod/hash of the key value if present.

Provides routing for topics to partitions, topics and partitions to brokers, and cached access to topic metadata. This class also manages the multiple IConnections to each Kafka broker. Routing logic for partitions is provided by IPartitionSelector.

The protocol has been divided up into concrete classes for each request/response pair. An Encoder class knows how to encode requests and decode responses into/from their appropriate Kafka protocol byte array. One benefit of this is that it allows for a nice generic send method on the Connection.

** WARNING ** This library is still work in progress and has not yet been deployed to production. It is also undergoing significant development, and breaking changes will occour. This notice will be removed once it's been stabilized and used in production.

The biggest missing piece at this point is stress testing; more comprehensive automated tests are also needed. For the full set, see the [backlog] (https://github.com/awr/KafkaClient/projects/1).