BCC 8 Evolution: Kafka

 In order to satisfy the current interests of our clients to have cloud deployments, we have included Apache Kafka and ZeroMQ as messaging tools to store and process the data flow within the BCC 8 platform. 

Incorporating Apache Kafka in the BCC solution allows us to replace Multicast as a protocol for the integration and communication of the different services that make up the solution. 

Among the most outstanding benefits of this platform are: 

- Tolerance of errors in the network 

- Excellent scalability 

- High-speed reading and writing 

- Security 

Integrated with Kafka, the BCC solution uses ZeroMQ, both of which are different implementations of a message bus solution. 

Being able to configure a Kafka and ZeroMQ cluster allows us to store and replicate data in a distributed way, thus allowing us to send and receive "heartbeat" messages, which are used for load balancing, FAILOVER operation, and for information transfer for the creation of DOCSIS files. 

The servers that make up the cluster can be located in different datacenters. The nodes (brokers) store the incoming data flows, which are classified into “topics”. The data is divided into partitions which are replicated and distributed across the cluster. 

BCC will use ZeroMQ for certain provisioning messages (messages to identify “FileID” files between DHCP and CFM or MPS and CFM) 

ZeroMQ and Kafka integrate, they are both different implementations of a message bus solution. The use of ZeroMQ is "behind the scenes" in BCC. Once you select to use message bus instead of Multicast, the BCC services will know to use ZeroMQ instead of Multicast for filehandles. 

ZeroMQ turns out to be faster because the way we use it is point to point. Whereas, with Kafka, the messages go first to a "broker" and then to a client. So, there is a bit of latency there, which is why we chose to use ZeroMQ for the “FileIDs” messages. 

In order to handle an optimal configuration in terms of high availability, our recommendation is to configure three Kafka servers (brokers) 

I invite you to start tests in your lab environment in order to incorporate Apache Kafka | ZeroMQ on your provisioning platform. Our Support Department, as well as our Professional Services Department, will assist you if you need it.