Kafka Consumer Group 및 Rebalancing

Consumer Group

1. Consumer 의 집합을 뜻하며, Consumer 대비 Partition 수에 따라서 동작이 발생된다.
2. Consumer > Partition 의 경우 Partition 수량을 초과하는 Consumer 는 동작하지 않는다.
3. Consumer < Partition 의 경우 1개의 Consumer 에서 다수의 Partition 에 대하여 지정된 방식에 따라 동작한다. ( ex . (P1,P2,P1,P2) - RoundRobin etc.. )
4. Consumer 와 Partition 은 1:1 맵핑관계이다. ( Partition 입장 )
5. Consumer 와 Partition 은 1:n 맵핑관계이다. ( Consumer 입장 )

단일 Partition 에서 다수의 Consumer 를 사용하는 경우 1개의 Consumer 만 Active 상태가 되며 다른 Consumer 들은 대기상태가 된다. 대기 Consumer 들은 Active Consumer에 문제 발생시 Kafka Server 로 부터 Partition Assign 에 대한 Event 를 받아서 전환된다.

Consumer Group 동작시에는 Partition 에 대한 지정은 불가능하다.

Rebalance 동작에 대해서 Partition 이 Consumer 보다 많을 경우는 아직 테스트 해보지 않았다.

Rebalance 는 Consumer Group 에서 각 Partition에 서비스를 하던 Consumer 와 연결을 회복하는 기능이다. 단, Rebalance 시 Group의 모든 Consumer 가 멈추기때문에 주의가 필요하다. ( Stop The World , kafka 2.3.0 미만 버전 )

이와 같은 Stop the World를 회피하기 위해서 Incremental Cooperative Rebalancing 를 사용한다. ( kafka 2.3.0 이상 version )

# Eager Rebalancing

1. All Consumer Alive
Partition1 ----- Consumer1 (Active)

Partition2 ----- Consumer2 (Action)

Partition3 ----- Consumer3 (Action) 


2. One is Dead

Partition1 ----- Consumer1 (Active)

Partition2       Consumer2 (Dead)

Partition3 ----- Consumer3 (Action) 


3. All Consumer Stop For Rebalance

Partition1 ----- Consumer1 (Stop)

Partition2       Consumer2 (Dead)

Partition3 ----- Consumer3 (Stop) 


4. Rebalance End

Partition1 ----- Consumer1 (Active)
             | 
Partition2 --    Consumer2 (Dead)

Partition3 ----- Consumer3 (Action) 


5. Consumer2 is Alive

Partition1 ----- Consumer1 (Active)
             | 
Partition2 --    Consumer2 (Active)

Partition3 ----- Consumer3 (Action) 


6. All Consumer Stop For Rebalance

Partition1 ----- Consumer1 (Stop)

Partition2       Consumer2 (Dead)

Partition3 ----- Consumer3 (Stop) 


7. Rebalance End

Partition1 ----- Consumer1 (Active)

Partition2 ----- Consumer2 (Action)

Partition3 ----- Consumer3 (Action) 



# Incremental Cooperative Rebalancing

1. All Consumer Alive

Partition1 ----- Consumer1 (Active)

Partition2 ----- Consumer2 (Action)

Partition3 ----- Consumer3 (Action) 


2. One is Dead

Partition1 ----- Consumer1 (Stop) ( New Partition Assignment Consumer Stop )

Partition2       Consumer2 (Dead)

Partition3 ----- Consumer3 (Action) 


3. Rebalance End

Partition1 ----- Consumer1 (Active) 
             |
Partition2 --    Consumer2 (Dead)

Partition3 ----- Consumer3 (Action) 


4. Consumer2 is Alive

Partition1 ----- Consumer1 (Active) ( Assignment Consumer Stop )

Partition2       Consumer2 (Active) ( Alive Consumer Stop )

Partition3 ----- Consumer3 (Action) 


5. Rebalance End

Partition1 ----- Consumer1 (Active)

Partition2 ----- Consumer2 (Active)

Partition3 ----- Consumer3 (Action) 

Confluent 에서 말하는 Kafka Rebalance 시 동작에 따른 성능 차이

자체 관리 Kafka 클러스터에 대한 90 개의 S3 커넥터 / 900 작업	Eager Rebalancing	Incremental Cooperative Rebalancing	Improvement with Incremental Cooperative Rebalancing
Aggregate throughput (MB/s)	252.68	537.81	113%
Minimum throughput (MB/s)	0.23	0.42	183%
Maximum throughput (MB/s)	0.41	3.82	833%
Median throughput (MB/s)	0.27	0.54	101%

Rebalance 를 사용할경우와 사용하지 않을 경우 아래와 같이 대조된다.

#use Rebalance
Consumer1,2,3 is same Consumer Group

1. First Connection Conumer is Main Processing 

Partition1 ----- Consumer1 (First Connect)
            |
            ---- Consumer2 (Not Action)
            |
            ---- Consumer3 (Not Action) 


2. First Consumer is Dead , Second Connection Consumer is Processing

Partition1 ----- Consumer1 (Dead)
            |
            ---- Consumer2 (Action)
            |
            ---- Consumer3 (Not Action) 


3. if First Consumer is Alive, Second Connection Consumer Not Processing

Partition1 ----- Consumer1 (Alive - Action)
            |
            ---- Consumer2 (Not Action)
            |
            ---- Consumer3 (Not Action) 


#not use Rebalnce

1. First Connection Conumer is Main Processing 

Partition1 ----- Consumer1 (First Connect)
            |
            ---- Consumer2 (Not Action)
            |
            ---- Consumer3 (Not Action) 


2. First Consumer is Dead , Second Connection Consumer is Processing

Partition1 ----- Consumer1 (Dead)
            |
            ---- Consumer2 (Action)
            |
            ---- Consumer3 (Not Action) 


3. if First Consumer is Alive, Second Connection Consumer Not Processing

Partition1 ----- Consumer1 (Alive - Not Action)
            |
            ---- Consumer2 (Action)
            |
            ---- Consumer3 (Not Action)