Verify/troubleshoot master leadership transfer

Alluxio Version: v2.x with Embedded Journal and Raft

Deployment Type: BareMetal, Kubernetes

Goal: To validate master-leader ship transfer 

Running the below command on the current leader (for example master3) to elect master2 as the new leader:

alluxio fsadmin journal quorum elect -address master2:19200

Log entries from master3.log (the newly elected leader will have higher priority):

INFO RaftJournalSystem - Applying new peer state before transferring leadership: [master1_19200|rpc:master1:19200|priority:1, master2_19200|rpc:master2:19200|priority:2, master3_19200|rpc:master3:19200|priority:1]

INFO RaftJournalSystem - Transferring leadership to master with address <master2:19200> and with RaftPeerId <master2_19200>

INFO RaftJournalSystem - Transferring leadership initiated

INFO RoleInfo -master3_19200: start master3_19200@group-ABB3109A44C1-FollowerState

INFO FaultTolerantAlluxioMasterProcess - Losing the leadership.

INFO VoteContext - master3_19200@group-ABB3109A44C1-FOLLOWER: accept ELECTION from master2_19200: our priority 1 <= candidate's priority 2

INFO AlluxioMasterProcess - Alluxio master ended (lost leadership)

INFO RaftJournalSystem - Losing primacy.

INFO DefaultMetaMaster - Standby master with address localhost:19998 starts sending heartbeat to leader master.

INFO  BackupWorkerRole - Established connection to backup-leader: master2:19998

The new leader (master2) will have the below entries in its master2.log (the newly elected leader will have higher priority)

INFO RoleInfo - master2_19200: start master2_19200@group-ABB3109A44C1-LeaderElection1

INFO LeaderElection - master2_19200@group-ABB3109A44C1-LeaderElection1 ELECTION round 0: submit vote requests at term 2 for 191: [master1_19200|rpc:master1:19200|admin:|client:|dataStream:|priority:1, master2_19200|rpc:master2:19200|admin:|client:|dataStream:|priority:2, master3_19200|rpc:master3:19200|admin:|client:|dataStream:|priority:1], old=null

INFO LeaderElection - master2_19200@group-ABB3109A44C1-LeaderElection1: ELECTION PASSED received 1 response(s) and 0 exception(s):

INFO FaultTolerantAlluxioMasterProcess - Becoming a leader.

INFO RaftJournalSystem - Gained primacy.

INFO FaultTolerantAlluxioMasterProcess - state is PRIMARY, standbyMetricsSinkEnabled is false, standbyWebEnabled is false

INFO FaultTolerantAlluxioMasterProcess - Primary started

INFO DefaultMetaMaster - registerMaster(): master: MasterInfo{id=6946457552985714316, address=localhost:19998, lastUpdatedTimeMs=1704299968551, startTimeMs=1704298989703, losePrimacyTimeMs=0, version=enterprise-2.8.0-5.2, revision=9cefdb092b566e74499ef5b6a8b0e2a8f8bcc224}

 
The newly elected (master2) leader will also receive handshake messages from the other 2 standby masters and the below entries will be printed in the master2.log:

INFO BackupLeaderRole - Received handshake message:BackupHandshakeMessage{BackupWorkerHostName=master3}

INFO BackupLeaderRole - Received handshake message:BackupHandshakeMessage{BackupWorkerHostName=master1}