in RLC AM mode once HARQ max retransmission reaches RLC will start re transmitting the data, so for every RLC single re-transmission HARQ will try its max retransmission. If RLC max retransmission occurs then RLC will initiate RLF procedure.
In RLC UM modem once HARQ max retransmission reaches MAC will discard the TB and follows the next TB.