Hi,
Protocol error detection is done in RLC AM mode through ARQ.
If maximum ARQ re transmission happens then there may be an configuration error so RLC reports this to higher layer. Recovery is done through ARQ re-transmission , If the packet is not received then it will be recovered by giving NACK to the sender.