MongoDB Secondary server shows higher record count that primary

Question

We have a 3 server MongoDB cluster where there is one Primary, one secondary and one hidden. In one of our collections data gets only inserted by the front end APP and cannot be deleted. I am seeing a strange behavior - when I run count query on this collection (from shell) I always see secondary's count 1-2 more than the count on Primary.

Is this possible or is there some mismatch in my configuration?

Answer 1

Have you had an unclean shutdown on some of your nodes? You may be running into SERVER-19472, where an unclean shutdown may cause an incorrect count results using WiredTiger. Note that this is not a bug. The incorrect count was a direct result of an unclean shutdown.

You could try to:

Restart the affected Secondaries as a standalone node (without the --replSet parameter)

Execute db.collection.validate(true) on the affected collections

Restart the node with the --replSet parameter

Check if the count returned is correct once the node rejoins the replica set as a Secondary