I am able to reproduce the situation you described.
After starting the Smart Dimension, when I pick the Boss circle first to locate its center, it won't pick.
If I first pick a reference, like an edge or sketch and then locate the center of the boss circle, it does pick. This was consistent so I am also able to verify this.
Alternatively, using the Distance Between command lets me pick the boss center first, consistently.
Make sure your Keypoints filter is set to allow that type of keypoint...