Image-guided radiation therapy during free-breathing requires estimation of the target position and compensation for its motion. Estimation of the observed motion during therapy needs to be reliable and accurate. In this paper we propose a novel, image sequence-specific confidence measure to predict the reliability of the tracking results. The sequence-specific statistical relationship between the image similarities and the feature displacements is learned from the first breathing cycles. A confidence measure is then assigned to the tracking results during the real-time application phase based on the relative closeness to the expected values. The proposed confidence was tested on the results of a learning-based tracking algorithm. The method was assessed on 9 2D B-mode ultrasound sequences of healthy volunteers under free-breathing. Results were evaluated on a total of 15 selected vessel centers in the liver, achieving a mean tracking accuracy of 0.9 mm. When considering only highly-confident results, the mean (95th percentile) tracking error on the test data was reduced by 12% (16%) while duty cycle remained sufficient (60%), achieving a 95% accuracy below 3 mm, which is clinically acceptable. A similar performance was obtained on 10 2D liver MR sequences, showing the applicability of the method to a different image modality.