Registering image data to Structure from Motion (SfM) point clouds is widely used to find precise camera location and orien- tation with respect to a world model. In case of videos one constraint has previously been unexploited: temporal smoothness. Without tem- poral smoothness the magnitude of the pose error in each frame of a video will often dominate the magnitude of frame-to-frame pose change. This hinders application of methods requiring stable poses estimates (e.g. tracking, augmented reality). We incorporate temporal constraints into the image-based registration setting and solve the problem by pose reg- ularization with model fitting and smoothing methods. This leads to accurate, gap-free and smooth poses for all frames. We evaluate differ- ent methods on challenging synthetic and real street-view SfM data for varying scenarios of motion speed, outlier contamination, pose estima- tion failures and 2D-3D correspondence noise. For all test cases a 2 to 60-fold reduction in root mean squared (RMS) positional error is ob- served, depending on pose estimation difficulty. For varying scenarios, different methods perform best. We give guidance which methods should be preferred depending on circumstances and requirements.