OpenCV's VideoCapture is broken and hasn't been fixed for the last 5 years: opencv/opencv#9053
This is a PyAV based replacement. Unlike other implementations it can seek at any time.
How to use:
reader = VideoReader('video.mp4')
reader.seek(reader.total_frames - 100) # frame number
while True: