Recently, I had an opponent that cast a pact on my turn. Then during his upkeep, we both don't say anything and he touches the card on top of his library slowly. He then slowly moves the card slowly from library to his hand but does not see what he's drawing. When he finally puts the drawn card in his hand, tell him that he's forgotten the pact trigger during upkeep and lost the game.

The question I have here is what is the earliest time my opponent can be considered to have moved to his draw step in this situation here? Is it when he's touching the top card on the library? Is it when he's moving the card from library to hand but hasn't seen it? Or is it when the card is put into his hand?