It’s not a sexual reference, although he’s not lacking this attraction either. He enjoys touching your hand or any part of you, to be honest.
He’ll say you have lovely hair and he’ll take a chance to touch it. He’ll also want to kiss you A LOT. It’s one of the powerful signs of his attraction to you.