I feel you! I have a friend that I am currently crushing on, though its hard to tell if she likes me back. She has ADHD and other disorders, so she will randomly clasp my hand when she's scared and say some semi-romantic things with me. Its difficult to tell what she means.
Though, if this crush of yours is really making you uncomfortable, then you really should let him know! A simple, "Please don't do that", or ," Im not comfortable with that, sorry", would suffice.
I hope this helps you!
:)