How come every time I start to come around to liking a guy (which can take awhile) they'll pull away?

I get that we're obviously not a good match if it happens but it's just pattern that I've noticed. Is there anyway to stop it or should I just not worry about it and chalk it up to being a bad match in the first place?