If he says something that annoys you or if he hurts your feelings voluntarily and you’re not letting him know, he will think that you’re too tolerant and he will know that he can do whatever he wants with you.
And that is when you lose respect from him.
Always have boundaries and stick to them. Don’t tolerate his bullshit only because you like him and you want to be the best for him.