Have you tried to mend fences a few times, but you still see that it’s not working? Maybe your only solution is a breakup?
Whatever you decide, if you see that your partner is having a hard time coming to terms with it, but he still understands your feelings and your decision, you know he cares.
Of course, it will be hard for him to come to terms with losing you, but if he understands and respects your decision, he will undoubtedly not allow himself to hurt you.