They’re aware that their marriage ended because of their mistake, and that means that they will probably have to pay for it. And it’s not just about the money.
A divorce may also cost them losing their friends and some of their loved ones. People around them will probably be angry at them because of how they hurt their partner and betrayed their marriage.
Also, no one can guarantee that their affair partner will stay with them, which means they might lose her too and be left alone. If they have had kids, they might also worry the kids will be angry at them because of their actions, and that’s probably what they’re afraid of most.