I heard from someone the other day...
"Do what is right--the others be damned!"
And this is right on the money.
You should always follow the dictates of your conscience.
Do not worry about pressure from others or what others will do that you cannot control.
My dad (A"H) used to say:
"YOU do what is right--YOU be the example!"
He was my example, and I will always follow in his footsteps. ;-)
(Source photo: here with attribution to cursedthing)