If that's the type of person your new neighbour is, do you really care what he thinks?
Listen. Everybody needs to start somewhere. If you're interested and passionate about getting healthy, then you have to learn to love yourself now. Embrace who you are, and make goals to improve, but without putting yourself down.
So if you're fat and you're exercising with a goal in mind, you're 1000x better than people who are fat and too lazy or self-loathing to bother trying to change their bad habits.
You go girl.
Besides, if you live in a small town, wouldn't you want to be the inspiring person in town, the girl who changed for the better? So people will talk about that fat chick who exercises. But if you stick to it, lose weight, there's not much they can put you down for, is there?
EDIT: If you have some dysfunctional psychological predisposition that makes it so you're literally immobilized by a lack of confidence, you could consider exercising in the comfort of your own home. That's how I got fit.