I've been feeling some angst about just this subject lately.
I broke up with my girlfriend of five years recently, and while her weight was not the principal reason I made that decision, it was a significant contributor. She wasn't even fat, just getting a little chubby. Anyway, I'm worried that she told everyone she knows that I broke up with her because she's fat.
So yeah, I feel you, OP. I will not get intimate with curvy, BBW, or chubby women. That shit is just not attractive to me at all. Nobody has really questioned me about it, and I don't really know how I would respond either.
Although maybe that speaks to the quality of the friends I've been blessed with; I think that to even ask something like "why do you have such high standards?" or "why won't you date heavier girls?" shows a reprehensible lack of respect for you and your preferences.
I like what like and I know that I don't like chubs. Take care of yourself, brother. Good luck out there.