Remember,
Brave is when you are scared and you do it anyway. Fear is common, we all get very scared sometimes. It's good to practice getting used to fear to avoid regrets later in life. The huge school looks really badass. Imagine: after you went there you are going to be one of the people who kind of intimidate you now at that place, someone who belongs there.
You become a new person over time. The things you are afraid of today are not going to be the ones that scare you later in your life. Now you are afraid of switching school, but by the time you go there, it is going to be just a natural thing to do. Your future self might get scared by a math exam, choosing grad school, or something distant in the future. The trick is to keep going, and to keep challenging yourself. Don't miss out on challenging things out of fear. You would regret it. You aren't going to regret bravery and trying new things though.
(Another thing I just thought about: 300+ students a year means that you would have a pretty good chance of finding friends you can relate to.)
Btw. About all the other things you wrote. Don't let yourself be judged like this. No teaching degree, no doctoral or any study qualifies anyone for passing judgement like the ones you described. Learn to ignore people calling you mentally slow and similar things. There are always going to be people who don't believe in you. Ignore them and just prove them wrong! You are the only person living your life, set your own goals and pursue them.
You are very young, and facing your problems today puts you way ahead of many people. (And smoking is stupid. :p Working out would calm your nerves better.)
I might sound preachy, but I just mean to say that fear doesn't have to be an obstacle. And if you keep going, you'll find cool things in life. Good luck!