Why can I socialize with individuals and enjoy public speaking, but get nervous in smaller groups?

Small group dynamics. My guess is that you're afraid of standing out in a small group, for some reason or another. Be it fear of coming off as snobbish, offensive, weird, stupid, awkward, rude, elitist…etc. One on one you can read your target, and quickly adapt. As a presenter, you and your message is the focus of attention. In a small group, you need to be able to quickly read and adapt to multiple individuals, while juggling this thought process, you admittedly kind of break down. If it's work/school related – I recommend taking the reins and try to establish yourself in a leadership position and see if that clears up the jitters. Don't be afraid to assert yourself, your ideas, or your opinion. Confidence is like a drug – both for yourself and others, and even just faking it can have positive effects. If it's social – Work slowly, establish a repertoire with a few others first and slowly branch out. You don't have to be friends or get along with everybody there, just don't be a jerk.