Why do men talk about me within earshot? (Male)

For example, my roommate had friends over. They knew particularities about me, like what specific supplements I take and brought it up with me in the room. But in a bad light. I learned when this happens I have to jump in and start talking or else my ears burn. Or I’ll tell one roommate something particular and the other brings it up randomly. It has also happened on jobsites. People will talk about things that seem to be about me, things I do that might be different, slightly weird things I told them, etc. It makes me feel bad. One time it happened and wasn’t that bad: two guys at a new job started quoting a hundred different movies within earshot to make me laugh and I think to be friendly and start a conversation. I am a quiet person. Why do people do this? What are their intentions? What is the solution to stop this happening? Why does this always happen to me?
I know my true thoughts on this are controversial. I believe a man, or human, should be direct with their complaints instead of gossiping or talking loud enough for someone to hear.