I feel left out everywhere I go, should I point fingers at myself?

My entire life I had a low number of friends. I felt okay with having one on one friendships. I was always a sensitive person, and I do tend to take things more personally than the average person, possibly from my insecurities and the need to defend myself for the very things I dont like myself for. I was close friends with 3-4 people my entire life, and those same ones stuck with me for a while. Scared to explore new friendships and reveal who I am to other people, I tolerated other old friends who didnt treat me well, and had zero chemistry with, constantly longing for someone better to remind me that friendships are and can be better. In high school, the same thing happened to me. In uni, I was having a hard time, my parents divorced, I had issues with who I was and accepting myself and my personality and my looks. I wasn’t bothering getting to know people and eventually everyone had a group they belonged to except me. Same thing happened at a hotline I worked in. There I found myself awkward and not able to accept other peoples approaches. Some did try, but I felt like I didn’t belong still. Now I have another job, and I had this sense of relief when I met my colleagues, I was comfortable being silent and not worrying about someone calling me “quiet”, as growing up thats all people labeled me as, so even when I believed I was something more, they’d undermine me and my traits by describing me as quiet, whereas I knew how I am with my few people. They find me talkative and enjoyable. However, the past two to three days my colleague and the driver have been ignoring me. The driver keeps disregarding my existence, told me “You dont talk unless ure asked?”, and then I caught them gettint coffee for the second time without bothering to invite me. I was left alone. Not that I cares as much to be with them because we have nothing in common, but it just surprised me how can someone be so rude? And I asked my colleague is there a problem, that Im not invited for the second time now? The drivers problem is that I am apparently “quiet”. And I said I am not, but even if I was those are the people youre supposed to bring and invite the most, not leave out unless they said no. And now it’s been extremely awkward and stressful for me. Its like I am bugging them meanwhile Im here to so my job. I feel left out once again, and Im scared I will never adapt anywhere during my life.
Ill always be just this quiet girl, who makes people uncomfortable whereas whenever Im out with my friends, they love me and want to introduce me to their friends as well so I dont know what to believe anymore. The second I gain some confidence that I can be social and I am an okay person, something bizarre has to happen for me to forget I ever thought I am not all that bad.