I keep finding myself being bullied and I am not sure what action I am doing that is causing this.

I got barged today at work by my supervisor (I work in construction). I was walking along and he shoulder bumped me. I immediately did it right back to him. Later he started shouting at me for something unrelated and I stood up for myself again. Later on in the day he started making fun of me and I stood my ground again. I try my best to stand my ground and not get pushed around by people and yet I continue to find myself in these positions. This kind of thing happens again and again. I am like a magnet for bullies. They seem to sense some kind of weakness in me and latch on even when I stand up for myself. I have had to move jobs because of this kind of thing and living spaces. I don’t understand what behavior I display that brings this across. I even got into martial arts to be able to defend my self in these situations and even though I could now Knock out most guys my size I doesn’t seem to change things. I feel confident but the treatment is still the same.