Is there something wrong with me

I get really annoyed when I mention briefly to someone that I “enjoy” certain things and then that comes up in another conversation later on where they “assume” I “really like” those things (I might like those hobbies but don’t associate my identity with them or call myself a fan). I don’t know if it makes sense. I guess I just don’t like it when I’m labeled. Any advice or enlightenment would be greatly appreciated.