Is yeah necessary or annoying?

Do you talk to people often who while you're speaking will just say "yeah" right in the middle of your sentence? I can't say I've experienced this a lot but I have had a few close relationships where I just couldn't speak without them saying yeah in the middle of what I was trying to say pretty much all the time. I get that they are trying to show that they are listening but by doing that it seems they have stopped listening and their adhd has kicked in and they just have to speak in the middle of me speaking. It really seems rude because I will be in the middle of a thought process and they just cut me off. If you're listening then be patient and listen. Does this bother anyone else or is it just me?