I forget to ask small talk questions back… Do you think people notice/is it a big deal?

Often happens with my boss – we’ll be on a random call and he asks me about my weekend plans or how I’m doing.

I answer and it doesn’t come to my mind to include ‘what about you?’ as a last sentence of what I’m saying, and then he just says ‘okay’ or whatever and continues to talk about something else, or says his goodbyes…

I know he’s always rushing off the phone quite quickly, he makes fun of it too that sometimes he’ll accidentally cut people off because he’s too quick to end the call. But I always feel like shit after that happens because I’m not sure if I’m being rude by not asking back straight away, or if it’s just a natural flow of conversation and I have nothing to worry about.