I've been like this all my life, but it especially happens when I'm at work. People will say a joke and I end up staring at them with a blank face or a slightly confused face for a few seconds trying to figure out if they're joking or not. Or I will answer literally/seriously and they normally end up awkwardly/fake laughing or saying "I'm joking/kidding" or "I'm just messing with you". I've basically just ended up laughing to cover my confusion most of the time, although I may laugh when they're being serious which makes it a bit awkward. My dad's the same way, when we talk we understand each other and are very straightforward with each other.
People have genuinely asked me if I'm on the autism spectrum because of it, I don't know if I am or not.