Why can’t I finish during sex?

I’ve only been with two men, and even though the sex has been good and I enjoy myself the whole time, I just never finish. I’ve come close, but then it just doesn’t happen for one reason or another. Not sure if my downstairs is just broken, but I finish just fine alone