Is sex a huge factor in a relationship?

We started getting to know each other may as friends nothing more. Fast forward to June and he kisses me out of nowhere which I returned. He tells me he is not ready for a relationship and he’s not looking for a relationship which I agree to as well. And the also says we can be whatever even just friends he’s cool with that. And I said we can continue doing what we are doing and I’d like for it to be exclusive. And I also mentioned that intercourse isn’t happening anytime soon due to bad trauma and bad experiences. Which he was cool with and said “it’s fine we don’t have to go that far”. Fast forward to like 1-2 weeks ago we have a conversation and he thought I meant that I will only have intercourse with someone if we are dating. To which he said “then let’s just be friends” which left me feeling really confused. And I explained that, that’s not what I said or meant I simply want to take it slow. And he said he’s fine with that and either way I’m not obligated to intercourse with him or anyone. The next day after that conversation we have call and he explains to me that intercourse is a huge factor for him in a relationship and when it comes to liking someone. And that it’s happened before where everything else is good but then when they get to bed they aren’t compatible and he can’t continue. Or in order for him to continue interest in someone intercourse plays a part. Which I can understand. But if he isn’t looking for a relationship why should sex with me matter? (We are exclusive, not dating. But we both also stated that we wouldn’t like it if one or the other went with someone else)(he also has a hard time communicating)