What is the consensus on sex mixed with substance use?

I guess this question is mostly about a scenario in which one partner is under the influence and the other is not. Obviously, if you’re taking advantage of someone you’re not in a romantic or sexual relationship with while they’re drunk or high, that’s an absolutely abhorrent thing to do and is easily classified as rape, but would that still be true if it’s with someone you are in a relationship with? I don’t WANT to be doing that with my boyfriend necessarily (ideally we would both be under the influence in this scenario if substances are involved), I was more just wondering about it. I was thinking about the first time I used weed during which my boyfriend was completely sober and we eventually did some sexual things together. I was only a bit high and didn’t feel like my perception of reality was all that warped at all, so it doesn’t bother me. We had also been teasing each other for a while at that point, so it was pretty obvious where that was going to lead once we got private time together. I basically just thought, hey, he’s my boyfriend and also the only person I want to have sex with, not to mention I feel totally fine and up for it right now, so I’m good with this. I also thought of a more recent time where my boyfriend was drunk and I was just a bit crossed (though at this point I basically felt 100% normal, and I could definitely tell he was starting to sober up a little as well since it had been probably an hour or two since either of us had used any substances). My boyfriend was super horny and we were making out, but I didn’t let him do anything beyond touching my thigh. Maybe I’m being dramatic, but it just felt kind of… wrong? I didn’t want to take advantage of him, and as the “more sober” person in that instance, I felt responsible for not letting it go further. We were talking about it a few days later for whatever reason, and I learned he had pretty similar feelings to me that I did when I was high. He basically said he would be totally fine doing things like that with me if he was drunk or high, since I’m his girlfriend and everything. I told him things were the same on my end. I just know that I wouldn’t want to do something that I’ve never expressed comfort in if I were in an altered state of mind, so basically just anything I either wouldn’t want to do, or would want to talk about first, while sober. I also definitely wouldn’t want sex if him or I were, like, insanely drunk and out of it, obviously. Still, I know that if you’re under the influence, you can’t exactly consent, even though I would be totally okay with doing these things with my boyfriend if I was, like I said. Would it be best to just not have sex if both partners are at varying levels of sobriety, though? Or does it not matter too much if you guys are already in a relationship? Where would you draw the line?