Age gap and maybe going too fast?

I am 29F and i went to this trip with a group tour and met this one guy who i clicked with towards the end of the trip. He is 46 though… he would always say during the trip that I probably wouldnt talk to him anymore when we get back. But the group has been meeting a lot since the tour since everyone just gets along well together. Since we got back, we usually hang out with the group. But we do message each other privately and have expressed that we enjoy each other’s company and like each other. But i did tell him that i wanna take things slow since my fiance and I just broke up. He thinks the same way too and is very respectful. Its been a month since our break up and im just enjoying being on my own. Yesterday though, i brought him some food to his condo since hes been doing favors for me (hes a surgeon so hes been making prescriptions and check ups for me). I was just going to stay for a bit then i stayed from 4pm til 1am. We were watching a movie, cooking, and chatting. Then he started massaging my back (since he says hes really good at anatomy). Then he started hugging me and cuddling. I told him i felt a little weird by it so he stopped but when he did stop, i wanted him to hug me more. So we ended up hugging and i fell asleep on his chest before having to leave. We both agreed to shelf the situation and pretend nothing happened the next day. But now its been kinda hard to do that. He doesnt think i should be with anyone (even him) now bc my break up is still fresh and we both think i should spend time on my own. But we just enjoy each others company.