How do I ask a coworker with few shared interests to hangout outside of work?

I have lunch with him 2-3 times a week and we get along pretty well. Now I look forward to hanging out with him outside of work sometime, but we barely have any shared interests and neither of us likes drinking or partying. The only option seems to be grabbing a coffee together or something (which we already do a lot at work), but it sounds quite weird to ask someone out with no real activites planned. What can we do together?

For the record, this is strictly a friendship situation.