How far in advance should you ask someone to hang out if you are coming from out of town?

I am going to ask a coworker of mine to hang out, when I am in their town (around 3 hours away from mine) next month. However, I am not sure how far in advance I should ask: my boyfriend is advising me to ask ASAP, so three weeks notice, whereas I was going to wait until 10 days beforehand.

Reasons to ask ASAP:
– Coworker might make alternative plans if I don’t do it soon and I am only in town for one weekend
– Asking people to hang out is a nice gesture and if someone picks apart how you have done it, maybe they’re not someone you want to hang out with anyway – so I haven’t lost anything by fumbling how I do it a bit
– I have a reputation for being an overly organised person and my coworker knows this, so it wouldn’t seem out of character to ask in advance
– My boyfriend plans dinners with his friends months in advance and considers this necessary to get a Yes, since his friends’ lives are so busy

Reasons to wait:
– Worry I might seem desperate to hang out with coworker and freak them out – I want to play it cool
– I am only asking them to go for a beer, so 3 weeks seems excessive
– I only just saw coworker at a work event, so again, fear of looking desperate and clingy
– I guess just putting it off because I worry something might happen at work that makes me change my mind about seeing them outside of it, or because I don’t want to face potential rejection
– Part of my reason for the invite is to signal that I would like to get to know them outside of work, which would happen even if they can’t make the actual invite.

I might also go for the middle ground – ask vaguely 3 weeks in advance, but give them my number to make actual plans closer to the time rather than planning the whole thing 3 weeks in advance. What would you advise doing in this situation?