I think Charlie is just the only "safe" choice to go with. Dieter is as much of a hermit as Rover is, Rover would probably think that Dumb would hulk out or do something while out there, no way in hell he'd take Duji, Rob's weird as fuck and Jeffrey...LOL .
I mean, the last time Rover took a girl on vacation with him, she ended up leaving. So maybe Rover takes Charlie for a couple reasons:
1) Charlie's loyal to his gf and knows Charlie wouldn't get in any trouble (or at least not too much)
2) Rover doesn't want to hook up with any girls and Charlie is good sex repellent
3) You're right, Rover and Charlie are secretly gay