Agreed. It is very frustrating when you get stick behind someone riding somewhere they shouldn't be. But the bigger concern of course should be theirs for their safety.
As a young rider, in the story I related above, there were times when I had no choice but to ride on roadways with no shoulder or bike lane. I worked graveyard shift at a convenience store at the entrance of a subdivision that was designed with only ONE entrance, and that entrance was on a major frontage road to a freeway. If you're not from Texas you might not understand what that means, but it's basically part of the freeway structure, but with business and street access though typically not direct residential access. What that means of course is I had to book it for about a mile along with high speed traffic. There were 3 lanes so people could pass, however, I had to be highly alert of traffic coming up behind constantly. More than once I had to ditch over the curb into the grass. Had one guy see me at work later and tell me "Man I wasn't gonna hit ya", but of course, when people run up right behind you at speed, you can't take the time to see if they're going to go around or not, you have to be ready to ditch.
I took my life in my hands every night at 11pm and every morning at 7am for nearly a year. I learned to ride VERY fast for that 1 mile - my sprint would usually be 25-40 mph, impressive considering I was not (and am not) a small fellow.
The issue is, our infrastructure shouldn't put cyclists and motorists into that kind of conflict, but it does. It's easy for people to talk about rights and freedoms, and they do have those rights and freedoms. But at the end of the day do you want to go home alive or in a box? Take care and be safe, even if it's not the most convenient thing, and even if it means not practicing for the Tour de France on open roads where traffic is an issue. Just my opinion.