To claim that a faster car has a "right" to the left lane in that situation is saying they have a "right" to speed. And, no, I'm not making them go the speed limit because A) I'm already going above the speed limit and B) the lane to the right is open.
Freak out if you wish.