If you are in the express lane and you are constantly getting passed by people in the adjacent left lane and/or you have people right behind you and the highway has plenty of free lanes, you should exit the express lane and move to one of the right lanes on the highway with slower traffic.
If there is congested traffic and you are going faster than regular traffic and you are going the speed limit, you are fine staying in the express lane.
The purpose of the express lane is not to create a comfortable safe lane for slow traveling carpoolers during non-busy hours.
Common sense