When I did I was a little stingy because some take advantage.
I would ignore when top performers were taking a long lunch or out for an appointment. If you were a middling to low producer I have a lot less tolerance for wasting working hours.
Of your manager is giving you hassle he might not be a bad manager. You might be an average to below average producer.