have been one of the first groups to shave regularly and widely, from what I have read. Then the French brought back shaving during the Enlightenment, in part to emulate the ancient Romans, but also as part of a broader “feminization” of men’s styles during that era (powdered wigs, tights, makeup, etc).
Even in relatively modern times, 12 of the first 27 US presidents had facial hair, including 10 of 12 between 1860-1913 — between the GOAT (Abe Lincoln) and Howard Taft, the last president to sport facial hair in office. The early POTUSes were heavily influenced by the feminized French Enlightenment style I mentioned above, and facial hair was scarce during that era. Among LDS Church presidents, 7 of the first 8 presidents had facial hair, from Brigham Young to George Albert Smith — so all of them from 1844-1951. And there was no lack of offspring produced among either of these groups.
In summary, it seems that men with facial hair have been sufficiently sexually attractive through most of human history, compared to clean-shaven men.