On the financial aid (which is available to all student body) ands makes it so these athletes of non Olympic sports might only pay 20% of room and board because of their financial support from their parents. Then athletic department only need to give a quarter scholarship to some male swimmer and he is making a little cash.
Of course the endowment is healthy from the alumni but it benefits the whole student body. Not just the athletes. They AD office is not pumping money into Olympic sports