This seems easy. Mileage to Sunday services is not deductible because, as you point out, that is not mileage for charity
but instead it is mileage for going to church where you also just happen do some charity. You’re not traveling more than any member of the congregation would. It’s hard to justify to the IRS that you are volunteering for a church that would wouldn’t even attend if you weren’t volunteering.
But pretty much all other travel for church callings that’s not the case. The vast majority of members of a ward don’t travel to YM activities or camps or service projects. Not even all parents of kids do. Traveling to volunteer to these activities is above and beyond what most members of the church do.
The IRS does specifically say that you can’t write off mileage used to go visit an individual person. But that’s not the case here.