charitable service, but would be driving there anyway even without the charitable service (e.g., to take a child there), the miles are not incremental and not a result of your charitable efforts.

Actually, I don't even think it's absurd. Imagine the opposite rule where every SS teacher and greeter deducts their mileage to/from church every Sunday because they were providing service while there. That would be silly.