Your kids are already keeping score. No way to avoid that imo.
We were in a similar position where my in-laws came into some wealth later in life and started taking trips to Europe with just the single kids. It really bothered us for a while, but we got over it.