Oakland, they used to be the "San Francisco Warriors" and deliberately declined to ever use "Oakland" hence "Golden State."
And even then, they never left the Bay Area - which is kind of the complexity of identity in that market. The 49ers play in Santa Clara, not San Francisco.
As for the Raiders - they've been controlled by a mid gangster family for all of their modern existence and got jerked between LA, Oakland, and eventually Vegas.
But yes - on top of all that Oakland has problems. But the A's fan base was never just Oakland.