Person, the more natural automatic commonality you feel. So things can blend over emotionally if not logically. Also, Europeans inventing and putting people on a racial colored wheel in the first place, ended up (maybe artificially) making people closer on the wheel feel closer together in real life.
I don't see any historical movies of Polynesian Plantation owners in the slave owning south calling black people the n word. Or bringing them across the Atlantic. But I do of white people. So white people would be at the top of black peoples list of those who can't use it. After that, it's not so black and white.