"I sometimes prefer "non-white people" to "people of colour" even though you could argue the former is problematic 'cos it lumps lots of different sorts of people into one category and defines them by reference to white people. That's only a translation of what I really mean, though. To my friends and me, we are the people. White people are the Mat Salleh, the ang moh, the whatever term you like to use. (Not that they aren't people, obvs. They're just the other people.)"
Is how it works in the fandoms I'm used to.
You get a bit of this in sf when there are lots of alien races other than human. Despite the references to "Terra", there's generally a sense of: here's a buncha Klingons, here's a Vulcan, here's a Bajoran ... and here are humans, who are the people.
Because there are these fascinating people with different cultures that I want to look at, and then there's the Earthman in his shiny space ship, and the Earthman is presented as necessary: you /must/ have his normative gaze looking at the other people or there's no place for you the human reader in the text. I couldn't care less about the earthman, frankly. He irritates me as much as those Hollywood films set in Japan or China or Cambodia where there must be a white protagonist or else (it's assumed) a western default-white audience won't watch them. It has to be about US (pun intended) or it's not interesting.