There *are* people who look good in them- lean, sharp-featured, close-cropped men; and I am none of those so I look lousy. But baseball caps are perfect for keeping the sun out of one's eyes when bicycling, far better than sunglasses which, when they work, make me, at least, dangerously blind. They also block the glare in my catracted eye so now I wear them all the time, indoors and out, which I hate. But still.
My caps all have dragons on them, with large hanzi as well, and for a long time I was a bit shy about wearing them into Chinese restaurants and such, and terribly self-conscious when I had to talk to Chinese prospective parents at work. But either the hanzi'd caps or my silver hair seem to set the Chinese I meet at ease; not sure which, but I'm the one people want to talk to and ask directions of and so on. (Then again, maybe I just have one of those direction-asking faces, because little old ladies in Tokyo also asked me the way to the Metropolitan Art Centre and the Splinter-pulling Jizou and the Sobu platform and such.)
What I find odd though is my instinctive cap-wearing behaviour. The etiquette of hats is something that died out, oh, a good fifty or sixty years ago; we Boomers didn't wear hats and that was that, until we started again, sort of, in the 90s. By then no one took their hat off when passing a church or meeting a lady or whatever. My notion of how to wear a hat comes from Victorian novels and it applies only to men, because AFAIK women didn't take their hats off in public at all.
But nonetheless, when meeting a stranger or addressing a shop clerk, when I want to be polite, my iron instinct is to take my baseball cap off-- to doff my cap as a friendly gesture. It just feels the right thing to do. I have not-- quite-- got to the point of touching the brim of my cap, but give it time and I just might.