A good rule of thumb for hats and sunglasses is " When in doubt, take it off.
To expand on this just a little apart from the obvious things like churches, ceremonies, etc.. : never wear a hat while eating, it's considered disrespectful to the person you are eating with. Also many of the older generation grew up in a generation where lice was very easily spread, so if you place your hat on a table in a restaurant, you're likely to piss a few people off.
Apart from that, I don't know anything about fashion so I can't tell you how to match colors.
To expand on this just a little apart from the obvious things like churches, ceremonies, etc.. : never wear a hat while eating, it's considered disrespectful to the person you are eating with. Also many of the older generation grew up in a generation where lice was very easily spread, so if you place your hat on a table in a restaurant, you're likely to piss a few people off.
Apart from that, I don't know anything about fashion so I can't tell you how to match colors.