Yes, people have to right to be both stupid and foolish. And so they have the right to be racist. Bigoted even.
We (should) have the right to associate with whom we chose, employ whom we chose, do business with whom we chose and even discriminate against whom we chose. These rights (should) extend to everyone.
I think the confusion here stems from someone, somewhere having deciding to create the "right" to be free of discrimination, as if rights could be created by fiat. Granted, it's worked well in combating the violations of human rights by the runaway bigotry of the past but now we're stuck with a ridiculous idea we can't seem to shake lose of. And so our efforts to grow, as a society, out of this particular stupidity have been stifled here just on the other side of that.
Our society should have long since reached the fabled "post-racist" mentality. I'm convinced we're stuck where we are because we can't get past this one particular point. By denying ourselves and one another that natural inclination toward discrimination, rather than opposing it with wisdom and reason, we create bitterness and anger, and inevitably bigotry.
In a post-racist world a person who refuses to do business with those of a particular race is an idiot. He's a weirdo you probably wouldn't be wise to do business with yourself. Anyone dumb enough to cull their own market on such a meaningless basis isn't going to be in business long.
In our modern world that man is forced to do business with those he'd much rather not. That makes him bitter and angry toward whomever he's forced to do business with. Other business people see this and feel threatened. They may be forced to do business against their own determination themselves. And finally those whom the man have been forced to do business with are themselves forced into a position where they feel like victims who require championing, as if they have no power to champion themselves.
I can't see how this can possibly be considered an acceptable model.