Within the laws of this country and state and city of New York, a U.S. Citizen or a Foreign National has the right to build a religious center for any faith they want provided it complies with the laws. Based on the laws, there is almost no legal reason to refuse them permission. Since building departments are charged with maintaining the public safety, they may be able to quash the permits by stating construction of such a controversial center so near to ground zero could lead to increased police costs to maintain order and prevent violence the Islamic center and its occupants. The costs of such increased security and the threat to public safety out weigh the owners rights to construct the center and the permits are not issued.
Since this thread asked how I feel, I feel that it is an incredibly stupid and insensitive move on the part of the Iman and the Islamic community.