Since you believe, wrongly I might add, these monuments glorify our racist past, let's tear down the Lincoln memorial in Washington DC as well....
Tear down a monument to the preserver of the Union and the man who, his own failings notwithstanding, ended slavery? That wouldn't make much sense, would it? A bit like defending monuments commemorating the men who attempted to destroy that union and perpetuate slavery. History isn't the issue. How we frame it and what we honor is.
What I believe is that monuments to Confederate leaders, largely in uniform and ready for battle, send the wrong message and have, historically, reflected and sponsored a noble sense of an ignoble struggle, which in turn went some way to supporting the worst vestiges of it within the culture. These men served a racist, dehumanizing cause, whatever their personal feelings it is that linkage that condemns their commemoration. It is a cautionary tale at best.