Some was to be sure. Much wasn't. The wealth in my neck of the woods, for example, was NOT built on slavery. In fact my great-great-grandfather risked his wealth and his life to end slavery.
Due primarily to disease.
Sometimes, yes
But now with modern definitions and concepts of consent, you have the peculiar situation that my children were conceived through rape. The rape of me.