Abraham believed that he would have a son. When that didn't happen for a while, they figured maybe it wasn't through Sarai.
Gen. 16:2 And Sarai said to Abram, “Behold now, the Lord has prevented me from bearing children. Go in to my servant; it may be that I shall obtain children by her.” And Abram listened to the voice of Sarai. 3 So, after Abram had lived ten years in the land of Canaan, Sarai, Abram's wife, took Hagar the Egyptian, her servant, and gave her to Abram her husband as a wife.
That wasn't God's plan and God set him straight.
Gen. 17:17 Then Abraham fell on his face and laughed and said to himself, “Shall a child be born to a man who is a hundred years old? Shall Sarah, who is ninety years old, bear a child?” 18 And Abraham said to God, “Oh that Ishmael might live before you!” 19 God said, “No, but Sarah your wife shall bear you a son, and you shall call his name Isaac. I will establish my covenant with him as an everlasting covenant for his offspring after him.
The promise was to his offspring, singular, which is Christ as Paul tells us.
Which promise?