He came back just as he had told them, when the temple was destroyed. It was destroyed in 70 AD.
Mat 24:1 And Jesus went out, and departed from the temple: and his disciples came to him for to shew him the buildings of the temple.
Mat 24:2 And Jesus said unto them,
See ye not all these things? verily I say unto you, There shall not be left here one stone upon another, that shall not be thrown down.
Mat 24:3 And as he sat upon the mount of Olives, the disciples came unto him privately, saying, Tell us,
when shall these things be? and what shall be the sign of thy coming, and of the end of the world?
They would not know the hour or the day, but they did know the temple would be destroyed.
Mat 24:15
When ye (disciples) therefore shall see the
abomination of desolation, spoken of by Daniel the prophet, stand in the holy place, (whoso readeth, let him understand
Mat 24:16 Then let them which be in Judaea flee into the mountains:
It happened in 70 AD, you and I didn't see it, but some of the disciples did.