But then, the authority of Jesus was passed on the apostles, who didn't write everything they taught, but gave it to their successors. It was passed on to the Church, not to individuals- the authority was not meant to rest exclusively in a book.
Hence, the Apostolic Office of the Church. The Pope is the successor of Peter, and this is a virtually unchallenged fact by everybody including the Eastern Orthodox who split and the Anglicans. It's only anti-Catholics who deny that inevitability.
You want the Church spoken of in Matthew 16, the biblical, historical Church? There it is :wave: