>On this tradition and on the indisputable preëminence of Peter in the Gospels and the Acts, especially the words of Christ to him after the great confession (Matt. 16:18), is built the colossal fabric of the papacy with all its amazing pretensions to be the legitimate succession of a permanent primacy of honor and supremacy of jurisdiction in the church of Christ, and—since 1870—with the additional claim of papal infallibility in all official utterances, doctrinal or moral. The validity of this claim requires three premises:
1. The presence of Peter in Rome. This may be admitted as an historical fact, and I for my part cannot believe it possible that such a rock-firm and world-wide structure as the papacy could rest on the sand of mere fraud and error. It is the underlying fact which gives to fiction its vitality, and error is dangerous in proportion to the amount of truth which it embodies. But the fact of Peter’s presence in Rome, whether of one year or twenty-five, cannot be of such fundamental importance as the papacy assumes it to be: otherwise we would certainly have some allusion to it in the New Testament. Moreover, if Peter was in Rome, so was Paul, and shared with him on equal terms the apostolic supervision of the Roman congregation, as is very evident from his Epistle to the Romans.
>2. The transferability of Peter’s preëminence on a successor. This is derived by inference from the words of Christ: "Thou art Rock, and on this rock I will build my church, and the gates of Hades shall not prevail against it."316 This passage, recorded only by Matthew, is the exegetical rock of Romanism, and more frequently quoted by popes and papists than any other passage of the Scriptures. But admitting the obvious reference of petra to Peter, the significance of this prophetic name evidently refers to the peculiar mission of Peter in laying the foundation of the church once and for all time to come. He fulfilled it on the day of Pentecost and in the conversion of Cornelius; and in this pioneer work Peter can have no successor any more than St. Paul in the conversion of the Gentiles, and John in the consolidation of the two branches of the apostolic church.
3. The actual transfer of this prerogative of Peter—not upon the bishops of Jerusalem, or Antioch, where he undoubtedly resided—but upon the bishop of Rome, where he cannot be proven to have been from the New Testament. Of such a transfer history knows absolutely nothing. Clement, bishop of Rome, who first, about a.d. 95, makes mention of Peter’s martyrdom, and Ignatius of Antioch, who a few years later alludes to Peter and Paul as exhorting the Romans, have not a word to say about the transfer. The very chronology and succession of the first popes is uncertain.
http://www.ccel.org/ccel/schaff/hcc1.i.IV_1.26.html History of the Christian Church, Volume I: Apostolic Christianity