The pope is only infallible in matters of morals and faith. And a history of "bad popes" as they are called has done nothing to sully the church on matters of faith and morals.
There is also scriptural backing for this:
>"I have prayed for you that your faith may not fail" (Luke 22:32)
>"And so I say to you, you are Peter, and upon this rock I will build my church, and the gates of the netherworld shall not prevail against it." (Matthew 16:18)
You are thinking like a secular person, of the church as man-made political organisation and that kind of uniformity is probably dangerous in human affairs, but you forget that Christ promised his faith will not fail, regardless of how despicable a person the pope may have been in the past or may be in the future.
Another good proof of this:
>"49 Then one of them, named Caiaphas, who was high priest that year, spoke up, “You know nothing at all! 50 You do not realize that it is better for you that one man die for the people than that the whole nation perish.”
>51 He did not say this on his own, but as high priest that year he prophesied that Jesus would die for the Jewish nation, 52 and not only for that nation but also for the scattered children of God, to bring them together and make them one."
(John 11:49-52)
Notice how, even though Caiaphas was clearly antagonistic toward Christ, by virtue of being high priest that year he was able to prophesy Jesus' death for the Jewish nation (without even realising it himself). It says clearly that he did not say it on his own, but it came to him as high priest.
This is further illustrated here:
>"1 Then Jesus spoke to the crowds and to his disciples, 2 saying, “The scribes and the Pharisees have taken their seat on the chair of Moses. 3 Therefore, do and observe all things whatsoever they tell you, but do not follow their example. For they preach but they do not practice."
(Matthew 23:1-3)
Jesus tells them not to follow the example of the Pharisees, but notice how he says that because they sit on the "seat of Moses" that the Jews should do as they say. So even when dealing with one of the bad popes, we can be guaranteed of truth in matters of morals and faith (when speaking ex cathedra). Even if their behaviour should not be emulated.