Well, the answer is that they do (diffuse into and out of the mitochondria), through the outer membrane.

But that is not were the "action" is. Based on our understanding of mitochondrial origins, the outer membrane of the mitochondria is derived from the ancient eukaryotic host, while the inner membrane is derived from the endosymbiont, an aerobic bacterium

It is across this inner mitochondrial membrane that the electron transport system is located. this is where electrons from reduced NADH (the product of the Krebs cycle) are pass to the ETS and used to generate the H+/pH gradient across the inner mitochondrial membrane.

This electric field and the H+ gradient combine to derive the the rotary ATP synthase to convert ADP + Pi into ATP (this same reaction occurs in the cytoplasm of aerobic bacteria, and a homologous protein is used in chloroplasts, where the H+ gradient is generated through a similar electron transport chain, using NADPH generated through light adsorption).

So, while the outer membrane of the mitochondria is permeable to H+, it only matters that there is a H+ gradient across the inner membrane, and that the movement of H+ across that membrane occurs primarily through the H+-driven ATP synthetase.

Of course, reversing the ATP synthetase can generate a H+/pH gradient.