|MadSci Network: Physics|
Your question is a good one, and to answer it, we have to get down to the molecular level of substances, as you have already surmised. This is because the wavelengths of photons are on the same scale as the size of molecules. Because the molecular structure of every substance is unique, the way in which that structure interacts with photons is also unique.
The index of refraction of a substance correlates to the speed at which photons propagate through it, but to know how to quantify the speed, we must first understand the physics behind this. When massless photons enter a substance, they bump into the atoms making up the molecules of the substance. The atoms absorb the energy from the photon which, in turn, causes the electrons in the atom to become excited and vibrate. Most of the time, this energy is then re-emitted by the atoms. The speed at which the photon is absorbed and re-emitted can be quantified using quantum mechanics or estimated experimentally. Scientists refer to this rate as the optical density of the substance. Optical density is directly proportional to the index of a substance.
In general, optical density follows the same trend as physical mass density, but this is not always the case. In the table below, you'll see several examples of where this general rule is broken:
These experimental results illustrate that mass density alone is not enough to quantify optical density. This is due to the complexity of molecular interactions with light and tends to produce non-intuitive results because of the atomic physics involved.
For more on this subject, this website from The Physics Classroom gives an overview of how photons interact differently with substances.
Hope this helps,
Try the links in the MadSci Library for more information on Physics.