|MadSci Network: Physics|
You correctly note that there are two physical processes involved in the interaction of light with the Earth's atmosphere. Let's review them briefly.
When light from the Sun or a star enters the atmosphere, it must pass through many tiny molecules (oxygen, water vapor, nitrogen, carbon dioxide, etc.) and many particles of dust and pollen. The dust particles are much larger than the molecules, and that size difference is crucial in determining what happens as the light goes past.
Visible light has wavelengths which range from about 400 to about 700 nm. The dust particles are usually at least this large, and often much larger. When light waves pass obstacles which are larger than their wavelength, the probability that they are blocked or scattered is inversely proportional to their wavelength: the longer the wavelength, the smaller the chance that the light ray will be blocked. So, for example, when the sun sets, its light must pass through a very long path through the Earth's atmosphere in order to reach us. That means that the longest wavelengths of light are the most likely to make it all the way without being scattered or blocked; and therefore, what we see is mostly reddish light.
If the Earth's atmosphere were thicker than it is now, and if that caused many more dust particles to float high in the sky than do now, we would expect our sunsets to become more dramatic. In fact, the Sun might disappear before it reached the horizon if there was so much dust that even the longest, reddest light rays could not traverse the long path of air without being scattered. We might start to notice the Sun changing color from yellowish-white at noon to orange and red by, say, 3 o'clock in the afternoon.
Now, the other sort of interaction you mention is Rayleigh scattering. This occurs when a light ray flies into a gas molecule. Remember that the light ray has a wavelength of around 400 to 700 nm, while an air molecule is much, much smaller: less than 1 nm in size. When waves encounter obstacles which are much SMALLER than their own wavelength, the probability that they scatter from the obstacle still depends on their wavelength; and again, light rays with short wavelengths are most likely to scatter. This means that blue light is more likely to bounce off an air molecule than red light.
Now, what happens when you go outside and look up into the sky? If you look directly at the Sun, you should see lots of light, and of course, you do (please don't stare at the Sun -- you could damage your eyes). What SHOULD happen if you look somewhere else? If you look, say, at the northern portion of the sky, when the Sun is in the southern portion, then you might expect to see .... no light. After all, ordinary air molecules don't emit light on their own -- they don't glow like the gas inside a neon lamp. But you will see some light coming from the northern sky: it is all light from the Sun which was originally going to strike the Earth somewhere far from your location (far to the north, in this instance). However, as those rays of light tried to reach the ground far from you, they ran into a gas molecule, and some of them bounced off it and flew away in a new direction. Some of those scattered light rays headed south and down, and ran into your eyes.
So, what you see when you look into the "blank" regions of the sky is light from the Sun which has scattered off air molecules. The light rays most likely to bounce off the molecules are the shortest ones: the blue light rays. Therefore, the daytime sky away from the Sun looks blue.
Now, what would happen on a planet with a much thicker atmosphere? If we increased the air density a little, then the regions of sky away from the Sun would still be blue. But if we increased the air density a LOT, the color would change. If the amount of dust in the upper atmosphere were to become very large, then the blue light rays from the Sun would be absorbed or scattered in the very, very high atmosphere; they might have to bounce or scatter several times before they could reach the ground. If the atmosphere were thick enough, they might rarely reach the ground. That would mean that the light rays we might see when we looked in the sky away from the Sun would be the shortest rays which could still reach the ground after scattering just once or twice. Those might be light rays with somewhat longer wavelengths: green instead of blue, or, if the atmosphere were even thicker, yellow. If the atmosphere were really, really thick, we might see a dull orange or reddish glow from the entire sky, because only those very long wavelengths of light could make it through the atmosphere to reach the ground at all.
Try the links in the MadSci Library for more information on Physics.