|MadSci Network: Chemistry|
The color of a liquid has to do with the molecular structure of the liquid and the whether it absorbs energy in the visible region. If you have three liquids of different colors it must mean that they are of different molecular structures (different chemicals). You can think of blood, for example. It is red because of the hemoglobin present in it. This hemoglobin has iron among other things and this causes it to be red. Water is colorless since it does not have any molecular group that absorbs light in the visible region. When you mix blood and water you will just get a solution that is colored a "lighter red". This because water and blood can mix well. The reason why things mix also has to do with molecular structure. For example, oil and water don't mix because, oil has fatty acid groups which are not compatible with water. So if you shake a mixture of water and oil you will find two layers or you will find oil droplets in water or water droplets in oil. This phenomenon is called liquid-liquid phase separation. This is what happens when you say the three different colored liquids don't mix. They do not mix because of ternary (three liquids) liquid-liquid phase separation. One can explain this separation based on thermodynamics. Every system (atom, collection of atoms, molecule, collections of molecules etc.) has a certain energy associated with it called free energy. This system will be stable (or will not change its status quo) when the free energy of the system is at a minimum. In you case, the "system" is the "mixture of three differently colored liquids". They do not mix and are found as separate layers because, only in this condition will its free energy be at a minimum.
In summary the basic reason for the three liquids do not mix is because they cannot have a minimum free energy in the mixed state. Whether the mixed state or the separated state represents a state of minimum free energy will be completely determined by the chemical nature of the three liquids. For example if you dissolve some red ink, yellow ink and some blue ink with water and mix the three liquids together you will find a single colored liquid results. The exact color of the final liquid will depend on the amounts of red, yellow and blue inks in the solution. This happens because ink is compatible with water. Whatever the chemical structure of ink is, it is compatible with the hydroxyl groups in water and hence the free energy will be a minimum in the mixed state. I am not telling you what the implications of free energy are and how does one find out what it is, because it is a graduate level subject.
The ordering of the layers has to do with the density of the different colored liquids. The liquid with the most density settles at the bottom and the lightest one remain at the top.
If you know what the three liquids are in terms of chemical structure then it is possible to explain why they don't mix. Otherwise, only a general explanation such as the above must be given.
Hope this helps.
References: (1) P.W. Atkins, Physical Chemistry, 4th Edition, W. H. Freeman & Co., NY, 1990
Try the links in the MadSci Library for more information on Chemistry.