Subject: Why do light detectors always square the electric field ?

Light detectors generally are said to produce an output that corresponds to the 
integral over a more or less short period of time of the SQUARED electric field 
component of the incoming electromagnetic (em) wave.
Is there an explanation from the microscopic point of view for the fact that 
these detectors always take the square of the electric field ? Is it pure 
conincidence that light detectors such as eye, photo-plate, photo-diode etc. 
produce a "reaction" to the incoming em field that is proportional to the 
square of the electric field and at the same time the electric energy itself of 
the em field is also proportional to the square of the electric field ?
Also, why is the electric energy defined as the square of the electric field in 
the first place ?
Please do not refer me to the Poynting vector since the amount of energy we 
actually measure by detectors is only related to the square of the electric 
field (i.e. electric energy) but not to the length of the Poynting vector.
If necessary it is fine with me to go into any quantum mechanical details to 
answer this question. Thank you.

Re: Why do light detectors always square the electric field ?