Claude Shannon says:
The entropy is a statistical parameter which measures in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy H is the average number of binary digits required per letter of the original language.
-- NIST Special Publication 800-63-1 - Appendix A: Estimating Password Entropy and Strength