View Full Version : Maximum Entropy

02-11-2008, 07:02 PM
I just stumbled upon this Generalized Maximum Entropy Home Page (http://are.berkeley.edu/~perloff/gme/index.html).

Anyone knows what maximum entropy is?

02-11-2008, 07:10 PM
No idea, something from Physics.. there is also a related book

Amazon.com: Maximum Entropy Econometrics: Robust Estimation with Limited Data: Books: Amos Golan,George G. Judge,Douglas Miller (http://www.amazon.com/Maximum-Entropy-Econometrics-Estimation-Limited/dp/0471953113)

02-11-2008, 08:00 PM
It is connected with second law of thermodynamics. It can be interpreted as a measure of chaos (or order) in the system. More order, lower entropy and vice versa.
For example you have two bowls, in one there are only red balls, white in the second. This system is very "organised". If you mix it, and randomly put the balls back into bowls, the colors will be mixed and the system will have much lower degree of "order" or higher entropy.
This "cause" that some natural processes are irreversible. For example, if oyu have two objects with different temperature, then you can increase the temperature of the colder but not the warmer, even though the total energy would be the same.
If you let your room, then there will be an inevitable tendency towards mess, which can be avoided only by some work :D

02-11-2008, 10:17 PM
From our future artificial overlord:

Principle of maximum entropy - Wikipedia, the free encyclopedia (http://en.wikipedia.org/wiki/Maximum_entropy)

02-11-2008, 10:40 PM
From our future artificial overlord

I have really come full circle on wikipedia. It's great for quick information but it is replete with errors and misinformation as well. Worse, if you're reading the article, you probably don't know enough to notice the errors. I have found the combination of amazon.com (to discover new books) and my university library to be a lethal combination. (lethal to the problem at hand that is....)

Furthermore, few people seem to use the library extensively these days. I have a collection of about 50 library books which I have checked out over the years. Some I have had for 3 years now. [my library allows perpetual renewal as long as there isn't an outstanding request]. Unfortunately when I graduate all my precious books will have to good back to their cold former lives on the stacks....

02-11-2008, 11:56 PM
I happen to know some about this cuz I've been working on this for a while. Followings are some random bits of my knowledge. May contain a lot of errors.

1. 'entropy' here means degree of unorderedness(or chaos), as in thermodynamics.

2. the concept of so called maximum entropy principle, however, has been developed since 1940's by a group of people studying statistical dynamics. Shannon defined the measure of entropy for a probability density, which of course can be derived axiomatically with 'good' properties.

3. Maximum entropy principle provides a way to 'estimate' (I would rather say 'characterize') probability density with only limited information on hand. That is, to find a pdf that maximizes entropy under the constraint of information on hand. The information here is typically given as a form of moment conditions.

4. As you may notice, the maximum entropy principle can be applied to the same problems that GMM can approach. Kitamura and Stutzer(guess it's 1997 econometrica) proposed a semiparametric estimation method using maximum entropy principle. Their method is basically the maximum empirical likelihood estimation, but with empirical likelihood probabilities obtained by maxmimum entropy principle.

5. There are many other application of the maxent principle. It seems like it is mostly applied in finance literature, extending ARCH type models, obtaining state price density and so on.

03-13-2008, 04:31 PM
hi julius,

thanks for sharing the concepts of GME.
if i have a ready pdf (obtained by plotting exptl data set) of the frequency of occuerence of a particular event (in a time series), how can i calculate the GME of my pdf ?
can u give some more details about it. also, some easy papers/books (preferably easily found in internet) would be of great help t start for me. i am not from a u pure economics background.

thanks & hoping for ur reply.

03-14-2008, 03:42 AM
This is a total aside, but I can't help it: as I was reading Critical Mass (http://www.amazon.com/Critical-Mass-Thing-Leads-Another/dp/0374530416/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1205465738&sr=8-1), I stumbled on these considerations on enthropy and the like. I readily thought that sooner or later some guy would translate this into an economic concept, just like we did with Mechanics. It was already done as I can see now.

03-20-2008, 01:34 PM

can anyone inform me how can we apply GME to our problem ? what are the basic equations and the methodology to be followed ?

thanks in advance.