
	Where results make sense

Topic: Joint entropy

Related Topics

Conditional entropy

Mutual information

Rate distortion theory

Information theory

Quantum information theory

Hadamard transform

Coding theory

List of important publications in computer science

Algorithmic information theory

Data communications

Book of Kells

Gallery of Book of Kells pages

In the News (Mon 21 Dec 09)

Northern Trust

Marvin Harrison

Peter Chernin

Gary Locke

Match Play

Penelope Cruz

Mickey Rourke

AR Rahman

Danny Boyle

Hugh Jackman

	PlanetMath: Shannon's entropy
		Entropy is maximized when the probability distribution is uniform.
		Entropy in the continuous case is called differential entropy.
		This is version 21 of Shannon's entropy, born on 2001-11-19, modified 2006-06-28.
	planetmath.org /encyclopedia/Entropy.html (487 words)

	UNC Image Processing Software (Site not responding. Last check: 2007-10-14)
		The total entropy of an image is computed as H = -SUM[ p(i)*ln(p(i)) ], 0 <= i < M, where p(i) = h(i)/N Here, h(i) is slot i of the histogram having M slots, and N is the number of pixels in the image.
		Joint entropy is computed using the same formula, except that h(i) is a slot in the joint histogram computed for the two images.
		That is, for each input pixel, the output is the entropy (individual or joint), joint probabil- ity or mutual information of the input pixel or pixels.
	www.cs.unc.edu /~fritsch/html/entropy.1I.html (552 words)

	Joint entropy - Wikipedia, the free encyclopedia
		The joint entropy measures how much entropy is contained in a joint system of two random variables.
		The joint entropy is used in the definitions of the conditional entropy:
		In quantum information theory, the joint entropy is generalized into the joint quantum entropy.
	en.wikipedia.org /wiki/Joint_entropy (388 words)

	ONLamp.com -- Calculating Entropy for Data Miners
		The main difference is that the joint entropy formula has two summation signs instead of one, and the probability distribution being summed over is a joint probability distribution instead of a univariate probability distribution.
		The calculation of the joint entropy score is the same as the calculation of the univariate entropy score.
		The joint entropy score thus signifies the minimum number of questions you need to ask, on average, to identify a signal coming from this bivariate distribution of signals.
	www.onlamp.com /pub/a/php/2005/03/24/joint_entropy.html (622 words)

	Image Registration by Maximization of Mutual Information
		While entropy for an image remains fixed, joint entropy and mutual information of two images vary as the 1-1 correspondence between the pixels from each image changes with every geometrical alignment.
		Using the above definitions, joint entropy is calculated to be 3.00bit for Fig.
		9 demonstrates the behavior of the entropy as a function of the misalignment for the 2D display of a pelvic CT image shown in Fig.
	home.uchicago.edu /~khahn/PhD_works/summer2001.html (520 words)

	BBC - h2g2 - Entropy versus Information
		Intuitively, the entropy of something is a measure of its intrinsic uncertainty, chaos, or disorder.
		This doesn't mean that the entropy in the system has decreased - erasing the data has generated heat in the disk, and in all likelihood hair loss in the owner, so the physical entropy will have increased significantly.
		Distance is defined as the difference between the joint entropy and the mutual information of X and Y. It is highest for pairs of ensembles which have a high individual entropy, and which are completely unrelated.
	www.bbc.co.uk /dna/h2g2/A540505 (2098 words)

	Entropy
		Entropy, or average self-information, measures the uncertainty of a source and hence provides a measure of the information it could reveal.
		Entropy is a measure of uncertainty in a random variable and a measure of information it can reveal.
		The entropy rate is a measure of the uncertainty of information content per output symbol of the source.
	cnx.org /content/m10164/latest (780 words)

	Parametric Method for Correction of Intensity Inhomogeneity in MRI Data - Feature Story - Scientific Computing and ... (Site not responding. Last check: 2007-10-14)
		We computed joint entropy from joint histograms of the anatomical and various EPI data sets.
		To aid in joint entropy analysis, the joint histograms were blurred with the Parzen windowing technique.
		The 10% joint entropy decrease is large compared to the field map corrected method that produced only a 1% decrease.
	www.sci.utah.edu /stories/2002/sum_mri-epi.html (548 words)

	Related Work (Site not responding. Last check: 2007-10-14)
		They showed interesting scatter-plots of the joint data as the registration is disturbed, and used third order moments of the joint histogram, as well as other measures to characterize the clustering of the joint data.
		The use of joint entropy as a criterion for registration of CT and MRI data has been explored [Collignon et al.1995b].
		Several researchers have investigated the use of joint entropy to characterize the proper registration of medical imagery [Studholme, Hill and Hawkes1995b] [Collignon et al.1995a], and found that it is not a robust measure of registration, with [Collignon et al.1995a] describing difficulties associated with partial overlap of the data.
	splweb.bwh.harvard.edu:8000 /pages/papers/swells/mia-html/node17.html (937 words)

	PlanetMath: conditional entropy
		The results for discrete conditional entropy will be assumed to hold for the continuous case unless we indicate otherwise.
		Cross-references: inequalities, equalities, function, joint entropy, continuous random variables, continuous, conditional probability, discrete random variables, probability space
		This is version 5 of conditional entropy, born on 2002-02-21, modified 2006-03-01.
	planetmath.org /encyclopedia/ConditionalEntropy.html (91 words)

	Analysis of an Idealized Example
		The joint distribution is subject to the constraints that the marginal distributions equal the overall tissue probabilities, this leads to the row- and column-sum constraints that are listed in the table.
		Guided by the tissue contrast properties (shown in Table 5), this distribution is easily constructed from the one shown in Table 4, by merging the probabilities in the first two columns and the last three rows.
		This may be the cause of the increase in entropy we have observed in the joint signal when mis-aligned.
	splweb.bwh.harvard.edu:8000 /pages/papers/swells/mia-html/node15.html (796 words)

	Information theory (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		It is concerned with information entropy, communication systems, data transmission and rate distortion theory, cryptography, data compression, error correction, and related topics.
		Recently however, it has emerged that entropy was defined and used during the Second World War by Alan Turing at Bletchley Park.
		Entropy as defined by Shannon is closely related to entropy as defined by physicists.
	information-theory.kiwiki.homeip.net.cob-web.org:8888 (957 words)

	Disc-Entropy.nb
		We first define the entropy, conditional entropy, and mutual information for probability mass functions using the package at the end of this document.
		However, we can not compute the conditional entropy or mutual information for a single random variable.
		Here is another joint probability mass function to investigate.
	www.cs.wright.edu /~fdgarber/740/Discrete-Entropy/mma (215 words)

	Method
		possible values, the joint entropy is computed as in equation 6.4:
		The relation between the individual entropies and their joint entropy is given by equation 6.5:
		It expresses the fact that the joint entropy is always smaller then the sum of the individual entropies.
	tecfa.unige.ch /~lemay/thesis/THX-Doctorat/node148.html (91 words)

	Definition of Entropy and related functions
		The entropy is a measure of the information content or `uncertainty' of x.
		The question of why entropy is a fundamental measure of information content will be discussed in the forthcoming chapters.
		The joint entropy, conditional entropy and marginal entropy are related by:
	www.cs.toronto.edu /~mackay/itprnn/1997/l1/node23.html (230 words)

	Joint entropy (Site not responding. Last check: 2007-10-14)
		Often we are interested in the entropy of pairs of random variables (X,Y).
		Interpretation: The uncertainty (entropy) about both X and Y is equal to the uncertainty (entropy) we have about X, plus whatever we have about Y, given that we know X.
		We can also have a joint entropy with a conditioning on it, as shown in the following corollary:
	www.engineering.usu.edu /classes/ece/7680/lecture2/node2.html (92 words)

	[No title] (Site not responding. Last check: 2007-10-14)
		The joint entropy score also would jump up, given that the capture region of the joint entropy probing surface is very narrow.
		We expect that the joint entropy objective function will be the most useful in aligning the mri — mri_inv image pair because joint entropy measures the how consistently each the intensities in the two images match up.
		The joint entropy probing surface, on the other hand, has not changed, aside from scaling, so it maintains its minimum at the origin.
	web.mit.edu /sijia/Public/Sijia-Liu-6555-Lab3b.doc (3020 words)

	Cross entropy - Wikipedia, the free encyclopedia
		In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p.
		The cross entropy for two distributions p and q over the same probability space is thus defined as follows:
		NB: The notation H(p,q) is sometimes used for both the cross entropy as well as the joint entropy of p and q.
	en.wikipedia.org /wiki/Cross_entropy (320 words)

	[No title] (Site not responding. Last check: 2007-10-14)
		Given two highly correlated sources, X and Y, with respective entropies H(X) and H(Y), and joint entropy H(X,Y), the entropy reached by jointly encoding the two sources is called the joint entropy of the two sources, denoted by H(X,Y).
		A surprising fact, established -at least in theoratical sense- by Slepian and Wolf is that it is possible to encode the two highly correlated sources, X and Y, separately, and still reach the joint entropy, H(X,Y), by decoding them at a joint decoder.
		The ultimate goal is to reach the joint entropy, H(X,Y), rate suggested by Slepian and Wolf.
	www.stanford.edu /class/ee368b/Projects/ozonat/introduction.html (315 words)

	Information theory (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		Entropy as defined by Shannon is not related to entropy as defined by physicists.
		There are no relationships between entropy in the thermodynamic and informational senses.
		Some argue for instance, that Maxwell's demon needs Information to reverse thermodynamic entropy and getting that Information exactly balances out the thermodynamic gain that the demon would otherwise achieve.
	information-theory.iqnaut.net.cob-web.org:8888 (1110 words)

	Language and Information
		•The joint entropy of a pair of discrete random variables
		X, Y ~ p(x,y) is the amount of information needed on average to specify both their values
		•The conditional entropy of a discrete random variable Y given another X, for X, Y ~ p(x,y) expresses how much extra information is need to communicate Y given that the other party knows X
	tangra.si.umich.edu /~radev/LNI-winter2004/notes/1_files/slide0109.htm (61 words)

	Joint Entropy Maximization in Kernel-Based Topographic Maps -- Van Hulle 14 (8): 1887 -- Neural Computation (via ... (Site not responding. Last check: 2007-10-14)
		so as to maximize the joint entropy of the kernel outputs.
		entropy of the kernel output will be maximal.
		optimal joint entropy performance can be derived for the case
	intl-neco.mitpress.org.cob-web.org:8888 /cgi/content/abstract/14/8/1887 (200 words)

	Joint Entropy-Constrained Multiterminal Quantization - Cardinal, Van Assche (ResearchIndex)
		The first method is a simple descent algorithm, while the second is based on index...
		Cardinal and G. Van Assche, "Joint entropy-constrained multiterminal quantization," in Proc.
		Joint Entropy-Constrained Multiterminal Quantization - Cardinal, Van Assche (2002)
	citeseer.ist.psu.edu /671324.html (296 words)

	Joint Entropy (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		Illustrations of the notion of entropy in the two...
		Conservation and Entropy Domains of Free and Bound Energy...
		The Conservation Trinity: Energy, Symmetry, Entropy, and its relation to Gravity...
	www.scienceoxygen.com.cob-web.org:8888 /electrical/389.html (213 words)

	Entropy -- from Wolfram MathWorld (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		In physics, the word entropy has important physical implications as the amount of "disorder" of a system.
		Ellis, R. Entropy, Large Deviations, and Statistical Mechanics.
		Ott, E. "Entropies." §4.5 in Chaos in Dynamical Systems.
	mathworld.wolfram.com.cob-web.org:8888 /Entropy.html (173 words)

	Relative entropy and mutual information (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		The difference in the number of bits is denoted as D(pq).
		The quantity D(pq) comes up often enough that it has a name: it is known as the relative entropy.
		We can summarize a bunch of statements about entropy as follows:
	www.engineering.usu.edu.cob-web.org:8888 /classes/ece/7680/lecture2/node3.html (325 words)

	Joint Entropy-Constrained Multiterminal Quantization - Cardinal, Van Assche (ResearchIndex) (via CobWeb/3.1 ... (Site not responding. Last check: 2007-10-14)
		Joint Entropy-Constrained Multiterminal Quantization - Cardinal, Van Assche (ResearchIndex) (via CobWeb/3.1 planetlab2.netlab.uky.edu)
		Two design algorithms are presented, both optimizing a Lagrangian cost measure involving distortions and information rates.
		2.4: Joint Entropy-Constrained Multiterminal Quantization - Cardinal, Van Assche (2002)
	citeseer.ist.psu.edu.cob-web.org:8888 /532396.html (215 words)

	I (Site not responding. Last check: 2007-10-14)
		Measuring information, and the game of twenty questions
		Entropy, joint entropy, conditional entropy (2.1, 2.2, 2.5)
		What to do when source entropy exceeds channel capacity
	www.dam.brown.edu /people/geman/am171/Syllabus.htm (119 words)

	Hexapedia - Joint quantum entropy (via CobWeb/3.1 planetlab2.netlab.uky.edu) (Site not responding. Last check: 2007-10-14)
		The joint quantum entropy is an entropy measure which attempts to generalize the classical joint entropy for quantum information theory.
		The classical joint entropy is always at least equal to the entropy of each individual system.
		This is not the case for the joint quantum entropy.
	www.hexafind.com.cob-web.org:8888 /encyclopedia/Joint_quantum_entropy (384 words)

Try your search on: Qwika (all wikis)