papers: extend topic model to deal with temporal dependency

Using topic models to analyze the tread and change of topics along the time line in a document corpus is definitely a cool idea. It has plenty of potentials in video analysis, human action understanding, etc. The following are a few papers related to this idea:

• Dynamic Topic Models, ICML 2006
• Continuous Time Dynamic Topic Models, UAI 2008
• Hidden Topic Markov Models, AISTATS 2007

non-parametric models:

• an HDP-HMM model is described in Yee Whye Teh's HDP paper
• An HDP-HMM for Systems with State Persistence, ICML 2008
• Infinite Hierarchical Hidden Markov Models. K. Heller, Y.W. Teh and D. Gorur. AISTATS 2009

Here is a good discussion on several infinite HMM:
a blog by Jurgen Van Gael, discussing several infinite HMM

Hierarchical Dirichlet Process: A Gentle Introduction

I wrote up a gentle introduction on HDP based on my two previous post:
Chinese restaurant process and Chinese restaurant franchise
from DP mixture to hierarchical DP mixture

The PDF file is here.

Chinese restaurant process and Chinese restaurant franchise

An illustration of Chinese restaurant process (CRP) and Chinese restaurant franchise (CRF). Materials are from Yee Whye Teh's 2004 technical report on "Hierarchical Dirichlet Process"

Chinese restaurant process (CRP):

For a set of random variables distributed according to , we have the following conditional distribution:

This can be described in a Chinese restaurant process metaphor:

• Consider a Chinese restaurant with an unbounded number of tables, 
• The first customer sits at table 1
• Suppose there are K tables occupied before the i-th customer comes, he can sit at 

            

The relationship between and can be best illustrated as in the following figure.

random variables	meaning	metophor
	random variables	customer i
	distinct values within all	table k
	the number of  associated to	the number of customers sitting around table k

Chinese restaurant franchise (CRF):

An essentially two-level Chinese restaurant process:

• Within a restaurant, customers   choose tables
• Within all restaurants, tables  choose dishes

In both levels, the choosing follows the Chinese restaurant process as illustrated above.

At the restaurant level, customers   choose tables according to the following distribution:

At this level, the metaphor of CRF is the same as the one of CRP as described above, except some changes on symbols

• Consider restaurant j with an unbounded number of tables, 
• The first customer sits at table 1
• Suppose there are  tables occupied before the i-th customer comes, he can sit at 

            

The relationship between and can be best illustrated as in the following figure:

At the franchise level, table   choose dishes according to the following distribution:

This can be described in a Chinese restaurant franchise metaphor:

• Consider a Chinese restaurant franchise, whose J restaurants share a menu with unbounded number of dishes, 
• At each table of each restaurant, one dish is ordered from the public menu by the first customer who sits there, and it is shared among all customers who sit at that table. Multiple tables at multiple restaurants can serve the same dish
  
• Suppose there are  tables occupied before the i-th customer comes restaurant j and there are total K dishes has been ordered among all restaurants in the franchise. He can sit at an occupied table or a new table with certain probability, as described above. If he sits at an occupied table, he shares the dish that has been ordered at that table. If he sits at a new table, he order a dish for that table according to its popularity among the whole franchise, while a new dish can also be tried. 

            

The relationship between and can be best illustrated as in the following figure.

random variables	meaning	metophor
	random variables	customer i in restaurant  j
	distinct values of in group j	table t in restaurant j
	index of associated to ,	the table taken by customer i in restaurantj,.i.e., Table()=, Customer() =
	the number of associated to in group j	the number of customers sitting around table t in restaurantj
	distinct values in within all groups,	dish k, which is shared within all restaurants
	index of associated to ,	the dish ordered by table t in restaurantj, i.e., Dish() = , Table() =
	the number of associated to in group j	the number of tables ordered dish k in restaurantj
	, i.e., the number of associated to over all j	the total number of tables ordered dish k within all restaurants