Sunday, September 20, 2009

a good review article for LDA

I happened to find a review article for LDA and its application for text, vision and music.
The link is Latent Dirichlet Allocation for Text, Images, and Music
and the slides is here

They are worth to read carefully.

Friday, September 18, 2009

papers: employing semantic hierarchy in object recognition

Semantic hierarchy could play an important role in object recognition. For example, if we know mini-van is a type of car, and we have already a model for car vs. the rest of the world, then we only need to differentiate mini-van from the car, which reduces lots of work. Similar idea has been noticed by object recognition researchers and there are several papers in recent years:
where some are listed in Trevor's course page.

Thursday, September 17, 2009

papers: supervised or discriminative topic model

Topic models are originally designed for topic discovery/clustering, not for classification. To use topic models for classification task, we have modify the structure of the topic model to add the class label and use it to bias the topic discovery process.

the following paper present some supervised/discriminative topic models by machine learning guys:
  • MedLDA: Maximum Margin Supervised Topic Models for Regression and Classification, ICML 2009
  • DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification, NIIPS 2008
  • Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora, EMNLP 2009
  • Supervised topic models, NIPS 2007
some supervised/discriminative topic models by computer vision guys:
  • A Bayesian Hierarchical Model for Learning Natural Scene Categories, CVPR 2005
  • Simultaneous Image Classification and Annotation, CVPR 2009
  • Spatially coherent latent topic model for concurrent object segmentation and classification, ICCV 2007
  • Towards Total Scene Understanding Classification, Annotation and Segmentation in an Automatic Framework, CVPR 2009
  • What, where and who? Classifying events by scene and object recognition, ICCV 2007
  • Learning Hierarchical Models of Scenes, Objects, and Parts, ICCV 2005

Wednesday, September 16, 2009

papers: extend topic model to deal with temporal dependency

Using topic models to analyze the tread and change of topics along the time line in a document corpus is definitely a cool idea. It has plenty of potentials in video analysis, human action understanding, etc. The following are a few papers related to this idea:

  • Dynamic Topic Models, ICML 2006
  • Continuous Time Dynamic Topic Models, UAI 2008
  • Hidden Topic Markov Models, AISTATS 2007
    non-parametric models:

    • an HDP-HMM model is described in Yee Whye Teh's HDP paper
    • An HDP-HMM for Systems with State Persistence, ICML 2008
    • Infinite Hierarchical Hidden Markov Models. K. Heller, Y.W. Teh and D. Gorur. AISTATS 2009
    Here is a good discussion on several infinite HMM:
    a blog by Jurgen Van Gael, discussing several infinite HMM


    Monday, September 14, 2009

    Friday, September 11, 2009

    Chinese restaurant process and Chinese restaurant franchise

    An illustration of Chinese restaurant process (CRP) and Chinese restaurant franchise (CRF). Materials are from Yee Whye Teh's 2004 technical report on "Hierarchical Dirichlet Process"

    Chinese restaurant process (CRP):

    For a set of random variables distributed according to , we have the following conditional distribution:

    This can be described in a Chinese restaurant process metaphor:
    • Consider a Chinese restaurant with an unbounded number of tables, 
    • The first customer sits at table 1
    • Suppose there are K tables occupied before the i-th customer comes, he can sit at 
                

    The relationship between and can be best illustrated as in the following figure.
    random variables
    meaning
    metophor

    random variables customer i

    distinct values within all   
    table k

    the number of  associated to
    the number of customers sitting around table k

    Chinese restaurant franchise (CRF):

    An essentially two-level Chinese restaurant process:
    • Within a restaurant, customers   choose tables
    • Within all restaurants, tables  choose dishes
    In both levels, the choosing follows the Chinese restaurant process as illustrated above.

    At the restaurant level, customers   choose tables according to the following distribution:


    At this level, the metaphor of CRF is the same as the one of CRP as described above, except some changes on symbols
    • Consider restaurant j with an unbounded number of tables, 
    • The first customer sits at table 1
    • Suppose there are  tables occupied before the i-th customer comes, he can sit at 
                

    The relationship between
    and can be best illustrated as in the following figure:


    At the franchise level, table   choose dishes according to the following distribution:


    This can be described in a Chinese restaurant franchise metaphor:
    • Consider a Chinese restaurant franchise, whose J restaurants share a menu with unbounded number of dishes, 
    • At each table of each restaurant, one dish is ordered from the public menu by the first customer who sits there, and it is shared among all customers who sit at that table. Multiple tables at multiple restaurants can serve the same dish
    • Suppose there are  tables occupied before the i-th customer comes restaurant j and there are total K dishes has been ordered among all restaurants in the franchise. He can sit at an occupied table or a new table with certain probability, as described above. If he sits at an occupied table, he shares the dish that has been ordered at that table. If he sits at a new table, he order a dish for that table according to its popularity among the whole franchise, while a new dish can also be tried. 
                

    The relationship between and can be best illustrated as in the following figure.



    random variables
    meaningmetophor
    random variables customer i in restaurant  j
    distinct values of in group table t in restaurant j
    index of associated to ,
    the table taken by customer i in restaurantj,.i.e., Table()=, Customer() =

    the number of associated to in group j
    the number of customers sitting around table t in restaurantj
    distinct values in within all groups,
    dish k, which is shared within all restaurants
    index of associated to ,
    the dish ordered by table t in restaurantj, i.e., Dish() = , Table() =

    the number of associated to in group j
    the number of tables ordered dish k in restaurantj
    , i.e., the number of associated to over all j
    the total number of tables ordered dish k within all restaurants