15 ICCV'11 paper I am interested in

More than one week passed after coming back from ICCV, Barcelona, I finally finished my summary of ICCV papers that I am interested. There are totally 15 papers in 5 topics. Certainly there are many more ICCV paper worth to read. I will update my summary later.  

Attributes:

In ICCV this year, attributes continue to attract interests from researchers in the community. In particular, the paper “Relative Attributes” wins the Marr prize.

Relative Attributes  (Marr prize paper)
Devi Parikh, Kristen Grauman

Idea: relative attribute can provide more informative and intuitive description for images, which overcome many restrictions of binary attributes. For example, it is more useful to say “Bill Clinton is younger than George H.W.Bush” than to say “Bill Clinton is young”. The latter is a binary attribute which is often difficult to judge true or false since it is a subjective judgment in many cases; while the former is a relative attribute which is more objective and easier to judge. This paper describe an approach to model the relative attributes as a ranking function and then apply it to zero-shot learning and textual description of images, and shows clear advantage over traditional binary attributes.

Describing People: A Poselet-Based Approach to Attribute Classification (oral)
Lubomir Bourdev, Subhransu Maji, Jitendra Malik

Idea: This paper applies poselet representation to recognize attributes of human, such as gender, hair style and types of cloths. It is generally a difficult task to recognize these human attributes due to the large variations of pose, viewpoint, etc. The poselet representation implicitly decomposes the aspect, i.e., the pose and viewpoint, and thus facilitates the detection of human attributes.

A Joint Learning Framework for Attribute Models and Object Descriptions (oral)
Dhruv Mahajan, Sundararajan Sellamanickam, Vinod Nair

Idea: This paper proposes to jointly learn attribute classifiers and the attribute labels.  This method eliminates the requirement of labeling the attributes in images. Giving a list of attribute names, some positive and negative training examples for each of the binary attribute classifiers and some training images of various objects with known class labels but not their attribute labels, the proposed method can automatically learn an attribute vector for each object class. An interesting finding of this research is that it detects lots of erroneous attribute labels in the existing dataset and find the performance of classification can be boosted after CORRECTing these erroneous attributes.

Depth Image and Kinect:

There are a number of ICCV papers this year study various problems related to depth image, possibly due to the widely available Kinect sensor.

Kinecting the dots: Particle Based Scene Flow From Depth Sensors

Simon Hadfield; Richard Bowden

Summary: The theme of this paper is to estimate scene flow, the 3-D motion field of an observed scene, as opposed to optical flow in 2-D field. The point motion in 3D is modeled as a collection of particle filters that support multiple hypotheses and does not oversmooth the motion field.

Efficient Regression of General-Activity Human Poses from Depth Images
Ross Girshick; Jamie Shotton; Pushmeet Kohli; Antonio Criminisi; Andrew Fitzgibbon

Summary: This paper employs random forest regression to directly estimate the human pose without segmentation of body parts; several techniques are proposed to speed up the regression process which enable super-realtime test performance.

Accurate 3D Body Pose Estimation From a Single Depth Image
Mao Ye; Xianwang Wang; Ruigang Yang; Liu Ren; Marc Pollefeys

Summary: use pre-captured motion exemplars to estimate the body pose in an depth image and then refine the results by fitting the body configurations with the input depth image.

A Data-Driven Approach for Real-Time Full Body Pose Reconstruction from a Depth Camera
Andreas Baak; Meinard Mueller; Gaurav Bharaj; Hans-Peter Seidel; Christian Theobalt

Summary: similar to the above paper, this paper also use a pose database to facilitate the problem of pose estimation from depth images.

This are two interesting demos using Kinect sensor:

KinectFusion: Real-time 3D tracking, reconstruction and Interaction with a depth camera S.Izadi, R.Newcombe et al.

Seeing Your Weight – An application in targeted advertisement T.Van Nguyen, S.Yan

Random Forest:

Random Forest is an ensemble of decision trees, where each decision tree is slightly different from one another. The randomness among different decision trees are achieved by being train with either different subset of training data or different subset of parameter space. Random Forest classifier can achieve similar max-margin like behavior like SVM with less computational cost. Random Forest has been also widely used in many other problems, such as regression, density estimation, manifold, semi-supervised learning, etc. There is one-day tutorial on Random Forest in ICCV this year. The PPT and technical report can be downloaded online at http://research.microsoft.com/en-us/groups/vision/decisionforests.aspx

The following ICCV papers are related to Random Forest:

Structured Class-Labels in Random Forests for Semantic Image Labelling (Oral)
Peter Kontschieder, Samuel Rota Bulò, Horst Bischof, Marcello Pelillo

Task: image labeling, ie., classify the pixels in an image to an object class label

Idea: Using Random Forest classifier, but each internal node is a classifier for an image patch rather than a classifier for an image pixel. The advantage of this approach is that the spatial context is represented in the classifier by the image patch and thus in testing the classifier can produce much smooth and coherent image labels

Decision Tree Fields  (Oral)
Sebastian Nowozin, Carsten Rother, Shai Bagon, Bangpeng Yao, Toby Sharp, Pushmeet Kohli

Task: image labeling

Idea: A CRF using the Decision Tree to model the mapping between the image and the parameters of a unary or pairwise interaction in the graphical model. The advantage of using Decision Tree to model this mapping is (1) it is non-parametric so can represent richer relationships; (2) more scalable to large training data

Action and Activity:

Human Action Recognition by Learning Bases of Action Attributes and Parts (oral)
Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas J. Guibas, Li Fei-Fei

Summary: the idea is similar to the paper of scene attribute published last NIPS and ECCV workshop by Jia-Li Li of their group. This sparse coding method is used to learn sparse bases of action attributes in static images.

Learning Spatiotemporal Graphs of Human Activities (oral)
William Brendel, Sinisa Todorovic

Summary:  a volumetric representation of human activities is presented.  A video is decomposed into a collection of spatiotemporal tubes at multiple scales. These tubes are connected by a graph model which represents the temporal and spatial constraints among these tubes. This paper address the issues how to extract/learn and match/recognize these 2D+t tubes and their configuration for activity recognition in video. 

A "String of Feature Graphs" Model for Recognition of Complex Activities in Natural Videos
Utkarsh Gaur, Yingying Zhu, Bi Song, Amit Roy-Chowdhury

Summary: STIP features are grouped together to form a feature graph that represent the spatial configuration in a single image; and the feature graphs in consecutive frames are linked to form a “String of Feature Graphs” to represent the temporal dynamics. Activity recognition in videos is then cast as a problem to match these feature graphs and the string of feature graphs.

Human Activity Prediction: Early Recognition of Ongoing Activities from Streaming Videos
Michael Ryoo

Summary: this paper discussed an interesting problem: how to recognize the human activity in a video stream without seeing the all-length video. The solution presented in this paper is a dynamic bag-of-words model, which is similar to dynamic time warping.

Machine learning:

Perturb-and-MAP Random Fields: Using Discrete Optimization to Learn and Sample from Energy Models (oral)
George Papandreou, Alan Yuille

Task: how to avoid the costly MCMC sampling in energy models (e.g., MRF) while maintain accuracy in parameter estimating

Idea: Do MAP after adding randomness (i.e., noise) in the parameter: MAP is faster while the randomness simulates the power of MCMC

Discriminative Learning of Relaxed Hierarchy for Large-scale Visual Recognition [Supplemental materials] (oral)
Tianshi Gao, Daphne Koller

Task: learning hierarchical classifier for large scale image database with hundreds or thousands of visual classes

Idea: hierarchical classifier is a natural solution for large scale visual recognition. The problem is that it is often not able to divide the visual classes nicely and there are often ambiguous classes in-between. The idea of this paper is very simple and smart: treat the ambiguous classes as if they do not exist (i.e., relaxed hierarchy). This idea can be modeled in a modified SVM where these ambiguous classes are labeled as 0 (while positive classes are labeled as +1 and negative classes are labeled as -1) so they will not take effect in computing the loss but still constraint the parameter space.

Twenty Questions Game and Object Recognition

Objects can be defined by many features/parts/attributes, each of which can be viewed as a test. The problem of object recognition/detection is then solved by combining outputs of these tests. Instead of performing all possible tests, a smart way is to select a small set of tests without sacrificing the recognition quality/accuracy. The process of selectinvision.ucsd.edu/sites/default/files/Visipedia20q.pdfg the right tests can be formulated as a 20-question game, and the recognition of object is achieved by sequentially asking a question to an Oracle, and analyzing the results returned by the Oracle. The criterion of selecting next question is the information gain brought by the answer of the question. This approach is also called "Active Testing" in "An Active Testing Model for Tracking Roads in Satellite Images", PAMI 1996.

So far, the earliest work using this idea for object recognition is due to Donald Geman of JHU, described in his 1993 technical report "Shape Recognition and Twenty Questions". Each test is a local functional of the image loosely corresponding to configurations (vertex labels) resembling "endings", "junctions", and "turns", or a invariant relations (relational labels) between two vertex labels, i.e., "same class", "same orientation".

The most recent work is "Active Testing for Face Detection and Localization", PAMI 2010, "Visual Recognition with Humans in the Loop", ECCV2010a, and "Indoor Scene Recognition Through Object Detection Using Adaptive Objects Search", ECCV 2010b. In the PAMI 2010 paper, the tests are specific type of image functional (i.e., proportion of edges in particular orientation and scale) within a local region. In the ECCV 2010b paper, the tests are object detectors. In the ECCV 2010a paper, the tests are object attributes while the Oracle is human.

This idea can be extended in many aspects. In the application domain, it can be used in scene and activity recognition; regarding the questions to ask, we can ask many richer questions besides What, e.g., Where, How Many, How Big, etc. We are currently investigating these problems.

papers: prototype theory

In my previous post, "basic level classes and subordinate class", I mentioned Aharon Bar-Hillel's paper Subordinate class recognition using relational object models . Now I am going to build topic models for object hierarchies and need to have a better understanding of the prototype theory by Rosch. Here are some papers I found about this topic:

the seminal paper: Basic Objects in Natural Categories, cognitive psychology 1976. Another link

several blogs on this theory: 

http://mixingmemory.blogspot.com/2005/03/basics-of-basic-level.html

http://poorbuthappy.com/ease/archives/2003/11/20/1944/basic-level-categories

http://everything2.com/title/basic+level+categories

papers: Estimation of Dirichlet Distribution Parameters

Recently, I am interested in apply Pachinko Allocation topic models to the object recognition problems. Mixtures of Hierarchical Topics with Pachinko Allocation, ICML 2007 mentioned several methods in training the hPAM model, and here are the related papers: 

• Gibbs EM: 
• An introduction to MCMC for machine learning, Machine Learning 2003
• implementations of the Monte Carlo EM Algorithm, 2001
  
• fixed point iteration method for estimating the Dirichlet parameters: 
• Estimating a Dirichlet distribution, Minka 2000, and code
• Maximum Likelihood Estimation of Dirichlet Distribution Parameters by Jonathan Huang

papers: syntax and topic model

Syntactic constraint is an important ingredient in NLP. At the beginning, topic models, such as LDA, assume bag-of-word model and thus ignore the syntax. Later on, this constraint is added to the topic model to improve the modeling power. Here are a few papers regarding this issue:

• Integrating topics and syntax, NIPS 2005
• Style and Topic Language Model Adaptation Using HMM-LDA, EMNLP 2006
• Hidden Topic Markov Models, AISTATS 2007, presentation video
  
• Topic Modeling: Beyond Bag-of-Words, ICML 2006, slides
• Syntactic Topic Models, NIPS 2008, supplement materials
• Bayesian Modeling of Dependency Trees Using Hierarchical Pitman-Yor Priors, In Proceedings of the Workshop on Prior Knowledge for Text and language (held in conjunction with ICML/UAI/COLT), 2008
• Topical N-grams: Phrase and Topic Discovery, with an Application to Information Retrieval, ICML 2007, technical report

paper: Rethinking LDA: Why Priors Matter

Rethinking LDA: Why Priors Matter, Hanna M. Wallach David Mimno Andrew McCallum, NIPS 2009

Abstract:


Implementations of topic models typically use symmetric Dirichlet priors with fixed concentration parameters, with the implicit assumption that such “smoothing parameters” have little practical effect. In this paper, we explore several classes of structured priors for topic models. We find that an asymmetric Dirichlet prior over the document–topic distributions has substantial advantages over a symmetric prior, while an asymmetric prior over the topic–word distributions provides no real benefit. Approximation of this prior structure through simple, efficient hyperparameter optimization steps is sufficient to achieve these performance gains. The prior structure we advocate substantially increases the robustness of topic models to variations in the number of topics and to the highly skewed word frequency distributions common in natural language. Since this prior structure can be implemented using efficient algorithms that add negligible cost beyond standard inference techniques, we recommend it as a new standard for topic modeling.

paper: On Smoothing and Inference for Topic Models

On Smoothing and Inference for Topic Models, UAI 2009
abstract:

Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

paper: Multilevel Bayesian Models of Categorical Data Annotation

A paper I found from LingPipe's blog: 

Multilevel Bayesian Models of Categorical Data Annotation

It seems to be close related to image annotation. More comments will follow after reading it. 

fast and parallel Gibbs sampling for LDA

Gibbs sampling for LDA is very simple to understand and implement, especially the collapsed Gibbs sampling. But one drawbacks of GS is its complexity is linear to the number of word tokens. This problem is even more serious when we apply LDA-based approaches to computer vision problems where we use visual words in images to replace words in documents. To maximize our chance to detect the object in an image, we need large number of visual word tokens. It is more and more popular to extract features at dense regular grids over images, and to one extreme, someone extract features at every pixel with several scales. Also we often need to extract several types of features and hope them to be complementary to each other since we usually do not which type of feature is more useful for a particular object category. Combine these factors together, there are often more than 10k ~ 50k word tokens per image extracted. For Gibbs sampling, this is a nightmare!

So a fast Gibbs sampling or parallel Gibbs sampling are absolutely rescues. There are two such papers recently, with published codes (that is great!):

PLDA: Parallel Latent Dirichlet Allocation for Large-scale Applications by Wang Yi et al at Google, code

Fast Collapsed Gibbs Sampling For Latent Dirichlet Allocation at UCI, code

here is a comment from LingPipe's blog:

Porteous et al. (2008) Fast Collapsed Gibbs Sampling for Latent Dirichlet Allocation

Another paper related to topic model inference in large scale corpus is

Efficient Methods for Topic Model Inference on Streaming Document Collections

paper: Boosted Bayesian Network Classifiers

Jing Yushi has a ICML 2005/ ML2008 paper:
Boosted Bayesian Network Classifiers
It seems very interesting to me. Now I am using topic models to implement my ideas. But generative models usually can not beat discriminative classifiers such as SVM in many cases. It is of interests of the generative guys to combine these two methods to benefit from both. Jing's paper show the boosted version of Naive Bayesian. Can we develop boosted topic models? It is good direction. I Googled and find no such work so far. 

paper: DeltaLDA

DeltaLDA is a modification of the Latent Dirichlet Allocation (LDA) model which uses two different topic mixing weight priors to jointly model two corpora with a shared set of topics, where one topic mixing weight prior to model the normal pattern and the other for the abnormal pattern.
The graphical model:

An illustration of topic mixture weights in two scenarios:

This looks like quite similar to the Adapted Vocabularies for Generic Visual Categorization, ECCV 2006 in the way they split the topic/vocabulary into two sets, though there are fundamental difference in their underneath mechanism.

paper: Sketch2Photo: Internet Image Montage

Amazing realistic montage!

A few students from Tsinghua Univ. present this montage using images downloaded from the internet.

The links are as follows:
project web page
The demo on youtube
The paper on ACM SIGGRAPH ASIA 2009, ACM Transactions on Graphics

papers: visual attribute and object class recognition

Several recent papers discussed the methods to extract visual attributes from images and/or use these attributes for object class recognition. We can view visual attributes as another type of annotation. While image annotation is applied to individual image, visual attributes are specified to an object class; image annotations are usually words, i.e., discrete values, visual attributes can be either discrete (e.g. color={red, blue, ...}) or continuous values (e.g., average size). Here are a few papers I am reading:

• Describing Objects by their Attributes, CVPR 2009
  
• Joint learning of visual attributes, object classes and visual saliency, ICCV 2009
• Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer, CVPR 2009
• Learning Visual Attributes, NIPS 2007

Here is a blog about visual attribute from Tombone.

segmentation vs. recognition

In What is segmentation-driven object recognition?,
Tomasz finally remarked that the learning-driven segmentation may be a hot topic in the next few years. I totally agree with him. The problem for us is how to design such algorithms to be robust to intra-class variations, scales and pose. This remains a challenging problem in recognition community.

In the comments of this blog, someone suggest to check the most up-to-date segmentation results in PASCAL 2009.

basic level classes and subordinate class

Comments: the following paper provides a good insight into the role of generative and discriminative models in learning a large number of object categories, i.e., we can use the generative models to distinguish categories at basic level, and discriminative models to differentiate lower-level and similar categories.  

In Subordinate class recognition using relational object models
Aharon Bar-Hillel, Daphna Weinshall, NIPS, 2006, the authors illustrate some interesting points:

"Human categorization is fundamentally hierarchical, where categories are organized in tree-like hierarchies. 

• higher nodes close to the root describe inclusive classes (like vehicles), 
• intermediate nodes describe more specific categories (like motorcycles), 
• lower nodes close to the leaves capture fine distinctions between objects (e.g., cross vs. sport motorcycles).
  

Intuitively one could expect such hierarchy to be learnt either bottom-up or top-down (or both), but surprisingly, this is not the case. In fact, there is a well defined intermediate level in the hierarchy, called basic level, which is learnt first [11]...."

"The primary role of basic level categories seems related to the structure of objects in the world. In [13], Tversky & Hemenway promote the hypothesis that the explanation lies in the notion of parts.Their experiments show that  

• basic level categories (like cars and flowers) are often described as a combination of distinctive parts (e.g., stem and petals), which are mostly unique. 
• higher levels (superordinate and more inclusive) are more often described by their function (e.g., ’used for transportation’), 
• lower levels (sub-ordinate and more specific) are often described by part properties (e.g., red petals) and other fine details."

Based on these assumptions, Bar-Hillel and Weinshall proposed a two stage approach for subordinate class recognition:

1. First we should learn a generative model for the basic category. Using such a model, the object parts should be identified in each image, and their descriptions can be concatenated into an ordered vector. This stage is used to solve the correspondence problem: features in the same entry in two different image vectors correspond since they implement the same part.
   
2. In a second stage, the distinction between subordinate classes can be done by applying standard machine learning tools, like SVM, to the resulting ordered vectors, since the correspondence problem has been solved in the first stage.

Another paper reinforce this idea from the psychology study:  Comparison Processes in Category learning: From Theory to Behavior, Rubi Hammer, Aharon Bar-Hillel, Tomer Hertz, Daphna Weinshall and Shaul Hochstein, Brain Research, Special issue on 'Brain and Vision', 2008.

a good summary on generative vs. discriminative models

The GenDisc2009 NIPS workshop is call for papers. Though I have no time to catch up the deadline, I found the brief discussion on the generative vs. discriminative models are quite useful. In case I lose the link or the link is broken in the future, I copy some contents as follows:

In generative approaches for prediction tasks, one models a joint distribution on inputs and outputs and parameters are typically estimated using a likelihood-based criterion. In discriminative approaches, one directly models the mapping from inputs to outputs (either as a conditional distribution or simply as a prediction function); parameters are estimated by optimizing objectives related to various loss functions. Discriminative approaches have shown better performance given enough data, as they are better tailored to the prediction task and appear more robust to model misspecification. Despite the strong empirical success of discriminative methods in a wide range of applications, when the structures to be learned become more complex than the amount of training data (e.g., in machine translation, scene understanding, biological process discovery), some other source of information must be used to constrain the space of candidate models (e.g., unlabeled examples, related data sources or human prior knowledge). Generative modeling is a principled way of encoding this additional information, e.g., through probabilistic graphical models or stochastic grammar rules. Moreover, they provide a natural way to use unlabeled data and are sometimes more computationally efficient.

Theoretical analysis of generative versus discriminative learning has a long history in statistics, where the focus was on asymptotic analyses (e.g. [Efron 75]). Ng and Jordan provided an initial comparison of generative versus discriminative learning in the non-asymptotic regime in the most cited paper on the topic in machine learning [Ng 02]. For a few years, this paper was one of the only machine learning papers providing a theoretical comparison, and was responsible for the conventional wisdom: "use generative learning for small amount of data and discriminative learning for large amounts". Recently, there has been new advances on our theoretical understanding [Liang 08, Xue 08] and their combination [Bouchard 07, Xue 09].

On the empirical side, combinations of discriminative and generative methodologies have been explored by several authors [Raina 04, Bouchard 04, McCallum 06, Bishop 07, Schmah 09] in many fields such as natural language processing, speech recognition, and computer vision. In particular, the recent "deep learning" revolution of neural networks relies heavily on a hybrid generative-discriminative approach: an unsupervised generative learning phase ("pre-training") is followed by discriminative fine-tuning. Given these recent trends, a workshop on the interplay of generative and discriminative learning seem especially relevant.

Hybrid generative-discriminative techniques face computational challenges. For some models, training these hybrids is akin to the discriminative training of generative models, which is a notoriously hard problem ([Bottou 91] for discriminatively trained HMM, [Jebara 04, Salojarvi 05] for EM-like algorithms), though for other models, learning can be in fact simple [Raina 04, Wettig 03]. Alternatively, the use of generative models in predictive settings has been be explored, e.g., through the use of Fisher kernels [Jaakkola 98] or other probabilistic kernels. One of the goal of the workshop will be to highlight the connections between these approaches.

The aim of this workshop is .... (ignored)

References

[Bishop 07] C. M. Bishop and J. Lasserre, Generative or Discriminative? getting the best of both worlds. In Bayesian Statistics 8, Bernardo, J. M. et al. (Eds), Oxford University Press. 3–23, 2007.

[Bottou 91] L. Bottou, Une approche théorique de l'apprentissage connexionniste: Applications à la reconnaissance de la parole. Doctoral dissertation, Université de Paris XI, 1991.

[Bouchard 04] G. Bouchard and B. Triggs, The tradeoff between generative and discriminative classifiers. In J. Antoch, editor, Proc. of COMPSTAT'04, 16th Symposium of IASC, volume 16. Physica-Verlag, 2004.

[Bouchard 07] G. Bouchard, Bias-variance tradeoff in hybrid generative-discriminative models. In proc. of the Sixth International conference on Machine Learning and Applications (ICMLA 07), Cincinnati, Ohio, USA, 13-15 December 2007.

[Efron 75] B. Efron, The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis. Journal of the American Statistical Association, 70(352), 892—898, 1975.

[Greiner 02] R. Greiner and W. Zhou. Structural extension to logistic regression: Discriminant parameter learning of belief net classifiers. In Proceedings of the Eighteenth Annual National Conference on Artificial Intelligence (AAAI-02), 167–173, 2002.

[Jaakkola 98] T. Jaakkola and D. Haussler. Exploiting generative models in discriminative classifiers. In Advances in Neural Information Processing Systems 11, 1998.

[Jaakkola 99] T. Jaakkola, M. Meila, and T. Jebara. Maximum entropy discrimination. In Advances in Neural Information Processing Systems 12. MIT Press, 1999.

[Jebara 04] T. Jebara, Machine Learning - Discriminative and Generative. International Series in Engineering and Computer Science, Springer, Vol. 755, 2004.

[Liang 08] P. Liang and M. I. Jordan, An asymptotic analysis of generative, discriminative, and pseudo-likelihood estimators. In Proceedings of the 25th International Conference on Machine Learning (ICML), 2008.

[McCallum 06] A. McCallum, C. Pal, G. Druck and X. Wang, Multi-Conditional Learning: Generative/Discriminative Training for Clustering and Classification. AAAI, 2006.

[Ng 02] A. Y. Ng and M. I. Jordan, On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes. In Advances in Neural Information Processing Systems 14, 2002.

[Salojarvi 05] J. Salojärvi, K. Puolamäki and S. Kaski, Expectation maximization algorithms for conditional likelihoods. In Proceedings of the 22nd International Conference on Machine Learning (ICML), 2005.

[Schmah 09] T. Schmah, G. E Hinton, R. Zemel, S. L. Small and S. Strother, Generative versus discriminative training of RBMs for classification of fMRI images. In Advances in Neural Information Processing Systems 21, 2009.

[Wettig 03] H. Wettig, P. Grünwald, T. Roos, P. Myllymäki and H.Tirri, When discriminative learning of Bayesian network parameters is easy. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI 2003), 491-496, August 2003

[Xue 08] J.-H Xue and D.M. Titterington, Comment on "discriminative vs. generative classifiers: a comparison of logistic regression and naive Bayes". Neural Processing Letters, 28(3), 169-187, 2008.

[Xue 09] J.-H Xue and D.M. Titterington, Interpretation of hybrid generative/discriminative algorithms. Neurocomputing, 72(7-9), 1648-1655, 2009.

Author-topic model and transformed LDA

Latent Dirichlet Allocation (LDA) is essentially a generative model for document analysis rather than classification, and it is an unsupervised rather than supervised learning algorithm. Given a new document, the output of LDA is the topic proportion instead of document category. So LDA can not be directly used for classification.

Author-topic model (ATM), on the other hand, can be used in classification, as long as we view the author as the category label.

Comparison between the above two models can be summarized as follows, where the figures are from the UAI paper by M. Rosen-Zvi, T. Griffiths, M. Steyvers, P. Smyth, 2004:

LDA	ATM
generative process: choose for each of the  words in document d choose choose	generative process: for each of the  words in document d choose an author x from  , the author set of document d following a uniform distribution choose choose

Notice the most significant difference in ATM compared to LDA is that the topic mixture weight is not generated for each document; rather, there are finite number of possible topics mixture weights, which is specified by the author information in each document.

For document classification, if we view the author as the class label, and let is a scalar, the ATM model can be directly applied.

My interest on the ATM model is due to Sudderth's transformed LDA model, which reduces to an ATM when ignore the spatial transformation (see the part inside the big red square).

a good review article for LDA

I happened to find a review article for LDA and its application for text, vision and music.
The link is Latent Dirichlet Allocation for Text, Images, and Music
and the slides is here

They are worth to read carefully.

papers: employing semantic hierarchy in object recognition

Semantic hierarchy could play an important role in object recognition. For example, if we know mini-van is a type of car, and we have already a model for car vs. the rest of the world, then we only need to differentiate mini-van from the car, which reduces lots of work. Similar idea has been noticed by object recognition researchers and there are several papers in recent years:

• Exploiting object hierarchy: Combining models from different category levels, ICCV 2007
• Learning and using taxonomies for fast visual categorization, CVPR 2008
• Unsupervised discovery of visual object class hierarchies, CVPR 2008
• Constructing category hierarchies for visual recognition, ECCV 2008
• Unsupervised learning of visual taxonomies, CVPR 2008
• Semantic Hierarchies for Visual Object Recognition, CVPR 2007
  
• Latent Topic Random Fields: Learning using a taxonomy of labels, CVPR 2008

where some are listed in Trevor's course page.

papers: supervised or discriminative topic model

Topic models are originally designed for topic discovery/clustering, not for classification. To use topic models for classification task, we have modify the structure of the topic model to add the class label and use it to bias the topic discovery process.

the following paper present some supervised/discriminative topic models by machine learning guys:

• MedLDA: Maximum Margin Supervised Topic Models for Regression and Classification, ICML 2009
• DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification, NIIPS 2008
• Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora, EMNLP 2009
• Supervised topic models, NIPS 2007

some supervised/discriminative topic models by computer vision guys:

• A Bayesian Hierarchical Model for Learning Natural Scene Categories, CVPR 2005
  
• Simultaneous Image Classification and Annotation, CVPR 2009
• Spatially coherent latent topic model for concurrent object segmentation and classification, ICCV 2007
• Towards Total Scene Understanding Classification, Annotation and Segmentation in an Automatic Framework, CVPR 2009
• What, where and who? Classifying events by scene and object recognition, ICCV 2007
• Learning Hierarchical Models of Scenes, Objects, and Parts, ICCV 2005