Welcome to Pengyu Hong's Homepage

Automatic Temporal-Spatial Pattern Modeling and Extraction

The major contribution of this research is that it develops the theory for automatic contextual pattern modeling, which is essential for object modeling/recognition. The applications include: image/video database, molecular modeling, network traffic monitoring, and so on.

Publications:

Hong, P. and Huang, T. S. (2004) Spatial pattern discovery by learning a probabilistic parametric model from multiple attributed relational graphs. Journal of Discrete Applied Mathematics, vol. 130, pp. 113-135. (PDF)

Hong, P. and Huang, T. S. Mining inexact spatial pattern. Workshop on Discrete Mathematics and Data Mining (DM & DM) 2002

Hong, P., Wang, R., and Huang, T. S. Learning Patterns from Images by Combining Soft Decisions and Hard Decisions. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Hilton Head Island, South Carolina, June 13-15, 2000.

Overview

	Introduction
	The approach
	Examples

Introduction:

A pattern usually consists of several primitives among which various contextual relations are defined. Attributed relational graph is chosen to represent the samples of patterns. We first develop the theory for automatic spatial pattern modeling and extraction to learn a probabilistic parametric model from the attributed relational graphs of multiple samples of a pattern. The learned model characterizes both the appearance and the structure of the pattern, which is observed under various conditions. It can be used for spatial information summarization and retrieval. Adding temporal constraints, we then extend the proposed approach for automatic temporal-spatial pattern modeling and extraction. We demonstrate the theory by applying it to the problem of unsupervised visual pattern extraction, texture modeling and synthesis, text summarization, and video summarization.

The approach:

We fist chose Attributed Relational Graph (ARG) to represent samples.

Theory is developed to automatically learn the pattern ARG model from the observed sample ARGs. The maximum-likelihood parameters of the pattern ARG model are estimated via an iterative learning procedure. The learning procedure calculates: (a) the attributed parameters (appearance/non-spatial information) of the pattern ARG, (b) the relational parameters (spatial information) of the pattern ARG, (c) the configuration (the number of nodes and that of the relations) of the pattern ARG, and (d) the node and relation correspondences between the components of the pattern ARG and the sample ARGs. In addition, the learning procedure is able to distinguish the pattern from its backgrounds by taking advantage of multiple samples.

In our experiments, the theory is applied to unsupervised visual pattern extraction. Though the examples shown below are about 2D images, the theory can also be applied to data in higher dimensional space because ARG can be used to represent data in any dimension.

Fig 1.Contextual Pattern Modeling (M << S).

Examples:

In the following examples, the images are segmented. Each node of the ARG represents a segment of the image. The attribute of the node is the mean color (RGB) feature vector of the segment. The adjacent relations among the segments are considered. The pattern ARG models are assumed to be Contextual Gaussian Mixture models.

Example 1.

(a) Sample images

(b) Segmented images of (a)

Fig. 2. A more complicated example. The pattern ARG only has one component. Due to the complexity of the ARGs in this example, the ARGs are not shown.

Example 2.

(a) Sample image 1	(b) Sample image 2	(c) Sample image 3
(d) Sample image 4	(e) Sample image 5	(f) Sample image 6
(g) The 1st component of the pattern ARG.		(h) The 2nd component of the pattern ARG.
Figure 3. We take the pictures of the MacDonald sign in various backgrounds, from The pattern ARG model is assumed to have two components. After learning, the training data set is summarized as two model components in the Contextual Gaussian Mixture model. Both of them have 8 nodes and 11 adjacent relations. To illustrate the learning results, we use the learned model to detect its isomorphic sub-graph in the ARG of Fig. 4(a) and repaint the corresponding image segments using the color attributes of the corresponding model nodes. The same process is repeated on Fig. 4(d). The detection results are shown in Fig. 4 (g) and (h) respectively. The detection results also show that correct matching results are achieved due to using the pattern ARG as the bridge.

Example 3 -- Extract temporal-spatial pattern from image sequence

(Click to see the original video)

(Click to see the segmentation results of the original video)

(Click to see the ARG representation of the original video)

(Click to see the learning results)

(Click to see the original image segments corresponding to the learning results)