Ph.D. Theses

A Flexible Integration Framework for Heterogeneous Content-Based Multimedia Information Retrieval

By Corey N. Bufi
Advisor: Sibel Adali
November 13, 2001

With recent advances in technologies for publishing and distributing multimedia information across readily accessible repositories such as those on the World Wide Web, it is becoming increasingly important to retrieve this information in ways that extend beyond the capabilities of existing retrieval methods. In response to this need, this thesis introduces a novel integration framework that facilitates the execution of generic content-based queries against existing heterogeneous multimedia repositories and provides for flexible result merging and ranking strategies.

We present a data model that formalizes the notion of a content-based query that takes into account several forms of heterogeneity associated with content-based retrieval from disparate multimedia repositories. This formalization considers the human subjectivity of multimedia information and forms the basis of a generic, SQL-like query language, where it serves as a work-handle for managing the potential loss of information due to differences in interpretations of an information problem. The syntax and semantics of this language are defined to show its expressiveness for new and highly complex relationships between multimedia objects, which no current query language will support. To cope with the rich forms of heterogeneity exhibited in answers to such queries, we motivate an efficient extension of SQL aggregation that allows multiple aggregates in expressions for combining results from multiple queries.

We propose to describe the sophisticated querying capabilities of multimedia repositories using a new rule-based approach that is designed to address the important and complex aspects of searching for information by content-based attributes. A query mapping algorithm is defined that utilizes these descriptions to translate generic queries to queries understood by the underlying repositories. A novel relaxation algorithm is described in which queries that are not readily supported are transformed to similar, yet weaker forms. A detailed survey of several prominent Web page search engines and image retrieval systems is presented to illustrate the diversity and peculiarities of real world repositories, and to demonstrate the applicability of our algorithms.

The result merging problem is to combine multiple ranked lists into a single list that represents the final query result. Unlike most previous research in result merging in multimedia information retrieval, we deal directly with the fact that few repositories return relevance scores and base our result merging algorithms on rank information alone. The problem of combining ranks is complicated when the repositories contain very large and different sets of information, and use different ranking algorithms. The consequence is that the rank information of a multimedia object may not be complete at the time its combined rank needs to be computed. Our approach for combining ranks addresses these issues by estimating unreported ranks and returning aggregated results incrementally as objects are retrieved from multiple repositories. Results based on an implementation over Web search engines are presented.

Return to main PhD Theses page