Ph.D. Theses

Light, Sound, and Semantics: The Web of Data as a New Sensory Modality

By Joshua Shinavier
Advisor: James A. Hendler
July 7, 2015

This dissertation explores the use of machine-accessible knowledge about the objects in our environment to augment our perception at an immediate, preconscious level.

In everyday life, we combine simultaneous natural stimuli, such as the sound of a voice and the sight of the speaker’s moving lips, into percepts without thinking about them individually. Artificial stimuli may be combined into the same percepts if they are semantically congruent with the perceptual context and if they arrive within brief temporal windows of the natural stimuli, among other conditions. Insofar as a knowledge-based system can recognize and respond to new context quickly and appropriately enough, its feedback may offer an advantage over natural signals alone, as it allows us to draw attention to nonphysical and non-obvious properties of the world, such as abstract relationships.

In order to truly extend a person’s natural senses, we need to understand the psychophysics of semantics and perception as well as the technological challenges of building such a system. The main focus of this dissertation is on the latter set of problems. We will first translate the known perceptual constraints into a set of functional requirements, then introduce a concrete Semantic Web architecture which fulfills them. The architecture combines cooperative activity detection, a SPARQL-based complex event processor, a Linked Data client, and a body area network of sensing and feedback devices. A number of Semantic Wearable applications are provided as proofs of concept, and a simulation-based evaluation of the system is also described, illustrating the performance of the system, for non-trivial scenarios and at a significant scale, within its real-time constraints.

In this architecture, the Web of Data serves as a read-write repository of knowledge about people and objects; it is queried on demand and updated for new context with new knowledge, enabling a feedback loop of perception and interaction which is independent of any single environment.

Return to main PhD Theses page