Semantic Multimedia Analysis

In our core resaerch area Semantic Multimedia Analysis, we address the extraction of semantic information in video data. This is known as one of the most difficult computational problems, because automatically computable features (e.g., colour, edge, motion, audio etc.) do not have a direct relation to semantic information perceived by human (e.g., object, action, scene, event etc.).
 

We are tackling this issue by formulating it as a machine learning problem, where a model is constructed to differentiate between videos with a certain meaning and  other videos. Machine learning techniques that interest us include (weakly or partially) supervised learning, multiple kernel learning, multi-task learning, hidden conditional random fields, and so on. Also, we are investigating how to adopt the mechanism of the human visual system using focus of attention and deep learning techniques, and how to utilise knowledge base (ontology) about human interpretation of semantic meanings.

 

Furthermore, we are trying to improve semantic video analysis using external data, such as textual descriptions collected by Web users' collaborative annotation, and artificial 3D data created by virtual reality technique. In addition, we are positive in applying the developed methods to previously-unexplored problems, such as environmental quality assessment by analysing microorganisms in images/videos, and surveillance video analysis of animals for investigating their habits. Finally, to transfer the developed methods to Web-scale video analysis and retrieval, we are interested in improving the computational efficiency with both high-performance computing and algorithm sophistication.