Multimodal Sentiment Analysis – Cooperation with TU Social Computing Research Group

Prof. Dr. Georg Groh (Social Computing Research Group - TU Munich)
Tobias Eder, M.A., M.Sc. (Social Computing Research Group - TU Munich)

Working Title: Traveling Images – Multimodal Sentiment Analysis for City Landmarks and Architecture

The circulation of digital photographs on the Web has increased with image specific social media platforms since the late 2000s. Among other things the dissemination of photographs of iconic buildings and their surroundings shapes the digital landscape and view of cities more than before, when such views were filtered through official channels. This constitutes a shift in control over images and narratives over these disseminated images from the formal producers of the built environment to the informal users.

Depot Boijmans Van Beuningen, City of Rotterdam. Image: CC 4.0 (https://commons.wikimedia.org/wiki/File:Boijmans_Depot_Rotterdam.jpg)

In collaboration with the chair of urban planning at the Faculty of Architecture at TUM we are building a machine-learning based system to help architects and urban planners track and understand this shift in perception and developing views on city landmarks and architecture. By using methods of graph-based social media analysis in conjunction with computer vision techniques for image classification and similarity matching we are generating time-aware representations of shifting views of cities on social media, which track the user-generated perspectives and how public spaces are perceived used and repurposed over time.

In our research we are using deep learning architectures for computer vision, with which to identify and cluster landmarks in photos on social media. For this we are utilizing both large-scale data scraping techniques, as well as existing architectures and transfer learning approaches to adapt systems to our specific use-cases such as the city of Rotterdam in the Netherlands. Thanks to the IBM OpenPower system and the help of the OpenPower team we can process much bigger quantities of data and iterate over models quickly in the development process, enabling us to do quicker feedback rounds in our interdisciplinary team and be at the forefront of exploratory research for the domain.

This research was supported by IBM providing free access to IBM® Power System™ AC922 & LC922.

Posted by OpenPower Team