We will share about MeraLion (Multimodal Empathetic Reasoning and Learning In One Network), our generative AI efforts in Singapore’s National Multimodal Large Language Model Programme. Speech and audio information is rich in providing more comprehensive understanding of spatial and temporal reasoning in addition to social dynamics that goes beyond semantics derived from text alone. Cultural nuances and multilingual peculiarities add another layer of complexity in understanding human interactions. In addition, we will draw use cases in education to highlight research endeavors, technology deployment experience and application opportunities.

Manga is a Japanese content that has gained global recognition. Manga is a unique multimedia format that combines both images and text. We created a dataset called Manga109, composed of 109 manga comic books. In 2015, we released a dataset containing approximately 20,000 manga pages, and in 2018, we published an extended version with annotations for more than 500,000 objects, including characters and speech balloons on each page. It is the largest manga dataset in the world with such detailed manual annotations. Manga109 allows for academic use, and we have distributed over 2,000 copies of the dataset to date. Various research efforts have been made using this dataset, both domestically and internationally. For example, different groups have tackled tasks such as character recognition, expression recognition, dialogue recognition, speaker identification, and onomatopoeia recognition and more. In this talk, the journey of Manga109, from its beginning to the present, and show MangaUB benchmark dataset for rapidly advancing large multimodal models (LMMs) to assess the current state of LMMs’ manga comprehension.

The tremendous progress of deep-learning-based approaches to image understanding problems has inspired new advanced perception functionalities for autonomous systems. However, real-world perception systems often require models that can learn from large bulks of unlabeled and uncurated data with few labeled samples, usually costly to select and annotate. In contrast, typical supervised methods require extensive collections of carefully selected labeled data, a condition that is seldom fulfilled in practical applications. Self-supervised learning (SSL) arises as a promising line of research to mitigate this gap by training foundation models using various supervision signals extracted from the data itself, without any human-generated labels.While most popular SSL methods revolve around web image datasets, new diverse forms of self-supervision are starting to be investigated for autonomous driving (AD). AD represents a unique sandbox for SSL methods as it brings among the largest public data collections in the community with different paired sensors (multiple cameras, Lidar, radar, ultrasonics) and provides some of the most challenging computer vision tasks: object detection, depth estimation, image-based odometry and localization, etc. Here, the canonical SSL pipeline (i.e., self-supervised pre-training of a model and fine-tuning it on a downstream task) is revisited and extended to utterly new SSL approaches for computer vision and robotics (e.g., world models), but also to new downstream usages of pre-trained foundation models, such as cross-sensor distillation, auto-labelling, data mining, architecture re-purposing. This talk will provide a tour of different forms of foundation models across multiple sensor types equipping today's and tomorrow's vehicles in a quest towards annotation-efficient and reliable perception systems.