Authors:
(1) Daniele Malitesta, Politecnico di Bari, Italy and daniele.malitesta@poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it);
(2) Giuseppe Gassi, Politecnico di Bari, Italy and g.gassi@studenti.poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it);
(3) Claudio Pomo, Politecnico di Bari, Italy and claudio.pomo@poliba.it;
(4) Tommaso Di Noia, Politecnico di Bari, Italy and tommaso.dinoia@poliba.it. Abstract and 1 Introduction and Motivation 2 Architecture and 2.1 Dataset 2.2 Extractor 2.3 Runner 3 Extraction Pipeline 4 Ducho as Docker Application 5 Demonstrations and 5.1 Demo 1: visual + textual items features 5.2 Demo 2: audio + textual items features 5.3 Demo 3: textual items/interactions features 6 Conclusion and Future Work, Acknowledgments and References 3 EXTRACTION PIPELINE The overall multimodal extraction pipeline is represented in Figure 1. Through the Dataset module, the load and preprocess steps take place, assuming that the user is providing the input data and the YAML configuration file (overridable from command line) to customize the extraction. Then, the Extractor module is in charge of building the extraction model(s) by setting the backends and output layer(s). Finally, after the multimodal feature extraction, features are saved to the output path in numpy format (the Dataset module again controls this latter phase). As previously described, the whole process is orchestrated by the Runner module. 4 DUCHO AS DOCKER APPLICATION To fully exploit the GPU-speedup implemented in all backends we use for the multimodal feature extraction, one of the basic requirements is to setup a suitable development environment where the backends’ versions are compatible with CUDA and, optionally, cuDNN. Generally, setting a workstation where all such libraries/tools are correctly aligned is challenging. To this end, we decide to dockerize Ducho by making it into a Docker image (available on Docker Hub[3] ) with all packages already installed in a tested and safe virtualization environment on your physical machine and safe virtualization environment on your physical machine. Our Docker image is built from an NVIDIA-based image which comes with CUDA 11.8 and cuDNN 8 on Ubuntu 22.04, Python 3.8 and Pip, and our cloned repository having all Python packages already installed and ready to be used. A possible container instantiated from the image should specify the gpus to use from the host machine (this feature is currently available on Docker but it depends on the version of CUDA to be installed), and the volume you may want to use to save the framework’s outputs permanently Note that a generic container instantiated from our image would prompt the user to a shell environment where one could run custom multimodal feature extractions via the command line, and also create custom configuration files for the same purpose. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Daniele Malitesta, Politecnico di Bari, Italy and daniele.malitesta@poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it); (2) Giuseppe Gassi, Politecnico di Bari, Italy and g.gassi@studenti.poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it); (3) Claudio Pomo, Politecnico di Bari, Italy and claudio.pomo@poliba.it; (4) Tommaso Di Noia, Politecnico di Bari, Italy and tommaso.dinoia@poliba.it. Authors: Authors: (1) Daniele Malitesta, Politecnico di Bari, Italy and daniele.malitesta@poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it); (2) Giuseppe Gassi, Politecnico di Bari, Italy and g.gassi@studenti.poliba.it with Corresponding authors: Daniele Malitesta (daniele.malitesta@poliba.it) and Giuseppe Gassi (g.gassi@studenti.poliba.it); (3) Claudio Pomo, Politecnico di Bari, Italy and claudio.pomo@poliba.it; (4) Tommaso Di Noia, Politecnico di Bari, Italy and tommaso.dinoia@poliba.it. Abstract and 1 Introduction and Motivation Abstract and 1 Introduction and Motivation 2 Architecture and 2.1 Dataset 2 Architecture and 2.1 Dataset 2.2 Extractor 2.2 Extractor 2.3 Runner 2.3 Runner 3 Extraction Pipeline 3 Extraction Pipeline 4 Ducho as Docker Application 4 Ducho as Docker Application 5 Demonstrations and 5.1 Demo 1: visual + textual items features 5 Demonstrations and 5.1 Demo 1: visual + textual items features 5.2 Demo 2: audio + textual items features 5.2 Demo 2: audio + textual items features 5.3 Demo 3: textual items/interactions features 6 5.3 Demo 3: textual items/interactions features 6 Conclusion and Future Work, Acknowledgments and References Conclusion and Future Work, Acknowledgments and References 3 EXTRACTION PIPELINE The overall multimodal extraction pipeline is represented in Figure 1. Through the Dataset module, the load and preprocess steps take place, assuming that the user is providing the input data and the YAML configuration file (overridable from command line) to customize the extraction. Then, the Extractor module is in charge of building the extraction model(s) by setting the backends and output layer(s). Finally, after the multimodal feature extraction , features are saved to the output path in numpy format (the Dataset module again controls this latter phase). As previously described, the whole process is orchestrated by the Runner module. preprocess building extraction output 4 DUCHO AS DOCKER APPLICATION To fully exploit the GPU-speedup implemented in all backends we use for the multimodal feature extraction, one of the basic requirements is to setup a suitable development environment where the backends’ versions are compatible with CUDA and, optionally, cuDNN. Generally, setting a workstation where all such libraries/tools are correctly aligned is challenging. To this end, we decide to dockerize Ducho by making it into a Docker image (available on Docker Hub[3] ) with all packages already installed in a tested and safe virtualization environment on your physical machine and safe virtualization environment on your physical machine. Our Docker image is built from an NVIDIA-based image which comes with CUDA 11.8 and cuDNN 8 on Ubuntu 22.04, Python 3.8 and Pip, and our cloned repository having all Python packages already installed and ready to be used. A possible container instantiated from the image should specify the gpus to use from the host machine (this feature is currently available on Docker but it depends on the version of CUDA to be installed), and the volume you may want to use to save the framework’s outputs permanently Note that a generic container instantiated from our image would prompt the user to a shell environment where one could run custom multimodal feature extractions via the command line, and also create custom configuration files for the same purpose. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Making AI Recommendations Smarter with Visual, Text, and Audio Data

A New Way to Extract Features for Smarter AI Recommendations

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Ducho’s Big Bet: A Unified Future for Multimodal AI

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Ducho: A Unified Framework for Multimodal Feature Extraction in AI-Powered Recommendations

How to Develop Data-Driven AI Apps: A Guide to Making AI Services Directly From the Database

Building Multimodal Generative AI Systems: Architecture, Refinement, and Enhancement

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Modular Enhancements for Stable Diffusion Architecture

Ducho: A Unified Framework for Multimodal Feature Extraction in AI-Powered Recommendations

How to Develop Data-Driven AI Apps: A Guide to Making AI Services Directly From the Database

Building Multimodal Generative AI Systems: Architecture, Refinement, and Enhancement

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Modular Enhancements for Stable Diffusion Architecture

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps