Get the multimodal embeddings #117

Activity

1. hessaAlawwad commented on May 22, 2024

   

   hessaAlawwad

   on May 22, 2024

   Author

   More actions

   Please bear with me if my questions does not make sense but I am still learning.
   I see after I give it an input consist of two modality ( text and image) it retrurn two different embeddings.

   # Load data
   inputs = {
       ModalityType.TEXT: data.load_and_transform_text(text_list, device),
       ModalityType.VISION: data.load_and_transform_vision_data(image_paths, device),
   }
   
   with torch.no_grad():
       embeddings = model(inputs)
   
   print(embeddings[ModalityType.VISION])
   print(embeddings[ModalityType.TEXT])
   

   but I couldn't find something about getting one embeddings for the both.
2. lixinghe1999 commented on Jun 25, 2024

   

   lixinghe1999

   on Jun 25, 2024

   Last edited by lixinghe1999

   More actions

   As far as I know, there is no multimodal embedding.
   You got embedding for each modality and compare them (Text * Image) to see whether they match that the ideas of Imagebind and CLIP in short.

   If you insist to get one, naive way is to add or take the average of them, however I believe it is not appealling.

   👍React with 👍1Reacted by Yiru Wang