What we're using to build Glimpses (Multi-modal AI)

Hey IH.

I'm Nihir -- a finishing PhD in Natural Language Generation (NLG). Over the past few years, I've been diving deep into the inner workings of Multi-Modal AI and Natural Language Generation. My research models improve performance of NLG systems by up to 15%.

I'm currently working on https://glimpses.ai -- which is a startup aimed at helping men look and dress better. Most of the entire backend is built utilising multi-modal models, and I'll use this post as an opportunity to explain a bit more about that.

Now, unless you've been living under a rock, you all have played with ChatGPT. ChatGPT is NLG model -- it takes language in as input, and outputs language.

Multi-modal AI is AI that can reason over multiple different modalities at the same time. A modality is essentially a data type. So, language would be one modality, vision would be another, tabular data a third and so forth. My experience lies in the language and vision modality so I'll focus on that for this post.

Now, those that there more technical here have likely played around with embeddings, or at least know what they are. In case you are not familiar, the best way to think about an embedding is as a way that an algorithm can understand the meaning of some data. To oversimplify, fundamentally, computers/algorithms can only deal with numerical representations of data. We want a way of telling a computer that the words "cat" and "kitten" are related, but the computer doesn't have a brain like we do. So how can it know that cat and kitten are meant to mean similar things? Embeddings give us a way of creating a representation to the algorithm, such that it can distinguish between cat and kitten. E.g. cat = [0.0, 0.1, 0.2]; kitten = [0.01, 0.11, 0.21]. We don't worry about what the numbers themselves mean. We just need to understand that similar words will have similar embeddings. (And that embeddings are 'definitions' to an algorithm).

The concept of an embedding can be extended beyond language. Infact, in a similar fashion, we can embed images. So if we have a picture of a dog, it will have an embedding. However, a different picture of the same dog will have a different embedding. Why? Because the lighting, pose, angle etc of the photo will be different. Yet, this aspect of similarity should still be captured within the embedding. What I mean by this is that if we had a picture of a hippo, the embedding of the hippo vector should be a lot more different than that of a dog. To provide an oversimplified example again:
dog_1 = [1.0, 1.1, 1.2]
dog_2 = [1.02, 1.12, 1.22]
hippo = [0.24, 2.61, 1.38]

Ok, now that we roughly understand how algorithms think about data, let me motivate multi-modal AI.

Wouldn't it be cool if we could have a way of comparing the language embeddings to the vision embeddings? Essentially, if we have the word "cat", we would want to have an embedding that could be directly compared to the embedding of an image of a cat. This is the core idea behind multi-modal AI. By creating a shared embedding space between different modalities, we enable our algorithms to understand and correlate data across these modalities. This has incredible applications, from improving search engines to creating more intelligent AI assistants that can understand both the text you write and the images you might refer to.

One of the most innovative works in multi-modal AI is CLIP. Developed by OpenAI, CLIP learns to understand images and text by being trained on a vast dataset of images and their corresponding textual descriptions. This training process allows CLIP to not only recognize objects within images but also understand the context and subtleties of how those objects are described in natural language. For example, CLIP can understand that a photo of a "sunlit forest" and the phrase "trees illuminated by the sun" refer to similar concepts, despite the different modalities of representation (visual and textual).

The brilliance of CLIP and similar multi-modal models lies in their ability to generate a shared representation space for both images and text. By doing so, these models can perform a variety of tasks that were previously challenging for AI, such as zero-shot classification, where the model can accurately classify images into categories it has never seen during training, simply by understanding the descriptions of those categories.

If you've made it this far, let me share something with you: https://glimpses.ai/search

Search is exactly where my start-up, Glimpses.ai, utilises multi-modal AI. We take images of items and, alongside running a bunch of classifiers on the items to determine different item attributes (e.g. colour, material, clothing type etc), we utilise multi-modal AI to create representations that can be searched for with natural language. This allows us to easily create flexible search without needing to manually tag items into categories. For example, if you visit the above webpage, you can search for "sport" and find examples of different clothing items that are for sports (despite the word sport not appearing anywhere in the title). Or you can use it the traditional way and search for terms such as "plain green t-shirt".

We use CLIP in a few more places on that website too. I'm cautious on this post being too long and it not evolving into me plugging my start-up. So if you wanted me to expand or have questions on the tech too, then happy to continue the discussion in the comments.