Generative AI & the senses
Multimodal AI is a model that can take in and combine several kinds of input, such as text, images, audio, and video, and often produce several kinds too.
'Modality' means a type of data. Text is one modality, images another, sound another. Early models handled one at a time. A multimodal model handles several together: you can show it a photo and ask a question about it in words, or give it a voice recording and get a written summary back.
The trick is to translate every kind of input into the same kind of internal representation, so that a picture of a dog and the word 'dog' land near each other inside the model. Once everything speaks the same internal language, the model can reason across them.
As of 2026, most leading assistants are multimodal. They read documents with charts, describe images for blind users, transcribe speech, and generate pictures. Robots and self-driving systems combine camera, radar, and map data in the same spirit.
You photograph the inside of your fridge, ask 'what can I cook tonight,' and the assistant lists recipes using the eggs, spinach, and cheese it can see.