Beyond Text: Images, Vision & Voice

"Modal" just means a type of data: text, images, audio, video. A multimodal model can take in (and sometimes produce) more than one type. So instead of only chatting in text, you can hand it a photo of a receipt and ask "what's the total?", or pass an audio clip to transcribe. Why care: it turns messy real-world inputs (a scanned PDF, a voice note, a screenshot) into structured data your app can use. The big unlock for you is that it's the same kind of HTTP API you already call for text, just with an image or audio attached.

The technique most image generators use is called diffusion. The plain-English version: the model starts with a screen of pure random static (TV-snow noise) and, step by step, removes the noise in the direction that makes the picture look more like your words, until a clean image emerges. It learned to do this by being shown millions of image-plus-caption pairs. You don't need any math; the mental model is "sculpt a photo out of noise, guided by the prompt." From your side it's just an API call: send text, get back an image (as a URL or some base64-encoded bytes).

Generation goes text to image (you describe, it draws). Vision goes the other way: image to text (you show it a picture, it tells you what's there). A vision-capable model can describe a photo, read the words inside it, find objects, or pull structured fields out of a form. Think of generation as a printer and vision as a scanner-with-a-brain. Most modern chat models are vision-capable, so you use the very same chat endpoint you already call for text; you just include an image alongside your question instead of text alone.

Speech-to-text (STT, also called transcription) takes an audio file in and returns the words as text, your "voice note to string" converter. Text-to-speech (TTS) takes a string in and returns an audio file (MP3/WAV) of someone saying it. Mentally: STT is a parser (bytes to text), TTS is a renderer (text to bytes). Both are plain API calls: POST the audio (or text), get back text (or an audio file you stream to the browser). As of 2026 common options include OpenAI's transcription/speech endpoints and ElevenLabs. Nothing about your request/response habits changes.

Yes, and this is the most reassuring thing to internalize. Every one of these is an HTTPS POST with an Authorization header carrying an API key from an env var, a JSON or multipart body, and a JSON (or file) response you read. The differences are tiny: the body sometimes carries base64 image bytes or a multipart audio file instead of plain text, and the response is sometimes raw image/audio bytes you save to a bucket. Auth, error handling, retries, timeouts, rate limits, secrets in env, all your existing habits apply unchanged. You're not learning a new networking model, just new payload shapes.

It's a normal POST, just like any JSON API you already call. As of 2026, with OpenAI's image model (e.g. gpt-image-2) you'd hit v1/images/generations: ``js const r = await fetch("https://api.openai.com/v1/images/generations", { method: "POST", headers: { Authorization: Bearer ${process.env.OPENAI_API_KEY}, "Content-Type": "application/json" }, body: JSON.stringify({ model: "gpt-image-2", prompt: "a red bicycle on the moon, photorealistic", size: "1024x1024" }) }); const { data } = await r.json(); // data[0] holds the image (base64 bytes) `` The key is read from an env var, exactly as you'd protect any secret. You store the returned bytes in your own bucket (S3, etc.) and serve them like any other static asset.

You use the normal chat/messages endpoint, but the user message becomes a list of content blocks: one image block plus one text block. You can pass the image as a public URL or as base64-encoded bytes (a way of stuffing binary data into a JSON string). With Claude (model e.g. claude-opus-4-8 as of 2026) it looks like: ``json { "role": "user", "content": [ { "type": "image", "source": { "type": "url", "url": "https://.../receipt.jpg" } }, { "type": "text", "text": "What is the total amount on this receipt?" } ]} `` Tip: put the image before the text. The response comes back as ordinary text you read like any JSON field.

Classic OCR (optical character recognition) just transcribes pixels into raw text, in reading order, with no understanding, the way an old scanner dumps a wall of characters. A vision model reads AND comprehends: you can ask "give me the invoice number, date, and total as JSON" and it returns exactly those fields, even if they're scattered across a messy layout. So for structured extraction (receipts, forms, IDs) the model often replaces a brittle OCR-plus-regex pipeline. Trade-off: it costs more per page and can occasionally misread or invent a value, so validate critical numbers, just like you'd never trust unsanitized user input.

Three steps you already know how to wire up. (1) The user uploads the image to your backend; you forward it to a vision model as base64 or a temporary URL. (2) In your prompt, ask for strict JSON and name the fields: "Return only JSON: { merchant, date, total, line_items[] }." Many APIs also let you pass a schema so the output is guaranteed to match a shape. (3) Parse the JSON, then validate, e.g. confirm the line items sum to the total, before saving. That last step matters: treat the model's output like data from an external API you don't fully trust, not as gospel.

It's a multipart upload, exactly like any file-upload endpoint you've built (multipart is the standard way browsers/servers send a file plus form fields together). As of 2026, OpenAI's transcription lives at v1/audio/transcriptions; you pick a model (the older whisper-1 still works, or a newer one like gpt-4o-transcribe): ``js const form = new FormData(); form.append("file", fs.createReadStream("voice-note.m4a")); form.append("model", "gpt-4o-transcribe"); const r = await fetch("https://api.openai.com/v1/audio/transcriptions", { method: "POST", headers: { Authorization: Bearer ${process.env.OPENAI_API_KEY} }, body: form }); const { text } = await r.json(); // the transcript ` Then treat text` like any user-submitted string. Same upload muscle memory you already have.

Both patterns exist, and knowing which you're using matters. A truly multimodal model (today's mainstream chat models like Claude or Gemini) takes text and image in the same request and reasons over them together, e.g. "here's a chart image AND last quarter's numbers, do they match?" That's one API call. Alternatively you chain: run STT to get text, send that text to a separate model, then run TTS on the reply, three calls, like stages in a pipeline. Voice assistants are usually the chained kind. Use one multimodal call when the model must look and read at once; chain when each stage is a distinct, swappable step.

Plenty that aren't sci-fi. Vision: receipt/invoice scanning for an expense tool (photo to structured JSON), auto-generating alt-text for uploaded images to improve accessibility, moderating user uploads, reading IDs for identity checks, or turning a whiteboard photo into editable text. Voice: transcribing voice notes or meeting recordings into searchable text, letting users dictate instead of type, or a phone-style support bot (STT then a text model then TTS). The pattern is always the same: take a real-world input your users naturally produce (a photo, a spoken sentence) and turn it into structured data or a response your existing app logic can handle.

They're heavier than text, so plan for it. Images cost more per request than a text prompt, and a vision model bills partly by image resolution, so a huge 4000px photo costs more and is slower than a downscaled one; resize before sending. Generating an image or a chunk of speech can take several seconds, not milliseconds, so don't block a request thread, do it in a background job/queue and notify when done, or stream. Transcribing a long recording scales with audio length. Practical defaults: downscale images, cache results you'll reuse, set generous timeouts, and show the user a loading state. Treat it like calling a slow third-party API, because you are.

A few that bite people. (1) Trust: a vision model can confidently misread a digit on a receipt, so never auto-approve money or identity decisions without a validation check or human review. (2) Prompt injection via images: text inside an uploaded image ("ignore your instructions and...") can hijack the model, so treat image-borne text as untrusted input, just like user form data. (3) Generated images may carry invisible "this is AI" provenance metadata (a standard called C2PA) and have usage rules, so don't assume you own them outright. (4) Faces/PII: generating real people or storing voice recordings raises privacy and legal duties. (5) Cost surprises: a loop that regenerates images can run up a bill fast, so add limits.