SwiftUI Notes
16 July 2026

Building an On-Device Vocab App with Core AI

Tooling•4 min read

An on-device vocabulary feature is not a single prompt. A person points at an object, expects the app to isolate it, then needs a useful word, translation, and example sentence. Treating all of that as one general-purpose model problem makes the feature harder to ship on a phone.

The better starting point is to decompose the experience. Use a vision model for the image task and a language model for the linguistic task, then design model delivery and first-run work as part of the feature itself.

Give each model one job

Apple's Core AI vocabulary demo uses SAM 3 for text-prompted image segmentation and a compact Qwen model for vocabulary generation. That split is useful beyond the demo: the segmentation model can return the cutout for the card, while the language model turns a label such as flower into content for the learner.

Two task-specific models also make the product easier to tune. You can change the language model without revisiting image preprocessing, and choose smaller variants that fit the storage and memory budget of the target device.

The coreai-models Swift package supplies model-specific helpers, so app code does not have to manage the underlying tensor inputs and masks directly. The segmentation call can stay close to the UI intent:

import CoreAIImageSegmenter
 
let segmenter = try await ImageSegmenter(resourcesAt: sam3ModelURL)
let result = try await segmenter.segment(image: photo, prompt: objectLabel)
let mask = result.segments.first?.mask

Use the result only after checking that a mask exists, and offer a normal vocabulary flow when the object cannot be segmented confidently. A camera feature should still be useful when the vision step is imperfect.

Generate a typed card, not a parsable string

Core AI language models can be passed to the familiar Foundation Models LanguageModelSession. This lets the vocabulary feature use guided generation instead of asking the model to invent a JSON shape in a text response.

import FoundationModels
import CoreAILanguageModels
 
@Generable
struct VocabularyCard {
    let term: String
    let translation: String
    let example: String
}
 
let model = try await CoreAILanguageModel(resourcesAt: qwenModelURL)
let session = LanguageModelSession(model: model)
let response = try await session.respond(
    to: "Create a Mandarin vocabulary card for \(objectLabel).",
    generating: VocabularyCard.self
)
let card = response.content

The value is not that a model becomes deterministic. It does not. The value is that the app receives the shape it needs to render, rather than maintaining a fragile parser for prose. Put language, tone, and the expected level of explanation in the prompt or session instructions, and test them with real learner inputs.

Treat specialization as a product state

Core AI specializes a portable .aimodel asset for the current device before inference. The specialized result is cached, so later loads are faster, but the first specialization of a large model can be noticeable.

Do not let that work begin after someone has already taken a photo. A better flow is:

  1. Introduce the optional camera vocabulary feature.
  2. Download the required models after the person opts in.
  3. Prepare the models while that introduction is on screen.
  4. Enter the camera flow only when the feature is ready.

Core AI Instruments can reveal whether a delay is model loading, specialization, or inference. If specialization is the expensive part, compile the model ahead of time with coreai-build. Ahead-of-time compilation moves the costly compilation step to the build machine, while the device still produces artifacts appropriate to its hardware and OS.

Keep large models out of the base download

The Core AI demo found that its two models added more than 1 GB to the app. Shipping that cost to every updater is a poor trade for an optional feature.

Background Assets is a better fit for this kind of delivery. It can manage separately downloaded asset packs, including on-demand content, while keeping the base app focused on the feature set everyone uses. The app should show download progress, handle unavailable storage or connectivity, and keep a non-AI vocabulary path available.

The central decision is not which model has the most parameters. It is whether the model lifecycle feels intentional. A feature that downloads deliberately, prepares before interaction, and returns typed content has a much better chance of feeling like an app feature rather than a model demo.

References