Part 5 - The project became a platform because every feature needed a place to live.

There was a point where I stopped looking at AI Playground as:

chat + models

and started seeing:

runtime orchestration
+
document ingestion
+
retrieval
+
vision
+
memory policy
+
image generation
+
diagnostics
+
model delivery
+
platform integration

At that point, adding another feature without architecture would have been reckless.

The application is built around Clean Architecture with an MVI presentation layer, using Kotlin Multiplatform and Compose Multiplatform where shared code is useful, while keeping Android and iOS runtime work native where it should be.

The architecture is not there to make the code look “enterprise”.

It is there because the project has enough independent lifecycles that without boundaries, every change would become risky.

1. The layer map

The simplified flow is:

flowchart TD
    Action["User action"] --> UI["Composable"] --> Store["MVI Store"] --> UC["Use Case"] --> Repo["Repository / Coordinator"] --> Runtime["Platform runtime"]
    Runtime -.->|StateFlow| Store
    Store -.-> UI

The UI renders state.

The UI does not become the owner of model loading, parsing, native lifecycles and memory policy.

That single rule eliminates a lot of accidental coupling.

2. The project is deliberately modular

The repository is organized into decoupled Gradle subprojects, including areas such as:

:features:chat
:features:discover
:features:settings
:features:diagnostics
:features:attachment
:features:navigation
:gpu_engine
:whisper-stt
:testing
:shared
:composeApp
iosApp

The module names tell a story.

For example:

:features:attachment

owns:

document ingestion
file validation
format capabilities
ZIP/Deflate processing
semantic extraction

while:

:features:diagnostics

owns:

RAM
GPU/Vulkan inspection
hardware diagnostics
benchmarking

And:

:gpu_engine

owns the separate image-generation native C++ JNI backend.

This means the codebase has organizational boundaries that match architectural boundaries.

3. The MVI store should never become the application’s brain

One of the easiest mistakes in an MVI architecture is to keep adding functionality to the store because the store already has state.

A store starts like:

class ChatStore {
    fun send() {}
}

Then it grows:

model loading
model switching
streaming
watchdog
history
export
attachments
memory
voice
navigation

At that point:

ChatStore

is a monolith.

The project therefore extracts focused coordinators.

Examples include:

ChatModelCoordinator
ChatGenerationCoordinator
ChatStreamParser
ChatExportManager
ChatTitleSummarizer
SettingsParamTracker

The store coordinates.

The focused class owns one lifecycle or responsibility.

4. Why this matters especially for AI

AI features tend to be long-running and stateful.

For example:

model loading

has a different lifecycle from:

generation

which is different from:

chat persistence

which is different from:

memory monitoring

If one object owns all four, cancellation and failure paths become tangled.

The project instead makes boundaries explicit:

flowchart TD
    ML["Model lifecycle"] --> CMC["ChatModelCoordinator"]
    GL["Generation lifecycle"] --> CGC["ChatGenerationCoordinator"]
    MP["Memory policy"] --> MAM["Memory admission / monitor"]

That is plain software engineering.

The AI aspect just makes the cost of getting it wrong higher.

5. Heavy work must remain off the main thread

The project explicitly routes heavy work to:

Dispatchers.IO
Dispatchers.Default

Examples:

file I/O
model verification
database work
document parsing
hashing
native inference dispatch
image decoding

A representative pattern:

scope.launch {

    val data = withContext(Dispatchers.IO) {
        repository.loadData()
    }

    state.update {
        it.copy(data = data)
    }
}

The main thread should mostly be doing:

input
layout
render
accessibility
animation

not:

read a 4 GB model package

6. The UI should appear immediately

One recurring class of performance issue was not a slow algorithm.

It was blocking screen initialization.

A user taps Settings.

The application starts inspecting everything.

The screen waits.

Then, several seconds later, content appears.

I would rather do:

flowchart TD
    Tap["tap"] --> Shell["render shell immediately"]
    Shell --> BG["load data in background"]
    Shell --> Progress["show progress / placeholders"]
    BG --> Content["populate content"]
    Progress --> Content

This is particularly important on lower-end devices.

Perceived performance matters.

7. Platform-native identity still matters

KMP gives me a way to share the parts that genuinely benefit from sharing.

It does not require me to make Android and iOS look identical.

For Android, I want:

Material 3
Android interaction patterns
Android navigation
Android accessibility conventions

For iOS, I want:

SwiftUI
Apple navigation patterns
native typography
native interaction
glass-style treatment where appropriate

The shared layer can say:

generation is active

The Android layer can decide how that feels like Android.

The iOS layer can decide how that feels like iOS.

That is a much better interpretation of multiplatform development than cloning one platform’s UI onto the other.

8. Accessibility is easier when it is designed in

The application uses shared design tokens and explicit policies for things such as:

touch targets
reduced motion
localized strings
RTL layout
contrast

The project targets a minimum touch size of:

48 dp

and keeps motion behavior behind an explicit reduced-motion policy.

Localization is also treated as a system constraint rather than a final translation pass.

The current project maintains eleven locales.

That matters because the amount of dynamic content in an AI application makes layout assumptions fragile.

9. Invariants are more useful than “best practices”

As the project grew, I started writing down statements that must remain true.

Examples:

Inference must not require network access.

Memory admission happens before expensive model loading.

Memory failure does not trigger meaningless backend retries.

Heavy work never blocks the UI thread.

Native runtimes have explicit lifecycle and unload paths.

Production builds cannot report fake or simulated inference success.

Attachment controls reflect actual model capabilities.

These statements are much easier to test than vague guidance such as:

"Keep things fast."

An invariant can be turned into a regression test.

10. The production-reality invariant is particularly important

I want to call this one out because it keeps the project honest.

An AI demo can be made to look successful very easily:

flowchart TD
    Start["start"] --> Prog["animate progress"] --> Place["return placeholder"] --> Done["show 'done'"]

That is not inference.

The project has an explicit production-reality gate that rejects simulated/emulated execution in production paths and prevents synthetic success from being presented as a real model result.

This matters even more when integrating third-party runtimes.

The application should either:

run the real runtime

or:

say that the runtime is unavailable

There should not be a third state called:

looks like it worked

11. Testing mirrors the architecture

The project contains tests across shared domain logic, Android host behavior, Compose/UI behavior and native integration boundaries.

The most useful way to understand them is by category:

Model tests
    -> compatibility
    -> capability
    -> memory

Runtime tests
    -> load
    -> routing
    -> cancellation

Document tests
    -> parsing
    -> malformed inputs
    -> format detection

Image tests
    -> memory safety
    -> resolution
    -> package validation
    -> progress
    -> race conditions
    -> true cancellation

Persistence tests
    -> migrations
    -> history integrity

Architecture tests
    -> invariant enforcement

That is the testing strategy I want for a system like this.

12. A database bug reminded me why architecture tests matter

The project uses SQLDelight for persistence.

A migration added cascading foreign keys from chat messages and generated images back to their thread.

That made the intended relational behavior better:

flowchart TD
    Thread["delete thread"] --> Msg["delete dependent messages"]
    Thread --> Img["delete dependent images"]

But a replace-style write path could accidentally delete the parent row and trigger the cascade.

The persistence operation looked like:

upsert

but the semantics were closer to:

delete
+
cascade
+
insert

The fix was to use:

UPDATE
+
INSERT OR IGNORE

instead of a destructive replace.

That is a perfect example of why I like testable boundaries.

Two individually reasonable database features can interact into a very unreasonable behavior.

13. Model catalogs are becoming a first-class subsystem

At the beginning, a model catalog could have been:

name
download button

It is now much closer to a compatibility database:

identity
format
architecture
quantization
runtime
platform
modality
required artifacts
memory profile
resolution profile
installation state

That enables queries like:

Find a model that:

    accepts images
    works offline
    supports this platform
    fits the current memory budget

The UI can then show useful information without hardcoding knowledge into every component.

14. I want model selection to become adaptive

Today, users still understand models better when the catalog exposes the details.

The next step I want is:

intent
+
device
+
memory
+
capability
+
thermal/battery state

producing:

model
+
runtime
+
context
+
resolution

Imagine two phones.

Device A

6 GB RAM
moderate pressure

Device B

12 GB RAM
low pressure

The same question:

Summarize this 80-page document.

does not necessarily need the same execution plan.

I would rather let the application choose:

Device A
    -> smaller model
    -> tighter context
    -> fewer retrieved chunks

than force a model choice that makes the app unreliable.

15. Local agents come after the foundations, not before

The obvious next feature is an agent.

But I do not want an agent that quietly runs around the application.

A local agent should have:

capability resolver
tool registry
permission boundary
memory budget
execution budget
cancellation
visible action log

The first tool set can stay tiny:

read local file
search local index

Then add more.

A reliable agent with two tools is more useful to me than an unpredictable agent with twenty.

16. Offline speech is still on the roadmap

The repository contains a Whisper-based on-device STT module, but the current v1.7.1 development branch explicitly defers offline Whisper STT to v1.8.0 and keeps the corresponding feature disabled.

That is exactly the kind of distinction I want to preserve in public writing.

The architecture is there.

The work is not being represented as shipped merely because a module exists.

The direction is:

flowchart TD
    Mic["microphone"] --> STT["local STT"] --> LLM["local LLM"] --> TTS["local TTS"] --> Speaker["speaker"]

But the project’s release discipline matters more than the diagram.

17. Model delivery deserves product-level engineering

A multi-file model package needs:

manifest
checksums
artifact validation
partial-download handling
canonical identity
repair

The project already separates model package validation and delivery from the core runtime.

That gives a clean boundary:

flowchart TD
    Cat["catalog"] --> DL["download"] --> Val["validate"] --> Inst["install"] --> Run["runtime"]

A runtime should not have to discover a corrupt download at the worst possible moment.

18. Benchmarking has to become a data set

I do not want future performance discussions to be:

"Model X feels fast."

I want something reproducible:

Device
RAM
SoC
OS
Model
Quantization
Runtime
Backend
Context
Time to first token
Tokens / second
Peak memory
Generation time

For image generation:

Device
RAM
Model
Resolution
Steps
Runtime
Peak memory
Generation time
Cancel latency

Then model recommendations can come from actual observations.

That will make the catalog better over time.

19. If I were starting again

I would build the project in this order:

1. single local model
2. runtime abstraction
3. capability inspection
4. memory admission
5. hardware diagnostics
6. persistence
7. document ingestion
8. RAG
9. vision
10. background inference
11. image generation
12. adaptive selection
13. local agent

The sequence is not accidental.

Each layer makes the next layer safer.

20. Projects I would give a senior engineer

Project A - Mobile Runtime Resolver

Input:

task
device telemetry
model catalog

Output:

runtime
model
context

Add an explanation trace:

selected_model = X

because:
- vision required
- platform supported
- peak memory below budget
- runtime available
- installed locally

Now the selection system is observable.

Project B - Resource-aware local agent

Build an agent that must remain under:

RAM budget
tool-call budget
time budget

Then make every action cancellable.

This is where local AI gets genuinely interesting.

Project C - Mobile AI benchmark harness

Build a command that records:

device
model
backend
latency
throughput
peak memory
temperature trend
battery change

Then store the results as JSON so they can become catalog data later.

That project would benefit the entire ecosystem.

What I learned from building this

The biggest lesson is not:

"Local AI is possible."

I knew that because the underlying runtimes already existed.

The lesson is:

Once the runtimes exist, the application engineer still has a large systems problem to solve.

You have to integrate:

model formats
runtime capabilities
native libraries
GPU backends
memory policy
file processing
retrieval
lifecycle
persistence
platform UX

and then make the whole thing feel like one product.

That is the part I find interesting.

The title of this entire journey

I think the project is best described as:

Beyond the Model

Because the model is only the center of the system.

Around it sits everything that makes the experience usable:

flowchart TD
    Model["MODEL"]
    Model --> Runtime["runtime"]
    Model --> Memory["memory"]
    Model --> Input["input"]
    Model --> Lifecycle["lifecycle"]
    Model --> UX["UX"]
    Runtime --> App["AI Playground"]
    Memory --> App
    Input --> App
    Lifecycle --> App
    UX --> App

That is the work I set out to do.

Not to replace llama.cpp.

Not to replace LiteRT.

Not to replace MLX.

Not to rewrite the inference research.

To integrate the pieces, respect their constraints, and build a product around them.

What I need from people who read this

I want the feedback that comes from real devices and real workflows.

Tell me:

Which device did you use?

Which model?

Which runtime?

What were you trying to do?

What was unexpectedly slow?

What failed?

What should the application have explained better?

What capability would make you use it every day?

A report like:

"8 GB phone, Q4 model, 4096 context,
model loaded, generation stable, image generation failed
because available memory was too low"

is incredibly useful.

A generic:

"Great app!"

is nice.

But it doesn’t help me improve the runtime decisions.

Final thought

I started with one constraint:

Keep the core AI experience on the device.

I then discovered that the constraint creates a chain of engineering problems:

flowchart TD
    A["existing runtimes"] --> B["integration"] --> C["capabilities"] --> D["memory"] --> E["native lifecycle"] --> F["documents / retrieval"] --> G["vision"] --> H["image generation"] --> I["platform UX"] --> J["testing + invariants"] --> K["adaptive execution"]

That is why I call this project Beyond the Model.

The model is essential.

But the software around the model is what turns an inference runtime into a product.

And I am still building it.

The series

• Part 1 - I Wanted the Model to Stay on the Phone
• Part 2 - The Day a Chat Box Became a Document Engine
• Part 3 - The Model Was Fine. The Phone Wasn’t.
• Part 4 - I Put an Image Generator Inside the Phone
• Part 5 - At Some Point I Stopped Building a Chat App

Project: AI Playground / Model Playground
Website: https://deardhruv.com
Repository: https://github.com/DearDhruv/Model-Playground

Part 4 - I did not build a diffusion engine. I integrated image-generation runtimes and then had to engineer everything around them.

Text generation had already forced me to learn about:

runtime selection
memory
native code
lifecycle
fallback

Image generation brought all of those problems back, but in a different shape.

The important distinction at the start is the same one I made in Part 1:

I did not invent the diffusion algorithms or write a new mobile image-generation framework from scratch.

AI Playground integrates established technologies and model ecosystems, including Google AI Edge/LiteRT multi-graph execution, MediaPipe image-generation paths, stable-diffusion.cpp/GGUF, Apple MLX and Core ML paths.

The engineering work was to give those runtimes a coherent product surface:

model catalog
+
package delivery
+
capability checks
+
memory admission
+
runtime selection
+
progress
+
cancellation
+
history
+
metadata
+
device-aware defaults

That is where this feature became interesting.

1. Image generation is a pipeline, not one model call

A simplified diffusion-style path looks like:

flowchart TD
    Prompt["prompt"] --> Enc["text encoding"] --> Cond["conditioning"] --> Denoise["denoising / DiT"] --> Latent["latent representation"] --> VAE["VAE decode"] --> Pixels["pixels"]

The project therefore treats image generation as a distinct runtime family instead of pretending it is the same thing as chat.

A simplified contract:

interface LocalImageGenerationRuntime {

    val loadedModel:
        StateFlow<LoadedImageGenerationModel?>

    suspend fun load(
        model: ImageGenerationModelRef
    ): Result<ImageGenerationLoadInfo>

    fun generate(
        request: ImageGenerationRequest
    ): Flow<ImageGenerationEvent>

    suspend fun cancel()

    suspend fun unload()
}

That interface gives me a clean boundary around the image pipeline.

2. Why a separate runtime was worth it

Text generation cares about:

tokens
context
KV cache
streaming text

Image generation cares about:

steps
latents
denoising
VAE
pixel buffers
output encoding

Forcing both into a single giant interface would make the abstraction harder to understand.

The better architecture is:

flowchart TD
    LMR["LocalModelRuntime"] --> Text["text generation"]
    LIGR["LocalImageGenerationRuntime"] --> Image["image generation"]

Then the domain layer can orchestrate both without pretending they have identical mechanics.

3. The Android image stack has multiple execution paths

The Android application currently integrates several image-generation paths.

The production v1.7.0 release includes:

flowchart TD
    LiteRT["Google LiteRT 2.2.0"] --> MultiGraph["multi-graph image pipeline"]
    subgraph MultiGraphPipeline["Pipeline"]
        TextEnc["Text Encoder"] --> DiT["DiT / denoising"] --> VAE["VAE"]
    end
    MultiGraph --> MultiGraphPipeline

    SDCPP["stable-diffusion.cpp"] --> GGUF["GGUF image models"]

as well as MediaPipe-based image-generation support for supported model families.

The application chooses the appropriate path through its image runtime abstraction.

That is orchestration work.

The underlying graph execution and model math come from the runtime/model ecosystems.

4. Bonsai 4B was a good example of integration complexity

One of the most technically interesting model families in the project is Bonsai Image Ternary 4B.

The project integrates a mobile-oriented quantized model bundle for the LiteRT path, and the image documentation describes its FLUX.2-klein-style execution details.

The interesting part for me was not “I found a 4B model”.

It was getting the entire artifact and execution contract to line up:

model package
+
tokenization/text encoding
+
position information
+
FlowMatch-Euler sampling
+
latent packing
+
denoising
+
unpatchify
+
VAE decode
+
image output

When those pieces do not align, the failure can be:

model won't load

or worse:

it runs
but the output is wrong

That is why I treat model support as a verification problem, not a download problem.

5. Multi-graph execution changed the memory problem

A naive implementation might keep every major component resident:

text encoder
+
denoiser
+
VAE
+
temporary tensors
+
output

That can create a very high simultaneous peak.

The project instead uses a multi-stage graph path:

flowchart TD
    A["text encoder"] --> B["conditioning"] --> C["denoising"] --> D["latent"] --> E["VAE"] --> F["RGB output"]

The engineering objective is:

Keep the maximum simultaneously active working set under the device’s safe envelope.

That is more useful than simply quoting the total model bundle size.

6. Resolution is a resource policy

A user sees:

256 × 256
512 × 512

as image-quality settings.

The runtime sees:

width
+
height
+
latent dimensions
+
intermediate tensors
+
VAE buffers
+
backend workspace

So the project does not treat resolution as an arbitrary UI slider.

The model profile describes supported resolutions and native constraints, and the suitability logic considers those constraints during admission.

A simplified resolver:

fun resolveBestResolution(
    availableBytes: Long,
    supported: List<ResolutionRequirement>
): Resolution {

    return supported
        .filter { it.estimatedPeakBytes <= availableBytes }
        .maxByOrNull { it.width * it.height }
        ?.resolution
        ?: throw NoSafeResolutionException()
}

The actual project has a richer policy.

The important idea is:

The safest configuration is not necessarily the smallest configuration. It is the largest configuration the current runtime can honestly support.

7. I learned that logical downscaling is not always a real memory optimization

One of the most useful bugs in the image work involved a very small requested resolution.

The UI could ask for something like:

128 × 128

but some models still performed their expensive internal work at a larger native resolution.

So:

requested size smaller

did not necessarily mean:

native peak smaller

That made the memory estimator depend on the model’s native resolution floor, not merely the final output dimensions.

The lesson is broader:

A UI control is only a performance control when the underlying runtime actually becomes cheaper.

8. Measured peaks are more useful than guesses

The project tracks measured memory peaks for selected image-model/configuration combinations and uses those measurements to improve suitability decisions.

Examples documented in the current image-generation work include values around:

Bonsai 512      ~3.1 GB
Bonsai 256      ~2.9 GB
SDXS-512        ~1.2 GB

These are not universal truths about every phone.

They are specific measurements used to calibrate the application’s own decisions.

That distinction is essential.

A benchmark is useful when it says:

this device
+
this model
+
this runtime
+
this resolution

not when it becomes a marketing number detached from the conditions under which it was measured.

9. Few-step models matter on a phone

The stable v1.7.0 release added verified few-step image models including model families such as:

DreamShaper 8 LCM
Realistic Vision HyperVAE
DreamShaper DMD2
SD-Turbo

The reason is obvious from the device perspective.

If a desktop workflow can afford dozens of denoising steps, a mobile workflow benefits enormously when the selected model can produce useful output in very few steps.

For a documented SDXS-512 test, the project observed:

1 step -> ~31 s
2 steps -> ~49 s

on a OnePlus 7 Pro.

That observation shaped model ranking.

The best mobile model is not automatically the biggest model.

It can be the model that gives the best quality / time / memory trade-off.

10. Progress is a real UX feature

A three-minute operation cannot realistically show:

Generating...

and expect the user to stay calm.

The image store therefore models explicit stages such as:

IDLE
PREPARING
GENERATING
DECODING
SAVING
COMPLETE

and exposes step progress.

Conceptually:

Step 1 / 4
Step 2 / 4
Step 3 / 4
Step 4 / 4

The UI can then explain that the first step may include the expensive warmup/paging phase rather than leaving the user wondering why the first increment takes longer.

11. I made cancellation cross the native boundary

The image generation path supports true cancellation rather than only hiding the progress UI.

The architecture is:

flowchart TD
    UI["Compose"] --> Store["ImageGenerationStore"] --> Runtime["LocalImageGenerationRuntime"] --> Bridge["platform bridge"] --> Loop["native generation loop"]

The project specifically tests cancellation and generation race conditions.

That includes resetting stale cancellation state at the beginning of a new generation.

The invariant is:

Cancel run A
    |
    v
run A stops

Run B starts
    |
    v
Run B starts clean

not:

Run A cancelled
    |
    v
Run B inherits A's state

12. Session continuation changed the product behavior

The stable release added support for continuing image generation while navigating between application surfaces.

The user can move from:

flowchart LR
    Image["Image"] --> Chat["Chat"] --> Discover["Discover"]

without automatically destroying the running image-generation session.

At the same time, chat model switching is guarded during active image generation.

The user can be given an explicit choice:

Image generation in progress.

[ Stop & Load ]
[ Keep Generating ]

That is a good example of application-level orchestration around a native runtime.

13. Image outputs should be reproducible

The project embeds generation information into PNG output using W3C-style tEXt chunks.

The metadata path includes values such as:

Title
Author
Description
Software
Source
Comment
Parameters

A simplified metadata record:

data class PngGenerationMetadata(
    val prompt: String,
    val modelId: String,
    val modelName: String,
    val width: Int,
    val height: Int,
    val steps: Int,
    val seed: Long,
    val guidanceScale: Float
)

The encoder then writes those fields into the PNG.

This is a tiny feature with a surprisingly useful outcome:

The image can remember where it came from.

14. Atomic file output matters

Writing a generated image directly to its final path creates a nasty failure mode.

If the application dies halfway through, the gallery can end up with:

partially written image

The image storage layer therefore stages writes through temporary files before promoting them to the final destination.

Conceptually:

flowchart TD
    Gen["generate bytes"] --> Tmp[".tmp file"] --> Val["validate"] --> Rename["rename"] --> PNG["final PNG"]

And importantly, saving is decoupled from successful generation.

The project supports a save-failure outcome so a generated in-memory result does not have to be thrown away just because the filesystem failed at the last step.

That is exactly the kind of edge case that disappears in a demo and matters in a product.

15. Model delivery is part of the feature

A multi-file image model is not a single download.

The current image system separates:

application feature delivery

from:

model package delivery

The package validator checks the expected bundle and supports repair actions when the downloaded model is incomplete or invalid.

That gives me a clean flow:

flowchart TD
    Cat["catalog"] --> Comp["compatibility check"] --> DL["download package"] --> Val["validate manifest / files"] --> Inst["install"]

I do not want the runtime to discover a broken model package only after allocating gigabytes of memory.

16. Image model catalogs need canonical identity

As multiple resolutions and legacy bundle layouts appeared, the catalog itself needed cleanup.

The v1.7.1 work consolidated Bonsai 4B into a canonical multi-resolution catalog entry while keeping compatibility with legacy installations.

That matters because these should not become different logical products:

Bonsai 4B 256
Bonsai 4B 512

if they can be treated as:

flowchart TD
    Model["Bonsai 4B"] --> R1["supported resolution = 256"]
    Model --> R2["supported resolution = 512"]

The catalog should describe the model identity.

The runtime configuration should describe the execution choice.

17. Android and iOS must not be presented as identical

For Android, the current production path has strong, concrete local image-generation integration.

For iOS, the project contains native MLX and Core ML integration points, including an MlxImageEngineBridge and the shared iOS runtime boundary.

I want to be careful here because the repository itself distinguishes production reality from simulation.

The project has an explicit invariant that production builds must not emit synthetic success events or pretend that simulated inference is real.

So I do not want this article to say:

"iOS image generation is completely finished and production-equivalent."

unless the release artifacts and production gates prove that.

The honest statement is:

The iOS architecture includes native MLX/Core ML integration
paths and shared runtime contracts, while the Android path is
the clearer production reference in v1.7.0. Current iOS image
work remains subject to the repository's production-reality
gates and active development.

That may be less flashy.

It is also much more useful to another engineer reading the code.

18. The iOS bridge still teaches an architectural lesson

The bridge exists to keep Swift-native concerns behind a narrow boundary:

final class MlxImageEngineBridge {

    func loadModel(...)
    func generate(...)
    func cancel()
    func unload()
}

The KMP layer does not need to know how MLX manages:

Metal
memory
Swift tasks
native lifecycle

It only needs a stable contract.

That is exactly the same architectural principle as JNI on Android.

19. Build this yourself

Project 1 - Tiny diffusion studio

Start with:

one model
one resolution
one runtime

Then add:

progress
cancel
seed
resolution
gallery
metadata
memory admission

Do not start with sixteen models.

Make the lifecycle correct first.

Project 2 - Adaptive resolution selector

Build:

data class ResolutionRequirement(
    val width: Int,
    val height: Int,
    val estimatedPeakBytes: Long
)

and choose the largest safe resolution.

Then compare:

estimated peak

with:

measured peak

on physical devices.

Project 3 - Reproducible image artifacts

Write:

model
prompt
seed
steps
guidance
resolution
runtime
timestamp

into PNG metadata.

Then build:

flowchart TD
    Open["open image"] --> Read["read metadata"] --> Recreate["recreate request"]

That gives you a reproducible experimentation loop.

The image-generation lesson

By now the pattern was unmistakable.

I did not need another abstraction because “AI is complicated”.

I needed explicit boundaries because different runtimes, different resources and different platforms have genuinely different behavior.

The project now had:

text runtime
document ingestion
retrieval
vision
image runtime
memory policy
native bridges

That was a lot of moving parts.

At this point the obvious question was:

What architecture lets me keep adding these capabilities without turning the codebase into a giant application class?

That is where the story stops being about individual features and becomes about engineering the product as a platform.

Part 5: At Some Point I Stopped Building a Chat App

Part 3 - The difficult bugs were memory, native lifecycle, GPU drivers and operating-system behavior.

There is a point in a local AI project where the model stops being the hardest part.

For me, that point was when I started testing the application on actual phones instead of treating a successful development run as proof of correctness.

A model would load.

A prompt would work.

Then I would increase the context.

Add a document.

Switch the model.

Turn the screen off.

Run image generation.

And suddenly the “AI bug” turned out to be:

memory accounting
GPU initialization
native allocation
process lifecycle
stale state
background execution

That changed the question I was asking.

Not:

“Does this model work?”

But:

“When is it responsible to start this operation on this device?”

That distinction became one of the foundations of AI Playground.

1. Model size is not peak memory

Suppose a model artifact is:

3 GB

That number describes the artifact.

It does not describe the full runtime footprint.

The operation may need:

weights
+
KV cache
+
runtime state
+
temporary tensors
+
GPU buffers
+
image/vision components
+
application memory
+
allocator overhead

The project therefore maintains explicit memory estimation code and tests.

The conceptual model is:

estimated peak
=
weights
+
KV cache
+
runtime overhead
+
modality overhead
+
safety margin

This is deliberately a policy input, not a fake claim of perfect precision.

2. KV cache is why context settings matter

A user sees:

Context: 4096

as a text setting.

The runtime sees more work and more memory.

A simplified attention KV-cache relationship is roughly proportional to:

2
× layers
× KV heads
× head dimension
× context tokens
× bytes per element

The exact value depends on architecture and implementation.

The practical lesson is enough:

flowchart TD
    Context["larger context"] --> KV["larger KV cache"]
    Context --> Work["more prefill work"]
    Context --> Latency["potentially higher latency"]

This is why context budgeting from Part 2 is also a memory feature.

3. I added an admission step before model loading

Instead of:

loadModel()

the application thinks in stages:

flowchart TD
    Req["request"] --> Est["estimate"] --> Adm["admission"]
    Adm --> Allow["ALLOW"]
    Adm --> Warn["WARN"]
    Adm --> Reclaim["RECLAIM"]
    Adm --> Unload["UNLOAD_OTHER"]
    Adm --> Deny["DENY"]

The admission result can explain itself:

data class AdmissionResult(
    val decision: AdmissionDecision,
    val estimatedPeakBytes: Long,
    val safeAvailableBytes: Long,
    val reason: String
)

That last field is important.

The user should not have to reverse-engineer a native exception.

4. Physical RAM is not the same as usable RAM

An “8 GB phone” does not mean my process gets 8 GB.

The runtime cares about:

physical RAM
available RAM
process RSS
native memory
current model residency
system pressure

The project therefore exposes a platform-neutral telemetry contract:

interface RuntimeMemoryTelemetry {
    fun snapshot(): RuntimeMemorySnapshot
}

Android and iOS can then gather their own native signals without leaking platform details into shared domain code.

5. Memory pressure changes over time

A static preflight check is not enough.

Imagine:

T0
model fits

T1
user opens another app

T2
system pressure rises

T3
image preprocessing allocates buffers

T4
native backend reaches another allocation peak

So the project watches memory during long-running work as well.

The conceptual path is:

flowchart TD
    Preflight["preflight"] --> Load["load"] --> Gen["generation"]
    Gen --> Monitor["memory pressure monitor"]
    Monitor --> Normal["normal"]
    Monitor --> Warning["warning"]
    Monitor --> Critical["critical"]

At higher pressure, the application can stop starting new heavy operations and reclaim disposable resources.

6. The stale-pressure bug was a good lesson

At one point, a critical memory-pressure signal could outlive the condition that caused it.

That meant the application could later observe:

live available memory: healthy
historical pressure: critical

and still refuse an operation.

The fix was a decaying memory-pressure monitor.

Conceptually:

flowchart TD
    A["platform event"] --> B["remember severity"] --> C["time-based decay"] --> D["combine with live memory"] --> E["decision"]

I prefer that approach to simply ignoring pressure callbacks.

The historical signal still matters.

It just should not own the decision forever.

7. Another bug: counting the same memory twice

Image generation exposed a different class of accounting problem.

The application already knew the current image runtime had a resident footprint.

Another part of the decision logic counted the same weight footprint again.

The result was:

available memory
<
reported requirement

even though the resident model was already loaded and the request did not need to pay the full cost again.

The fix was to distinguish:

resident footprint

from:

incremental peak

That distinction is essential whenever multiple features share a runtime.

8. Switching models became a resource decision

Suppose:

chat model loaded

and then the user asks for image generation.

The application should not simply say:

memory too low

without considering whether the chat model can be unloaded safely.

A better decision tree is:

flowchart TD
    Req["new request"] --> Est["estimate incremental cost"]
    Est -->|fits with current residency| Run["run"]
    Est -->|doesn't fit| Unload["unload reclaimable model"]
    Unload --> Recheck["re-check"]
    Recheck -->|fit| Run
    Recheck -->|fail| Fail["fail"]

This is why the model lifecycle coordinator and memory admission policy are separate concepts.

One owns lifecycle.

The other decides whether the lifecycle transition is responsible.

9. GPU fallback is only useful if it is safe

The Android GGML integration uses a hierarchical backend approach:

flowchart TD
    Vulkan["Vulkan"] -->|fallback| OpenCL["OpenCL"] -->|fallback| CPU["CPU"]

But fallback should not happen blindly.

The runtime distinguishes things like:

GPU initialization failure
unsupported backend
missing binary
native runtime error
insufficient memory

The first few may justify trying another backend.

The last one may not.

That is the difference between:

fallback

and:

retry until the app dies

10. Why the Vulkan probe runs in another process

This is one of the most unusual decisions in the project, and one of the most practical.

A GPU driver is native software outside my control.

A buggy vendor implementation can crash a process during probing.

If the probe happens in the main application process:

GPU probe crash
      |
      v
whole app dies

So the project isolates Vulkan probing in a child process:

flowchart TD
    Parent["parent process"] -->|fork| Child["child process"]
    Child --> Probe["Vulkan probe"]
    Probe -->|success| OK["report OK"]
    Probe -->|crash| Survives["parent survives"]

The point is not that Vulkan is “bad”.

The point is:

An optional acceleration path should not be able to take down the product while you are deciding whether it is available.

That is a systems-engineering decision around someone else’s runtime.

11. Native libraries need build-time guarantees too

The Android application packages native pieces for the selected runtime paths, including the llama/GGML stack and GPU backends, along with the separate image-generation JNI module.

The repository also contains scripts for engine setup, native builds, QA and invariant checks.

That matters because:

source code says backend exists

does not mean:

APK actually contains backend

Packaging is part of runtime correctness.

12. Background execution is another resource problem

A local generation can run longer than a user expects.

The user starts it.

Then:

screen off

or:

navigate away

The operating system is now part of the runtime.

The production Android architecture uses a foreground inference path for long-running inference, with the appropriate service type and a visible user-facing notification. The image-generation feature also supports session continuation while navigating between application surfaces.

The lifecycle looks conceptually like:

flowchart TD
    Start["start"] --> Session["foreground inference session"] --> Work["native work"]
    Work --> Cancel["cancel"]
    Work --> Finish["finish"]
    Cancel --> Release["release"]
    Finish --> Release

That is not an implementation detail.

It is how I tell the operating system:

This work is important, visible and currently active.

13. Cancellation is a real contract

A button labeled:

Cancel

does not prove cancellation.

Real cancellation has to travel through the stack:

flowchart TD
    UI["UI"] --> Store["Store"] --> Coord["Coordinator"] --> Cancel["runtime.cancel()"] --> Native["native cancellation"] --> Exit["native loop exits"]

The project has explicit tests around true cancellation and race conditions because stale cancellation state can otherwise poison the next generation.

A subtle bug looked like:

Run A
  |
cancel
  |
Run A fails

Run B
  |
stale isCancelled = true
  |
Run B immediately stops

The fix was to reset cancellation state at the start of every new run.

Simple.

Easy to miss.

Exactly the kind of thing that deserves a regression test.

14. Screen-awake state needs ownership too

The project uses a reference-counted screen-awake controller.

The logic is:

acquire
  -> count + 1

release
  -> count - 1

count > 0
  -> keep awake

count == 0
  -> restore normal behavior

Why?

Because multiple operations can overlap.

If operation A turns the state off when it completes while operation B is still active, B loses its protection.

Reference counting makes ownership explicit.

15. Trimming should target disposable memory first

When the system reports pressure, I do not want the first response to be:

unload the model

Model loading is expensive.

The better sequence is to reclaim things that are genuinely disposable:

image caches
temporary buffers
native allocator caches
short-lived objects

and preserve expensive resident state where the resource envelope still permits it.

This leads to a general principle:

Reclaim what is cheap to recreate before destroying what is expensive to rebuild.

16. Resource-management code needs real tests

The repository contains tests for things such as:

KV-cache estimation
model-memory estimation
memory-pressure classification
resident model footprint
model-load decisions
model switching
image memory admission
runtime memory snapshots

That is not accidental.

The memory layer can produce incorrect user-visible behavior even when the inference engine itself is perfectly healthy.

So it deserves unit-level coverage independent of native inference.

17. The hardware test matrix matters

I think about testing in dimensions:

flowchart TD
    Hardware["Hardware Test Matrix"]
    Hardware --> RAM["RAM Tier
(low / mid / high)"]
    Hardware --> Model["Model
(text / vision / image)"]
    Hardware --> Op["Operation
(load / generate / switch / background / cancel)"]

The question is:

Where are the boundaries?

Not:

“Does my phone work?”

That is how I want to collect real device knowledge over time.

Build this yourself

Project 1 - Mobile model admission library

Create:

data class DeviceMemory(
    val physicalBytes: Long,
    val availableBytes: Long,
    val processRssBytes: Long
)

data class ModelRequirement(
    val weightsBytes: Long,
    val kvCacheBytes: Long,
    val runtimeBytes: Long
)

and:

fun decide(
    memory: DeviceMemory,
    requirement: ModelRequirement
): AdmissionDecision

Then add:

resident model state
+
memory pressure
+
safety margin
+
reclaim strategy

Test the decision engine before connecting a real model.

Project 2 - GPU crash-isolated probe

Build:

flowchart TD
    Main["main process"] -->|fork| Child["child process"]
    Child --> Probe["GPU probe"]
    Probe --> Report["report success / failure"]

Intentionally make the child fail.

Your parent should survive and choose a safe fallback.

Project 3 - Long-running local inference

Build a local inference session that survives:

screen off
activity recreation
navigation
cancellation

and records:

start
finish
duration
cancel latency
peak memory
completion state

That is a much more realistic mobile-AI exercise than another chat screen.

The systems lesson

By now, I had learned something I wish I had written down on day one:

Mobile AI reliability is mostly about making fewer bad assumptions.

Do not assume:

8 GB RAM = 8 GB available to me

Do not assume:

GPU available = GPU reliable

Do not assume:

model file size = peak memory

Do not assume:

cancel button = cancellation

Do not assume:

background process = process that will remain alive

And do not assume:

runtime failure = model failure

Once I started treating those as explicit contracts, the application became much easier to reason about.

Then I did something that added a completely different class of pressure to the system.

I tried to generate images locally.

Part 4: I Put an Image Generator Inside the Phone

Part 2 - I thought “attach a PDF” would be a UI feature. It became an ingestion pipeline.

Once local chat was reliable enough, I wanted the next capability that makes a local assistant genuinely useful:

Let me ask questions about my own files without uploading them.

That sounds like:

paperclip icon
+
file picker

It is not.

A PDF is a container. A Word document is structured data. An Excel workbook is a workbook model. A scanned document may have no text layer at all. An image may contain words, diagrams, tables and visual context simultaneously.

And a mobile language model should not receive an entire 300-page document just because the user asked one question.

That is the point where the application changed from a chat client into a local document-processing system.

1. The simplest version is also the wrong one

The obvious implementation is:

flowchart TD
    File["file"] --> Extract["extract everything"]
    Extract --> Huge["huge string"]
    Huge --> Prompt["prompt"]
    Prompt --> LLM["LLM"]

For a tiny file, this works.

For anything serious, it creates three problems at once:

too much context
too much memory
too much irrelevant information

So the application needs an intermediate representation.

2. I gave attachments their own architecture

The repository separates attachment processing into a dedicated feature module.

The pipeline is:

flowchart TD
    A["selected file"] --> B["file identity"]
    B --> C["capability validation"]
    C --> D["format processor"]
    D --> E["semantic document blocks"]
    E --> F["chunking"]
    F --> G["local index"]
    G --> H["retrieval"]
    H --> I["context budget"]
    I --> J["local model"]

This is deliberately different from chat.

The chat feature should not know how to parse a DOCX package.

The document feature should not know how a Composable renders a message bubble.

That separation kept the scope under control.

3. I stopped trusting file extensions

A filename can say:

report.pdf

while the bytes are not actually a PDF.

So the attachment pipeline uses magic-byte inspection and file identity checks before handing content to a processor.

Conceptually:

fun detectType(bytes: ByteArray): FileType =
    when {
        startsWith(bytes, "%PDF-") ->
            FileType.PDF

        looksLikePng(bytes) ->
            FileType.PNG

        looksLikeZipContainer(bytes) ->
            detectOfficeOrOpenDocument(bytes)

        else ->
            FileType.UNKNOWN
    }

The principle is:

The extension tells me what the user called the file. The content tells me what it is.

That distinction is useful for correctness and for avoiding parser confusion.

4. The KMP decision mattered here

The document engine supports a broad family of text and structured formats through shared code, including formats represented in the project as:

TXT
LOG
MD
CSV
TSV
JSON
XML
YAML

DOCX
DOC
XLSX
XLS
PPTX
PPT

RTF
ODT
ODS
ODP

The important architectural choice was to keep the core ingestion path pure Kotlin Multiplatform where practical.

I did not want:

Android parser

and:

iOS parser

to slowly diverge.

Instead:

flowchart TD
    Contracts["shared parser contracts"] --> Repr["shared semantic representation"]
    Repr --> Android["Android UI"]
    Repr --> iOS["iOS UI"]

That is exactly the kind of work KMP is good at.

5. Container formats are where things stop looking like “documents”

Several office formats are effectively package containers.

Inside a single user-visible file I may have:

XML
relationships
media
styles
metadata
embedded objects

That pushed the document subsystem toward a bounded ZIP/Deflate implementation.

The key word is bounded.

I do not want a parser to blindly inflate an archive into memory.

The conceptual path is:

flowchart TD
    A["archive entry"] --> B["validate entry"]
    B --> C["enforce output limit"]
    C --> D["bounded decompression"]
    D --> E["parse required content"]

That is normal systems engineering.

It just happens to be sitting underneath an AI feature.

6. I needed a semantic representation

A giant String loses structure.

Suppose the document contains:

Chapter 4
4.1 Introduction
paragraph...
table...
image...

I want the retrieval layer to understand that 4.1 Introduction is not just another sentence.

A simplified representation looks like:

sealed interface DocumentBlock

data class Heading(
    val level: Int,
    val text: String
) : DocumentBlock

data class Paragraph(
    val text: String
) : DocumentBlock

data class Table(
    val rows: List<List<String>>
) : DocumentBlock

data class ImageReference(
    val id: String
) : DocumentBlock

This gives later stages enough structure to carry provenance.

7. Retrieval is where local RAG becomes useful

A large document can be indexed once and queried many times.

The flow becomes:

flowchart TD
    Doc["document"] --> Parse["parse"] --> Chunks["chunks"] --> Index["local index"]

Then:

flowchart TD
    Q["user question"] --> R["retrieve relevant chunks"] --> Rank["rank"] --> Budget["budget"] --> LLM["LLM"]

This is not only about quality.

On a phone, it is also about resource control.

8. Why context budgeting matters on a phone

Suppose my retrieved document text is:

25,000 tokens

Even if the model accepts that context, sending all of it means more:

prefill work
KV cache
latency
memory
battery

So the retrieval layer is deliberately followed by a strict context budget.

The architecture is:

flowchart TD
    A["retrieve generously"] --> B["rank carefully"] --> C["budget aggressively"] --> D["generate"]

A simplified version:

var used = 0

for (result in rankedResults) {

    val snippet = formatForContext(result)

    if (used + estimateTokens(snippet) > budget) {
        break
    }

    context.append(snippet)
    used += estimateTokens(snippet)
}

The exact estimator can vary.

The policy should not.

9. Why I prefer hybrid retrieval

Keyword search is excellent at exact identifiers:

API-1847
SKU-42
Section 9.2

Semantic retrieval is better when the user asks:

How does the company verify customer identity?

but the document says:

Identity verification is performed during onboarding...

So I want both signals:

             query
               |
       +-------+-------+
       |               |
       v               v
 lexical search   semantic search
       |               |
       +-------+-------+
               |
               v
         merge + rank
               |
               v
          top chunks

The project exposes a hybrid retrieval coordinator so this logic stays in the domain layer.

10. Provenance matters

A local document assistant should not just produce:

The contract allows termination after 30 days.

I want it to be possible to say:

Source:
contract.pdf
Page 17
Section 8.2

The retrieval result therefore carries citation information.

Conceptually:

data class ChunkCitation(
    val chunkId: String,
    val documentId: String,
    val fileName: String,
    val section: String?,
    val pageNumber: Int?,
    val score: Float
)

This is one of the places where I think product trust and systems design meet.

The answer is only as useful as the user’s ability to verify it.

11. Then I had to handle images

Documents are often visual.

A scanned PDF may contain:

image of page

with no usable text layer.

A screenshot may contain:

UI
buttons
text
icons
charts

A photo may contain:

objects
people
text
spatial relationships

That means there are two different tools:

OCR

and:

vision understanding

12. OCR and vision are not the same feature

OCR asks:

What text is visible?

Vision asks broader questions:

What is this?
What is happening?
How are these objects related?
What does this chart appear to show?

A screenshot might produce:

OCR
----
Settings
Account
Privacy
Notifications

while a vision-capable model can reason:

This is the Settings screen.
The Privacy option appears below Account.

I do not want to choose one permanently.

The application can use:

flowchart TD
    Img["image"] --> OCR["OCR"]
    Img --> Vision["vision"]
    OCR --> Combined["combined context"]
    Vision --> Combined

depending on the model and the task.

13. Vision is another capability check

This is where the model capability system proved useful again.

A user picks a text-only model.

Then attaches a photo.

The worst UX is:

Send
  |
  v
native runtime exception

The better UX is:

flowchart TD
    Model["selected model"] --> Resolver["capability resolver"]
    Resolver -->|image input supported| Enable["enable"]
    Resolver -->|not supported| Disable["explain limitation &
disable incompatible action"]

This is a direct consequence of treating model support as data.

14. Image dimensions are part of the resource policy

A camera photo might be:

4032 × 3024

That does not mean the vision model should receive those exact dimensions.

A mobile image path needs something like:

flowchart TD
    A["original image"] --> B["orientation correction"] --> C["model-aware resize"] --> D["bounded pixel representation"] --> E["vision runtime"]

Otherwise a simple attachment can create an unnecessary memory spike.

Again:

Input preprocessing is part of the runtime contract.

15. The hard cases are the real feature

A parser demo with one clean PDF is not enough.

The test matrix I care about looks more like:

empty file
truncated file
corrupt PDF
corrupt ZIP
large archive
wrong extension
invalid text encoding
image-only PDF
scanned PDF
very long document
duplicate attachment
unsupported type
unsupported model modality
missing runtime companion artifact

The project contains dedicated attachment and capability tests because this is exactly where regressions appear.

A good document system is not the one that parses one beautiful document.

It is the one that fails predictably when the input is ugly.

16. The final local document loop

The whole workflow now looks like:

flowchart TD
    Q["'Find the termination clause in my contract.'"] --> Index["local document index"]
    Index --> Retrieval["hybrid retrieval"]
    Retrieval --> Chunks["relevant chunks"]
    Chunks --> Budget["context budget"]
    Budget --> LLM["local LLM"]
    LLM --> Ans["answer + source"]

The contract never needed to become an upload.

That was the original reason for doing this locally.

17. Build this yourself

Project 1 - Offline PDF Q&A

Start simple:

flowchart TD
    PDF["PDF"] --> Extract["text extraction"] --> Chunk["chunking"] --> Keyword["keyword retrieval"] --> Model["local model"]

Then add:

embeddings
+
keyword search

Then add citations.

Do not start with an agent.

Learn retrieval first.

Project 2 - A vision/OCR laboratory

Build an app with:

flowchart TD
    Source["gallery / camera"] --> Check["capability check"] --> Resize["image resize"]
    Resize --> OCR["OCR"]
    Resize --> Vision["vision"]
    OCR --> Answer["answer"]
    Vision --> Answer

Test it on:

receipt
chart
screenshot
scanned form
street photo
product label

The interesting results will not all be successful.

That is the point.

Project 3 - File capability matrix

Build a matrix like:

                    Model A     Model B
TXT                    ✓           ✓
PDF                    ✓           ✓
DOCX                   ✓           ✓
IMAGE                  ✗           ✓

Make the UI derive attachment availability from it.

Now your UI is no longer guessing.

What changed because of documents

Before attachments, the core question was:

“How do I run a model locally?”

After attachments, the real question became:

“What should the model see?”

That is a more interesting product problem.

And once documents, images and models all started competing for the same phone resources, another question became unavoidable:

What happens when the phone itself says no?

Part 3: The Model Was Fine. The Phone Wasn’t.

Part 1 - I Wanted the Model to Stay on the Phone

The engineering journey of integrating established on-device AI runtimes into one multimodal mobile product - and everything it takes to make the pieces behave like a coherent system

There was one constraint behind AI Playground that shaped almost every architectural decision that came after it:

I did not want the core AI workflow to depend on a server.

I want to be precise about what that means.

I did not build a new mobile inference engine. I did not invent a new way to execute transformer models on phones. The project stands on top of established work from the open-source and platform ecosystems - llama.cpp, GGML, Google AI Edge/LiteRT-LM, MediaPipe, Apple MLX, and other native technologies.

My job was different.

I wanted to take those building blocks and make them behave like one application.

That meant answering the questions that only appear once you move from “model demo” to “mobile product”:

Which runtime should handle this model?

Does this model actually support the requested input?

Can this device safely load it?

What happens when acceleration fails?

How do I keep native work away from the UI thread?

What happens if the user switches models mid-operation?

How do I make Android and iOS feel like themselves
without duplicating the entire application?

That is where the project became interesting.

1. The promise is simple. The engineering is not

From a user’s point of view, the experience sounds almost boring:

flowchart TD
    A["Install application"] --> B["Download model"] --> C["Turn off network"] --> D["Start conversation"] --> E["Receive answer"]

The repository makes the offline-first boundary explicit.

The inference path is local:

flowchart TD
    Prompt["Prompt"] --> App["Shared application logic"]
    App --> Runtime["Selected local runtime"]
    Runtime --> Native["Native inference"]
    Native --> Response["Local response / history"]

The internet-enabled pieces are separated from inference:

flowchart TD
    Network["Network"] --> Discovery["Model discovery"]
    Network --> Download["Model download"]
    Network --> Search["Live Hugging Face search"]
    Network --> Other["Other explicitly allowed surfaces"]

That separation is more important than the phrase “offline”.

A product can call itself local while still quietly sending data through a logging service, a fallback endpoint, a remote parser, or a convenience API.

I wanted the architecture to make accidental data movement harder.

2. I needed Android and iOS to share the right things

The project uses Kotlin Multiplatform and Compose Multiplatform, but I never wanted “multiplatform” to mean “pretend both platforms are identical”.

The architecture is closer to:

flowchart TD
    subgraph KMP["Shared KMP"]
        Domain["Domain"]
        Data["Data"]
    end
    KMP --> Boundaries["Runtime boundaries"]
    Boundaries --> Android["Android
Kotlin / C++"]
    Boundaries --> iOS["iOS
Swift / Metal"]

The shared layer is responsible for:

models
use cases
repositories
policies
state
feature contracts

Platform code owns things that actually depend on the operating system or native acceleration stack.

That is the line I keep coming back to:

Share the decision. Keep the hardware-specific execution native.

3. The first abstraction I cared about was the runtime boundary

Once the app had more than one way to execute a model, the UI could no longer be allowed to know all of them.

I did not want:

if (model.isLlamaCpp()) {
    // llama-specific UI logic
} else if (model.isMediaPipe()) {
    // MediaPipe-specific UI logic
} else if (model.isMLX()) {
    // MLX-specific UI logic
}

That kind of condition spreads everywhere.

Instead, the project converged on a common runtime contract.

A simplified version is:

interface LocalModelRuntime {

    val loadedModel: StateFlow<LoadedModel?>

    suspend fun load(
        model: LocalModelRef
    ): Result<ModelLoadInfo>

    fun generate(
        request: GenerationRequest
    ): Flow<GenerationEvent>

    suspend fun cancel()

    suspend fun unload()
}

The UI asks for:

load
generate
cancel
unload

The runtime implementation decides how those operations are actually performed.

4. That does not mean the runtimes are interchangeable

The abstraction hides implementation details.

It does not erase real capability differences.

The current project integrates runtime families including:

Android
-------
GGUF -> llama.cpp / GGML
.task / .litertlm -> LiteRT-LM / MediaPipe
LiteRT multi-graph -> image generation
stable-diffusion.cpp -> GGUF image models

iOS
---
llama.cpp / Metal path
Apple MLX
Core ML

The application has to know which path is appropriate.

That led to one of the most useful concepts in the project:

A model is not “supported” just because the file downloaded successfully.

5. Model support became a capability problem

I started thinking of a model as a tuple of properties:

flowchart TD
    Model["Model"]
    Model --> F["Format"]
    Model --> A["Architecture"]
    Model --> M["Modality"]
    Model --> R["Runtime"]
    Model --> P["Platform"]
    Model --> C["Companion artifacts"]
    Model --> Req["Minimum runtime requirements"]
    Model --> Res["Resource profile"]

A model can be:

GGUF                 = yes
Android              = yes
llama.cpp             = yes
vision                = no

or:

.task                 = yes
MediaPipe runtime     = yes
text                  = yes
image input           = no

Or even:

architecture supported
runtime available
companion file missing

The last case is particularly important.

The UI needs to know before it sends the user into a native error path.

6. Capability detection belongs below the UI

The application has Android-side inspectors for GGUF and LiteRT-related artifacts.

The broader architecture looks like:

flowchart TD
    Artifact["Model artifact"] --> Inspection["Binary / metadata inspection"]
    Inspection --> Report["Capability report"]
    Report --> Resolver["Runtime resolver"]
    Resolver --> UI["UI"]

A simplified data structure:

data class ModelCapabilityReport(
    val textInput: Boolean,
    val imageInput: Boolean,
    val audioInput: Boolean,
    val supportedRuntimes: Set<Backend>,
    val platformSupported: Boolean,
    val missingArtifacts: List<String>
)

The actual project goes deeper than this example, but the principle is the important part.

Capability is data.

It should not be duplicated as assumptions in five screens.

7. Memory turned out to be the next abstraction

The next surprise was how misleading model file size can be.

If a model artifact is:

2 GB on disk

the application should not make a decision from:

if (fileSize < availableRam)

The runtime also needs:

weights
+ KV cache
+ native overhead
+ temporary allocations
+ accelerator buffers
+ multimodal components
+ application pressure

So the application has a model-memory estimation layer.

Conceptually:

estimatedPeakBytes =
      weightBytes
    + kvCacheBytes
    + runtimeOverheadBytes
    + modalityOverheadBytes
    + safetyMarginBytes

This is an admission heuristic, not a promise that the native allocator will use exactly that amount.

The job of the admission layer is:

Do I have enough evidence to start this expensive operation safely?

8. mmap changed how I thought about model files

For large native model artifacts, memory mapping is useful because it changes the relationship between:

storage
virtual address space
resident memory

The conceptual difference is:

flowchart LR
    subgraph ReadAll["Read-all approach"]
        direction TB
        F1["file"] --> B1["large memory buffer"]
    end
    subgraph MMap["mmap approach"]
        direction TB
        F2["file"] --> R2["mapped address range"]
        R2 --> P2["pages become resident as needed"]
    end

This does not make memory pressure disappear.

It does make it possible to reason more accurately about file size versus actual resident pages.

9. GPU acceleration is another integration layer

The Android GGML integration uses a modular backend layout.

The project packages a baseline CPU/native stack and loads acceleration backends dynamically.

The intended backend order is:

flowchart TD
    Vulkan["Vulkan
Preferred accelerated path"]
    OpenCL["OpenCL
Fallback accelerated path"]
    CPU["CPU
Final stable fallback"]
    Vulkan -->|fallback| OpenCL
    OpenCL -->|fallback| CPU

That is a project-level decision.

It is not an attempt to replace the GPU runtimes themselves.

The useful engineering work is in deciding:

when to use
when to fall back
how to detect failure
how to keep the application alive

10. Why I classified failures

This became important very quickly.

Consider these two errors:

GPU initialization failed

and:

not enough memory

They both mean:

load failed

but they are not the same problem.

The first may reasonably trigger:

Vulkan -> OpenCL -> CPU

The second often should not.

If RAM is the limiting factor, trying a different GPU backend does not magically create RAM.

So the routing logic is intentionally reason-aware:

for (backend in candidateBackends) {

    val result = runtimeFor(backend).load(model)

    if (result.isSuccess) {
        return result
    }

    if (result.isInsufficientMemory()) {
        return result
    }
}

return lastFailure

That small distinction eliminates a surprising amount of pointless retry behavior.

11. Native code is a boundary, not a magic escape hatch

The Android app has native C++ integration around llama.cpp and a separate native image-generation module.

The conceptual stack is:

flowchart TD
    Kotlin["Kotlin"] --> JNI["JNI"]
    JNI --> CPP["C / C++"]
    CPP --> Runtime["Third-party inference runtime"]
    CPP --> Backend["Accelerator backend"]

Once a feature crosses JNI, I need to think about:

ownership
lifecycle
threading
cancellation
error mapping
resource cleanup
process crashes

I learned to treat every native context like a resource with an explicit lifecycle:

flowchart TD
    Load["load"] --> Resident["resident"]
    Resident --> Generate["generate"]
    Resident --> Cancel["cancel"]
    Resident --> Unload["unload"]

The UI should never own that lifecycle directly.

12. Why I kept heavy work away from the main thread

The project has an explicit concurrency model:

flowchart TD
    UI["UI"] --> Store["MVI Store"]
    Store --> UseCase["Use Case"]
    UseCase --> Repo["Repository / Runtime"]
    Repo --> IO["Dispatchers.IO"]
    Repo --> Default["Dispatchers.Default"]

Heavy work includes:

file reads
SHA-256 verification
model inspection
database queries
document parsing
native inference dispatch
image decoding
generation orchestration

A simple example:

viewModelScope.launch {

    val result = withContext(Dispatchers.IO) {
        repository.inspectModel(modelPath)
    }

    state.update {
        it.copy(model = result)
    }
}

The point is not “always use IO”.

The point is:

Do not make the UI thread your accidental systems-processing thread.

13. Streaming needs pacing too

A native model can produce a lot of tiny generation events.

The UI does not need to render each event as a separate expensive state transition.

The project keeps a dedicated generation coordinator that coalesces updates at roughly a 65 ms UI cadence and uses longer watchdog windows for stalled generation.

Conceptually:

flowchart TD
    Events["Native events"] --> Coalesce["Coalesce (~65ms)"]
    Coalesce --> StateFlow["StateFlow"]
    StateFlow --> Compose["Compose"]

That is a small detail with a large effect on perceived performance.

14. I wanted failure to be understandable

One of the first product lessons was that:

"Load failed"

is almost useless.

A better message tells the user what happened:

This model is estimated to exceed the device's safe
memory budget at the selected context length.

Try:
- a smaller quantization
- a shorter context
- unloading the current model

The engineering policy becomes part of the UX.

That is something I want AI applications to do more often.

15. What the first version changed in my thinking

I started with:

chat screen
+
local model

I ended up with:

chat
+
runtime contracts
+
model capability inspection
+
memory admission
+
hardware diagnostics
+
native boundaries
+
persistence
+
MVI
+
KMP

The model was still the important dependency.

But the application around it had become the real system.

And that led to the next question:

What happens when the input is not text?

A PDF.

A spreadsheet.

A screenshot.

A photo.

That is where the next phase started.

Build this yourself

Project 1 - Offline Pocket LLM

Do not start with a giant model.

Start with:

one GGUF
one runtime
one chat screen
local persistence
streaming
cancel
unload

Instrument:

time to first token
tokens / second
load time
peak process memory
UI frame stability

Run the same test on CPU and accelerated paths.

The exercise is not to beat a server.

It is to understand what your device is actually doing.

Project 2 - Capability-aware model catalog

Build a model details screen that answers:

Can this model:
    run on this device?
    use this runtime?
    accept this input?
    fit the current memory budget?
    find all required artifacts?

Make the UI derive its attachment controls from this report.

That single project teaches an important product lesson:

The UI should reflect runtime truth, not just model metadata.

What I want to measure next

I still want better answers for:

How should thermal state affect runtime choice?

How much should battery state affect model selection?

How accurate can memory admission become without being expensive?

At what point is a smaller model with smarter retrieval
better than a larger model with a giant context?

Those questions belong in the next layers of the platform.

And that is exactly what happened.

Part 2: The Day a Chat Box Became a Document Engine