# Google AI Edge Gallery — Administrator & Developer Guide

## 1. Architecture Overview

Google AI Edge Gallery is an Android application that runs large language models entirely on-device using the LiteRT-LM and MediaPipe LLM Inference runtimes. The app is built with Kotlin, Jetpack Compose, and Hilt dependency injection, following a single-activity architecture with Compose navigation.

### High-Level Component Map

```
┌─────────────────────────────────────────────────────────┐
│  MainActivity (SingleActivity)                          │
│  ├── GalleryNavGraph (Compose Navigation)               │
│  │   ├── HomeScreen (categories → tasks → models)       │
│  │   ├── LlmChatScreen / LlmSingleTurnScreen           │
│  │   ├── BenchmarkScreen                                │
│  │   ├── ModelManagerScreen                             │
│  │   └── CustomTask.MainScreen (plugin screens)         │
│  └── ModelManagerViewModel (shared model lifecycle)     │
├─────────────────────────────────────────────────────────┤
│  Runtime Layer                                          │
│  ├── LlmModelHelper (model init, inference, cleanup)    │
│  ├── AICoreModelHelper (Google AI Core integration)     │
│  └── ModelHelperExt (extension functions)               │
├─────────────────────────────────────────────────────────┤
│  Data Layer                                             │
│  ├── ModelAllowlist (remote JSON → curated model list)  │
│  ├── SkillAllowlist (skill discovery + management)      │
│  ├── DataStoreRepository (protobuf-backed persistence)  │
│  ├── DownloadRepository + DownloadWorker (WorkManager)  │
│  └── Config / ConfigValue (runtime tuning)              │
├─────────────────────────────────────────────────────────┤
│  External Services                                      │
│  ├── Hugging Face (OAuth 2.0 + model file downloads)    │
│  ├── Firebase Analytics + FCM                           │
│  └── Google AI Core (optional, device-dependent)        │
└─────────────────────────────────────────────────────────┘
```

### Data Flow

1. On launch, the app loads the model allowlist for its version from GitHub raw content or a cached copy.
2. The home screen renders tasks grouped by `CategoryInfo`. Each task declares which model names it supports.
3. When a user selects a model, `DownloadWorker` fetches the `.litertlm` or `.task` file from Hugging Face via `HttpURLConnection`, using OAuth tokens when required.
4. `LlmModelHelper.initializeModelFn()` loads the downloaded model into the LiteRT-LM or MediaPipe runtime.
5. Inference runs entirely on-device (CPU, GPU, or AI Core accelerator). Results stream back to the Compose UI.

### Module Dependency Graph

```
app (module)
 ├── :data          (Task, Model, Config, Allowlist, DataStore)
 ├── :common        (ProjectConfig, Types, Utils)
 ├── :runtime       (LlmModelHelper, AICoreModelHelper)
 ├── :di            (AppModule — Hilt bindings)
 ├── :customtasks   (CustomTask interface + implementations)
 ├── :ui            (Compose screens, navigation, theme)
 └── :worker        (DownloadWorker — WorkManager)
```

> [!NOTE]
> This is a single-module Gradle project. The packages above are logical modules within the `app` module, not separate Gradle modules. All code lives under `com.google.ai.edge.gallery`.

### Key Design Decisions

- **No backend server.** All inference is on-device. The only network calls are model downloads from Hugging Face and allowlist fetches from GitHub.
- **Model allowlists are versioned per app release.** Old app versions receive deprecation warnings in model descriptions rather than losing access entirely.
- **The CustomTask plugin system** allows adding new task types without modifying core navigation or model management code. Tasks register via Hilt `@IntoSet` injection.
- **Skills extend LLM capabilities with JavaScript** executed in a sandboxed WebView. The LLM decides when to invoke a skill via function calling.

## 2. System Requirements

### Build Environment

| Requirement | Version |
|:---|:---|
| Android Studio | Ladybug or later (2024.2+) |
| JDK | 17 |
| Kotlin | Version catalog managed (see `libs.versions.toml`) |
| Gradle | Kotlin DSL, wrapper included |
| Android SDK (compile) | 35 (Android 15) |
| Android SDK (target) | 35 |
| Android SDK (min) | 31 (Android 12) |

### Device Requirements

| Requirement | Minimum | Recommended |
|:---|:---|:---|
| Android version | 12 (API 31) | 14+ (API 34+) |
| RAM | 6 GB | 8-12 GB |
| Free storage | 2 GB | 5+ GB (multiple models) |
| GPU | Optional (CPU fallback) | GPU with OpenCL support |

> [!IMPORTANT]
> Models with `minDeviceMemoryInGb: 12` (Gemma 3n E4B, Gemma 4 E4B) will not load on devices with less than 12 GB of RAM. The app filters the model list based on device memory to prevent out-of-memory crashes.

### Model Size Reference (v1.0.11 Allowlist)

| Model | File Size | Min RAM | Accelerators |
|:---|:---|:---|:---|
| Gemma-4-E2B-it | 2.4 GB | 8 GB | gpu,cpu |
| Gemma-4-E4B-it | 3.4 GB | 12 GB | gpu,cpu |
| Gemma-3n-E2B-it | 3.4 GB | 8 GB | cpu,gpu |
| Gemma-3n-E4B-it | 4.6 GB | 12 GB | cpu,gpu |
| Gemma3-1B-IT | 557 MB | 6 GB | gpu,cpu |
| Qwen2.5-1.5B-Instruct | 1.5 GB | 6 GB | gpu,cpu |
| DeepSeek-R1-Distill-Qwen-1.5B | 1.7 GB | 6 GB | gpu,cpu |
| TinyGarden-270M | 276 MB | 6 GB | cpu |
| MobileActions-270M | 276 MB | 6 GB | cpu |

## 3. Configuration Reference

### Build Configuration

The root build file is `Android/src/build.gradle.kts`. The app module build file is `Android/src/app/build.gradle.kts`. Version management uses a Gradle version catalog at `Android/src/gradle/libs.versions.toml`.

**Gradle properties** (`Android/src/gradle.properties`):

```properties
# Standard Android/Kotlin properties managed by the version catalog.
# No custom properties are required for a basic build.
```

**App Auth redirect scheme** -- the `build.gradle.kts` file configures the OAuth redirect scheme via:

```kotlin
android {
    defaultConfig {
        manifestPlaceholders["appAuthRedirectScheme"] = "your-redirect-scheme"
    }
}
```

This must match the `redirectUri` constant in `ProjectConfig.kt`.

### OAuth Configuration (ProjectConfig.kt)

Located at `Android/src/app/src/main/java/com/google/ai/edge/gallery/common/ProjectConfig.kt`:

| Constant | Purpose | Default |
|:---|:---|:---|
| `clientId` | Hugging Face OAuth client ID | `REPLACE_WITH_YOUR_CLIENT_ID_IN_HUGGINGFACE_APP` |
| `redirectUri` | OAuth redirect URI | `REPLACE_WITH_YOUR_REDIRECT_URI_IN_HUGGINGFACE_APP` |
| `authEndpoint` | HF authorization URL | `https://huggingface.co/oauth/authorize` |
| `tokenEndpoint` | HF token exchange URL | `https://huggingface.co/oauth/token` |

> [!WARNING]
> The default `clientId` and `redirectUri` are placeholders. You must register an application on Hugging Face and replace these values before building a release. Without valid OAuth credentials, gated model downloads will fail.

### Model Runtime Configuration

Each model in the allowlist carries a `defaultConfig` block. These values are user-adjustable at runtime via the config dialog in the app bar.

| Config Key | Type | Description |
|:---|:---|:---|
| `topK` | Int | Number of top-K candidates for sampling (1-128) |
| `topP` | Float | Nucleus sampling probability threshold (0.0-1.0) |
| `temperature` | Float | Sampling temperature. 0.0 = greedy/deterministic |
| `maxTokens` | Int | Maximum output tokens per response |
| `accelerators` | String | Comma-separated accelerator preference: `cpu`, `gpu` |
| `maxContextLength` | Int | Maximum context window (Gemma 4 models: 32000) |
| `visionAccelerator` | String | Accelerator for vision processing (Gemma 4: `gpu`) |

The `ConfigValue` class in `data/ConfigValue.kt` handles typed config values with validation bounds.

### App Bundle Configuration

`Android/src/app/src/main/bundle_config.pb.json` controls Android App Bundle settings for Play Store distribution.

### Protobuf Schemas

Three protobuf schemas define the DataStore persistence format:

| Schema | File | Purpose |
|:---|:---|:---|
| Benchmark | `proto/benchmark.proto` | Stores benchmark run results |
| Settings | `proto/settings.proto` | User preferences (theme, accepted TOS) |
| Skill | `proto/skill.proto` | Installed skill metadata and state |

Serializers for these schemas (`BenchmarkResultsSerializer`, `SettingsSerializer`, `SkillSerializer`, `UserDataSerializer`, `CutoutsSerializer`) live in the root package alongside `GalleryApplication.kt`.

### Firebase Configuration

Firebase Analytics and FCM are integrated via standard `google-services.json` (not checked into the repository). `FcmMessagingService.kt` handles push notification tokens. Analytics events are defined in `Analytics.kt`.

## 4. Deployment & Scaling (Build & Distribution)

### Building from Source

```bash
# Clone the repository
git clone https://github.com/google-ai-edge/gallery.git
cd gallery/Android/src

# Build debug APK
./gradlew assembleDebug

# Build release APK (requires signing config)
./gradlew assembleRelease

# Build Android App Bundle for Play Store
./gradlew bundleRelease
```

> [!TIP]
> The debug build does not require OAuth credentials for basic functionality. Model downloads from public (non-gated) Hugging Face repositories work without authentication.

### Signing Configuration

Release builds require a signing keystore. Configure in `Android/src/app/build.gradle.kts`:

```kotlin
android {
    signingConfigs {
        create("release") {
            storeFile = file("path/to/keystore.jks")
            storePassword = "..."
            keyAlias = "..."
            keyPassword = "..."
        }
    }
    buildTypes {
        release {
            signingConfig = signingConfigs.getByName("release")
        }
    }
}
```

### Distribution Channels

| Channel | Format | Notes |
|:---|:---|:---|
| Google Play | AAB (App Bundle) | Primary distribution. Play Store handles splitting by ABI and density |
| GitHub Releases | APK | Direct install. Debug-signed unless configured otherwise |
| Apple App Store | IPA | iOS version has its own allowlist (`ios_1_0_0.json`) |

### Version Management

App versions map directly to model allowlist files. The naming convention is `{major}_{minor}_{patch}.json` in the `model_allowlists/` directory:

```
model_allowlists/
├── 1_0_4.json      # Oldest supported Android version
├── 1_0_5.json
├── 1_0_6.json
├── 1_0_7.json
├── 1_0_8.json
├── 1_0_9.json
├── 1_0_10.json
├── 1_0_11.json     # Latest Android version
└── ios_1_0_0.json  # iOS version
```

When a new app version is released, a corresponding allowlist file is created. Older versions can be updated to show deprecation warnings (models with descriptions starting with "Your current app version is too old...").

## 5. Operations

### Model Allowlist Lifecycle

The model allowlist is the central control mechanism for which models are available to which app versions. The lifecycle is:

1. **Author:** Create or update a JSON file in `model_allowlists/` matching the target app version.
2. **Publish:** Push to the GitHub repository. The app fetches allowlists from the raw GitHub content URL.
3. **Cache:** The app caches the fetched allowlist locally. `ModelAllowlist.kt` handles fetching and parsing.
4. **Filter:** Models are filtered by device memory (`minDeviceMemoryInGb`) before display.

**Allowlist JSON schema** -- each model entry:

```json
{
  "name": "Display Name",
  "modelId": "huggingface-org/repo-name",
  "modelFile": "filename.litertlm",
  "description": "Markdown-enabled description",
  "sizeInBytes": 2583085056,
  "minDeviceMemoryInGb": 8,
  "commitHash": "full-sha-of-hf-commit",
  "llmSupportImage": true,
  "llmSupportAudio": true,
  "llmSupportThinking": true,
  "defaultConfig": {
    "topK": 64,
    "topP": 0.95,
    "temperature": 1.0,
    "maxTokens": 4000,
    "maxContextLength": 32000,
    "accelerators": "gpu,cpu",
    "visionAccelerator": "gpu"
  },
  "taskTypes": ["llm_chat", "llm_prompt_lab", "llm_agent_chat"],
  "bestForTaskTypes": ["llm_chat"]
}
```

**Field reference:**

| Field | Required | Description |
|:---|:---|:---|
| `name` | Yes | Display name shown in the model picker |
| `modelId` | Yes | Hugging Face repository ID (`org/repo`) |
| `modelFile` | Yes | Exact filename within the HF repo to download |
| `description` | Yes | Markdown description shown on the task detail screen |
| `sizeInBytes` | Yes | File size for download progress calculation |
| `minDeviceMemoryInGb` | No | Minimum device RAM. Models above device capacity are hidden |
| `commitHash` | No | Pinned HF commit hash for reproducible downloads |
| `llmSupportImage` | No | Whether the model accepts image input |
| `llmSupportAudio` | No | Whether the model accepts audio input |
| `llmSupportThinking` | No | Whether the model supports thinking/reasoning mode |
| `defaultConfig` | Yes | Default inference parameters (see Configuration Reference) |
| `taskTypes` | Yes | Array of `BuiltInTaskId` values this model supports |
| `bestForTaskTypes` | No | Array of task IDs where this model is preferred (sorted first) |

### Allowlist Version Evolution

The allowlist system has evolved across app versions:

| Version Range | Runtime | File Extension | Key Changes |
|:---|:---|:---|:---|
| 1.0.4 - 1.0.6 | MediaPipe LLM Inference | `.litertlm` | 3 models (Gemma 3n E2B/E4B, Gemma3-1B) |
| 1.0.7 - 1.0.9 | LiteRT-LM | `.litertlm` | Added Qwen2.5, DeepSeek-R1, extended models |
| 1.0.10 | LiteRT-LM | `.litertlm` | Added TinyGarden-270M, MobileActions-270M |
| 1.0.11 | LiteRT-LM | `.litertlm` | Added Gemma 4 (E2B/E4B) with thinking mode, 32K context |

The legacy root-level `model_allowlist.json` uses the older MediaPipe format with `.task` file extensions and `estimatedPeakMemoryInBytes` instead of `minDeviceMemoryInGb`.

### Model Download Pipeline

`DownloadWorker.kt` uses Android WorkManager for reliable background downloads:

1. User taps a model's download button on the task detail screen.
2. `DownloadRepository.kt` enqueues a `DownloadWorker` with the model's Hugging Face URL constructed from `modelId`, `commitHash`, and `modelFile`.
3. `DownloadWorker` opens an `HttpURLConnection` to the Hugging Face CDN, streaming the file to local storage.
4. If the model is gated, the OAuth token from AppAuth is attached as an `Authorization: Bearer` header.
5. Progress updates are reported via WorkManager's `setProgress()` mechanism.
6. On completion, the model file path is stored in the DataStore, and the model status transitions to "downloaded."

### Task System

Tasks are the primary organizational unit on the home screen. Each task represents a use case (chat, image analysis, benchmarking, etc.) and declares which models it supports.

**Built-in task IDs** (defined in `BuiltInTaskId` object):

| Task ID | Label | Description |
|:---|:---|:---|
| `llm_chat` | Chat | Multi-turn conversational AI |
| `llm_prompt_lab` | Prompt Lab | Single-turn prompt testing |
| `llm_ask_image` | Ask Image | Multimodal image analysis |
| `llm_ask_audio` | Ask Audio | Audio transcription/analysis |
| `llm_agent_chat` | Agent Chat | LLM with skill/function calling |
| `llm_mobile_actions` | Mobile Actions | Device automation via FunctionGemma |
| `llm_tiny_garden` | Tiny Garden | Natural-language game control |
| `mp_scrapbook` | Scrapbook | MediaPipe-based image tasks |

**Legacy tasks:** `llm_chat`, `llm_prompt_lab`, `llm_ask_image`, `llm_ask_audio`, and `llm_agent_chat` are marked as legacy via `isLegacyTasks()`. This distinction affects UI rendering and model initialization paths.

**Thinking mode:** Only `llm_chat`, `llm_ask_image`, and `llm_ask_audio` support thinking mode, gated by `Task.allowThinking()` which checks the task ID. Thinking mode is further gated by the model's `llmSupportThinking` capability flag.

### Monitoring and Analytics

Firebase Analytics tracks user interactions. Events are defined in `Analytics.kt`. Key tracked events include:

- Link clicks (model documentation, source code URLs)
- Task selection and model selection
- Download initiation and completion
- Skill installation and invocation

FCM push notifications are handled by `FcmMessagingService.kt` for remote messaging capabilities.

### Known Operational Issues

| Issue | Symptom | Workaround |
|:---|:---|:---|
| GPU not recognized (GitHub #682) | Pixel 10 does not use GPU acceleration | Force `cpu` in accelerator config |
| Memory after long chat (#82) | App becomes unresponsive after extended conversations | Restart the app; context accumulation exhausts RAM |
| Model download crash (#668) | RuntimeException after download completes | Clear app data and re-download |
| INVALID_ARGUMENT engine error (#585) | `Failed to create engine` on model initialization | Ensure model file matches expected format for app version |
| Only answers "over" (#601) | Model produces single-word responses | Re-download the model; possible corrupt download |
| Custom skill install failure (#619) | Skills fail to install from URL or local import | Verify skill manifest format and JavaScript entry point |

## 6. Performance

### Inference Configuration Tuning

Model performance is primarily controlled by the accelerator selection and sampling parameters.

**Accelerator selection strategy:**

| Accelerator | Best For | Trade-off |
|:---|:---|:---|
| `gpu,cpu` | General-purpose models (Gemma 4, Gemma3-1B) | Fastest inference; higher power consumption |
| `cpu,gpu` | Multimodal models (Gemma 3n) | More stable on devices with limited GPU memory |
| `cpu` | Small specialized models (TinyGarden, MobileActions) | Lowest latency for small models; avoids GPU init overhead |

**Temperature and determinism:**

- `temperature: 0.0` forces greedy decoding. Used by TinyGarden and MobileActions for consistent function-calling output.
- `temperature: 1.0` with `topK: 64` and `topP: 0.95` provides diverse, creative responses. Default for conversational models.
- Qwen2.5 uses `temperature: 0.7`, `topK: 20`, `topP: 0.8` for more focused output.

### Model Default Configurations (Complete Reference)

| Config | Gemma 4 E2B | Gemma 3n E2B | Gemma3-1B | Qwen2.5-1.5B | DeepSeek-R1 | TinyGarden | MobileActions |
|:---|:---|:---|:---|:---|:---|:---|:---|
| topK | 64 | 64 | 64 | 20 | 64 | 64 | 64 |
| topP | 0.95 | 0.95 | 0.95 | 0.8 | 0.95 | 0.95 | 0.95 |
| temperature | 1.0 | 1.0 | 1.0 | 0.7 | 1.0 | 0.0 | 0.0 |
| maxTokens | 4000 | 4096 | 1024 | 4096 | 4096 | 1024 | 1024 |
| accelerators | gpu,cpu | cpu,gpu | gpu,cpu | gpu,cpu | gpu,cpu | cpu | cpu |

### Benchmarking

The built-in benchmark system measures model performance on the specific device. Results are persisted via protobuf DataStore (`benchmark.proto`). The benchmark UI is in `ui/benchmark/`.

Benchmark metrics include:
- Time to first token (prefill latency)
- Tokens per second (decode throughput)
- Total inference time
- Memory usage during inference

### Memory Management

- `LlmModelHelper.cleanUpModelFn()` must be called to release native resources when switching models or tasks.
- `GalleryLifecycleProvider.kt` hooks into Android lifecycle events to manage model state across app backgrounding.
- Models with `estimatedPeakMemoryInBytes` (legacy allowlist format) or `minDeviceMemoryInGb` (current format) provide memory budgeting guidance.

> [!CAUTION]
> Running multiple models simultaneously is not supported. The app initializes one model at a time per task. Attempting to load a model while another is active in a different task will cause memory pressure and potential crashes.

### Context Window Limits

| Model Family | Max Context Length |
|:---|:---|
| Gemma 4 (E2B, E4B) | 32,000 tokens |
| Gemma 3n (E2B, E4B) | 4,096 tokens |
| Gemma3-1B-IT | 1,024 tokens (maxTokens) |
| Qwen2.5-1.5B | 4,096 tokens |
| DeepSeek-R1 | 4,096 tokens |
| TinyGarden/MobileActions | 1,024 tokens |

## 7. Security

### OAuth Token Handling

Hugging Face OAuth 2.0 is implemented via the AppAuth library (`net.openid.appauth`). The flow:

1. User initiates login from the model download screen.
2. `AuthorizationServiceConfiguration` directs to `https://huggingface.co/oauth/authorize`.
3. After consent, the redirect URI returns the authorization code to the app.
4. The app exchanges the code for an access token via `https://huggingface.co/oauth/token`.
5. Tokens are stored in-memory via the AppAuth `AuthState` object.

> [!WARNING]
> The `clientId` and `redirectUri` in `ProjectConfig.kt` are shipped as placeholders. Any fork or custom build must replace these with credentials from a registered Hugging Face OAuth application. The redirect URI scheme in `ProjectConfig.kt` must match the `appAuthRedirectScheme` manifest placeholder in `build.gradle.kts`.

### Skill Sandbox Security

Skills are JavaScript packages that execute inside an Android `WebView`. The security boundary is:

- **Execution context:** Skills run within `GalleryWebView`, which loads HTML/JS from either bundled assets or imported packages.
- **Local URL base:** All local asset URLs are served under `https://appassets.androidplatform.net` via Android's `WebViewAssetLoader`. This avoids `file://` protocol restrictions.
- **Data passing:** The LLM constructs a JSON data payload and an optional secret string. These are passed to the skill's entry point function.
- **Secret management:** Skills that require API keys (e.g., Gemini API key for Restaurant Roulette, Loudly API key for Mood Music) use the `SecretEditorDialog` to store secrets per-skill. Secrets are passed as the second argument to `window['ai_edge_gallery_get_result']`.
- **Return contract:** Skills return JSON with constrained fields: `result` (text), `error` (error text), `image` (base64-encoded), `webview` (URL/iframe for rich rendering).

> [!CAUTION]
> WebView JavaScript execution is not fully sandboxed at the OS level. A malicious skill with a valid manifest could exfiltrate data accessible to the WebView context. Only install skills from trusted sources. The app does not perform code signing verification on imported skills.

### Data Storage Security

- **DataStore (Protobuf):** User settings, benchmark results, and skill metadata are stored in protobuf-serialized DataStore files in the app's private data directory. These are protected by Android's app sandbox.
- **Model files:** Downloaded models are stored in the app's local files directory, accessible only to the app process.
- **No remote data transmission:** All inference runs on-device. No user prompts, model outputs, or conversation history are transmitted to any server.

### Permissions

The `AndroidManifest.xml` declares:

| Permission | Purpose |
|:---|:---|
| `INTERNET` | Model downloads from Hugging Face, allowlist fetching, skill network calls |
| `FOREGROUND_SERVICE` | Background model downloads via WorkManager |
| `CAMERA` | Ask Image: capture photos for multimodal analysis |
| `RECORD_AUDIO` | Voice input and Ask Audio features |

## 8. Developer Internals

### Custom Task System

The `CustomTask` interface (`customtasks/common/CustomTask.kt`) is the extensibility mechanism for adding new task types.

**Interface contract:**

```kotlin
interface CustomTask {
    val task: Task

    fun initializeModelFn(
        context: Context,
        coroutineScope: CoroutineScope,
        model: Model,
        onDone: (error: String) -> Unit,
    )

    fun cleanUpModelFn(
        context: Context,
        coroutineScope: CoroutineScope,
        model: Model,
        onDone: () -> Unit,
    )

    @Composable
    fun MainScreen(data: Any)
}
```

**Implementing a custom task:**

1. Create a class implementing `CustomTask` in a new package under `customtasks/`.
2. Define the `task` property with a `Task` data class specifying: `id`, `label`, `category`, `icon`, `description`, `models`, and `modelNames`.
3. Implement `initializeModelFn()` -- this runs on `Dispatchers.Default`. Call `onDone("")` on success or `onDone("error message")` on failure.
4. Implement `cleanUpModelFn()` -- release native resources, call `onDone()` when complete.
5. Implement `MainScreen()` -- the Composable receives a `CustomTaskData` instance (cast from the `Any` parameter) containing the `ModelManagerViewModel` and layout callbacks.
6. Register via Hilt:

```kotlin
@Module
@InstallIn(SingletonComponent::class)
object YourTaskModule {
    @Provides
    @IntoSet
    fun provideYourTask(): CustomTask = YourCustomTask()
}
```

**CustomTaskData** provides access to the shared model management layer:

```kotlin
data class CustomTaskData(
    val modelManagerViewModel: ModelManagerViewModel,
    val bottomPadding: Dp = 0.dp,
    val setAppBarControlsDisabled: (Boolean) -> Unit = {},
    val setTopBarVisible: (Boolean) -> Unit = {},
    val setCustomNavigateUpCallback: ((() -> Unit)?) -> Unit = {},
)
```

The `setAppBarControlsDisabled` callback lets your task disable the back button, config button, and model selector during operations that should not be interrupted. `setTopBarVisible` hides the entire app bar for full-screen experiences. `setCustomNavigateUpCallback` allows overriding the default back navigation behavior.

A separate `CustomTaskDataForBuiltinTask` class exists for built-in tasks, containing `modelManagerViewModel` and an `onNavUp` callback.

**Existing custom task implementations:**

| Package | Task | Purpose |
|:---|:---|:---|
| `customtasks/examplecustomtask/` | ExampleCustomTask | Reference implementation -- displays model file text content |
| `customtasks/agentchat/` | AgentChat | Agent Chat with skill/function calling (15 source files) |
| `customtasks/mobileactions/` | MobileActions | Device automation via FunctionGemma |
| `customtasks/tinygarden/` | TinyGarden | Natural-language controlled garden game |

### Task Data Class (Full Field Reference)

The `Task` data class in `data/Tasks.kt` defines task metadata:

| Field | Type | Default | Description |
|:---|:---|:---|:---|
| `id` | String | Required | Unique task identifier (use `BuiltInTaskId` constants for built-ins) |
| `label` | String | Required | Display label on the home screen |
| `category` | CategoryInfo | Required | Determines which tab the task appears under |
| `icon` | ImageVector? | null | Compose vector icon for the task tile |
| `iconVectorResourceId` | Int? | null | Android vector resource ID (takes precedence over `icon`) |
| `description` | String | Required | Shown at the top of the task detail screen |
| `shortDescription` | String | "" | Brief description (6 words or fewer) |
| `docUrl` | String | "" | Documentation link shown below description |
| `sourceCodeUrl` | String | "" | Source code link shown below description |
| `models` | MutableList\<Model\> | Required | Models assigned to this task |
| `modelNames` | List\<String\> | [] | Model names to match from the allowlist |
| `handleModelConfigChangesInTask` | Boolean | false | If true, task handles config changes itself instead of auto-reinitializing |
| `experimental` | Boolean | false | Marks the task as experimental |
| `newFeature` | Boolean | false | Shows a "new" badge on the home screen |
| `useThemeColor` | Boolean | false | Uses theme color instead of task tint color |
| `defaultSystemPrompt` | String | "" | System prompt prepended to conversations |
| `agentNameRes` | Int | `R.string.chat_generic_agent_name` | String resource for agent name in chat bubbles |
| `textInputPlaceHolderRes` | Int | `R.string.chat_textinput_placeholder` | String resource for input field placeholder |

### Skills API

Skills extend Agent Chat's capabilities by providing JavaScript-based tools that the LLM can invoke via function calling.

**Skill package structure:**

```
skill-name/
├── SKILL.md              # Manifest (name, description, parameters)
├── scripts/
│   ├── index.html        # WebView entry point
│   └── index.js          # JavaScript logic
└── assets/               # Optional static assets (HTML, images, etc.)
```

**SKILL.md manifest format:**

The `SKILL.md` file is a markdown document that serves as both the skill's documentation and its function-calling schema. The LLM reads this manifest to understand when and how to invoke the skill. It includes the skill name, a description of what the skill does, and the expected JSON input parameters.

**JavaScript entry point contract:**

```javascript
window['ai_edge_gallery_get_result'] = async (data, secret) => {
    // data: JSON string with skill-specific parameters
    // secret: optional API key string from SecretEditorDialog
    try {
        const jsonData = JSON.parse(data);
        // ... perform work ...
        return JSON.stringify({
            result: "Text result shown in chat",
            // Optional fields:
            error: null,                          // Error message
            image: { base64: "..." },             // Base64-encoded image
            webview: {                            // Rich rendering
                url: "webview.html?params=...",    // URL (local or remote)
                iframe: false,                    // Whether to use iframe
                aspectRatio: 1.333                // Width/height ratio (default 4:3)
            }
        });
    } catch (e) {
        return JSON.stringify({ error: e.message });
    }
};
```

**Return value types:**

| Field | Type | Description |
|:---|:---|:---|
| `result` | String | Plain text result shown in the chat bubble |
| `error` | String | Error message; displayed as an error state |
| `image.base64` | String | Base64-encoded image rendered inline |
| `webview.url` | String | URL loaded in an embedded WebView card |
| `webview.iframe` | Boolean | Whether the URL should load in an iframe |
| `webview.aspectRatio` | Float | Width-to-height ratio for the WebView (default: 1.333, i.e., 4:3) |

**Kotlin types for skill results** (in `common/Types.kt`):

```kotlin
@JsonClass(generateAdapter = true)
data class CallJsSkillResult(
    val result: String?,
    val error: String?,
    val image: CallJsSkillResultImage?,
    val webview: CallJsSkillResultWebview?,
)

@JsonClass(generateAdapter = true)
data class CallJsSkillResultImage(val base64: String?)

@JsonClass(generateAdapter = true)
data class CallJsSkillResultWebview(
    val url: String?,
    val iframe: Boolean?,
    val aspectRatio: Float?,
)
```

These are parsed from the JavaScript return value using Moshi (`@JsonClass(generateAdapter = true)`).

**Skill sources:**

| Source | Discovery | Example |
|:---|:---|:---|
| Built-in | Bundled in APK assets (`assets/skills/`) | calculate-hash, qr-code, query-wikipedia |
| Featured | Fetched from a remote featured skills list | mood-music, restaurant-roulette, virtual-piano |
| URL import | User provides a URL to a skill package | Any conforming skill package |
| Local import | User selects a local directory | Development/testing workflow |

**Built-in skills reference:**

| Skill | Function | Requires Secret |
|:---|:---|:---|
| calculate-hash | SHA-1 hash of input text | No |
| interactive-map | Renders a location on a map | No |
| kitchen-adventure | Text-based cooking adventure game | No |
| mood-tracker | Tracks and visualizes mood entries | No |
| qr-code | Generates QR codes from text | No |
| query-wikipedia | Searches Wikipedia articles | No |
| send-email | Composes email via intent | No |
| text-spinner | Visual text transformation animations | No |

**Featured skills reference:**

| Skill | Function | Requires Secret |
|:---|:---|:---|
| mood-music | Generates music based on mood (Loudly API) | Yes (Loudly API key via loudly.com/developers/apps) |
| restaurant-roulette | Spin wheel of nearby restaurants (Gemini API) | Yes (Gemini API key) |
| virtual-piano | Interactive piano keyboard | No |

### Agent Action System

The Agent Chat task uses an action dispatch system to communicate between the LLM inference layer and the UI:

```kotlin
open class AgentAction(val name: AgentActionName)

enum class AgentActionName {
    CALL_JS_SKILL,
    SKILL_PROGRESS,
    ASK_INFO,
}

// Invoke a JavaScript skill
class CallJsAgentAction(
    val url: String,          // Skill's index.html URL
    val data: String,         // JSON data payload
    val secret: String = "",  // API key (from SecretEditorDialog)
    val result: CompletableDeferred<String> = CompletableDeferred(),
)

// Prompt the user for input
class AskInfoAgentAction(
    val dialogTitle: String,
    val fieldLabel: String,
    val result: CompletableDeferred<String> = CompletableDeferred(),
)

// Update skill progress indicator
class SkillProgressAgentAction(
    val label: String,
    val inProgress: Boolean,
    val addItemTitle: String = "",
    val addItemDescription: String = "",
    val customData: Any? = null,
)
```

Actions use Kotlin `CompletableDeferred` for asynchronous result delivery. The UI layer observes the action, renders the appropriate UI (WebView, dialog, or progress indicator), and completes the deferred when the result is available.

The `SkillInfo` data class ties skills to their metadata:

```kotlin
data class SkillInfo(
    val skillMd: String,       // SKILL.md content (function schema for LLM)
    val skillUrl: String?,     // URL to skill's index.html
    val tryoutChip: SkillTryOutChip?,  // Optional quick-try chip for UI
)
```

### LLM Runtime Layer

**LlmModelHelper.kt** manages the complete model lifecycle:

1. **Initialization:** Loads the model file into the LiteRT-LM or MediaPipe runtime. Configures accelerators, context length, and sampling parameters.
2. **Inference:** Accepts prompts (text, image, audio) and streams token output via callbacks.
3. **Cleanup:** Releases native memory and GPU resources.

**AICoreModelHelper.kt** provides an alternative initialization path for devices with Google AI Core support. AI Core is Google's on-device AI runtime that provides optimized inference for supported chipsets. The helper checks for AI Core availability and falls back to standard LiteRT-LM when unavailable.

**ModelHelperExt.kt** contains extension functions that bridge between the model helper and the UI layer, handling response processing and error formatting. The `processLlmResponse()` utility in `Utils.kt` normalizes escaped newlines (`\\n` to `\n`) in model output. The `cleanUpMediapipeTaskErrorMessage()` function strips internal stack trace details (everything after `=== Source Location Trace`) from MediaPipe error messages before showing them to users.

### Data Layer

**DataStoreRepository.kt** wraps Android DataStore with protobuf serialization:

- `BenchmarkResultsSerializer` -- benchmark run data
- `SettingsSerializer` -- user preferences
- `SkillSerializer` -- installed skill state
- `UserDataSerializer` -- user-specific data
- `CutoutsSerializer` -- display cutout information

**DownloadRepository.kt** manages download state and coordinates with `DownloadWorker`:

- Tracks download progress per model
- Handles pause/resume (via WorkManager constraints)
- Manages file integrity after download completion

**Config.kt and ConfigValue.kt** handle runtime configuration:

- `Config` aggregates all config values for a model session
- `ConfigValue` is a sealed type hierarchy supporting Int, Float, String, and Boolean config values with validation ranges

### Networking

The app uses `java.net.HttpURLConnection` directly (no OkHttp or Retrofit). The `getJsonResponse<T>()` utility in `Utils.kt` provides generic JSON fetching:

```kotlin
inline fun <reified T> getJsonResponse(url: String): JsonObjAndTextContent<T>?
```

JSON parsing uses Gson for general-purpose deserialization (`parseJson<T>()`) and Moshi for skill results (via generated adapters with `@JsonClass(generateAdapter = true)`).

### Audio Processing

`Utils.kt` includes audio utilities for the Ask Audio feature:

- `convertWavToMonoWithMaxSeconds()` -- converts stereo WAV files to mono, resamples if needed, and truncates to a maximum duration (default 30 seconds)
- Supports 8-bit to 16-bit upconversion
- Reads WAV headers to extract channel count, sample rate, and bit depth
- The constant `SAMPLE_RATE` (defined in `data/Consts.kt`) specifies the target sample rate

### Dependency Injection (Hilt)

`di/AppModule.kt` is the single Hilt module providing application-scoped dependencies:

- `Set<CustomTask>` -- all registered custom tasks (via `@IntoSet`)
- DataStore instances for each protobuf schema
- Repository singletons

The `GalleryApplication.kt` class is annotated with `@HiltAndroidApp` as the Hilt entry point.

### UI Architecture

The UI follows Jetpack Compose with Material 3 theming:

- **Navigation:** `GalleryNavGraph.kt` defines the navigation graph. Single-activity architecture with Compose destinations.
- **Home screen:** `ui/home/` -- categories as tabs, tasks as cards within each tab. Tab bar hidden when only one category exists.
- **Chat UI:** `ui/common/chat/` -- 36 files implementing the chat interface. Supports multiple message body types: text, image, audio, benchmark results, WebView embeds, loading indicators, and classified results.
- **Model management:** `ui/modelmanager/` -- `ModelManagerViewModel` is the shared ViewModel for model state across all screens. Handles initialization, config changes, and model switching.
- **Theme:** `ui/theme/` -- Material 3 dynamic color support with dark/light mode.
- **Voice input:** `ui/common/textandvoiceinput/` -- combined text field and voice recording input.
- **Terms of Service:** `ui/common/tos/` -- first-run TOS acceptance flow.
- **Shared components:** `ui/common/` includes reusable Composables such as `Accordions`, `AudioAnimation` (AGSL shader-based waveform), `GalleryWebView`, skill management bottom sheets, and model configuration dialogs.

### Source File Reference

**Root package** (`com.google.ai.edge.gallery`):

| File | Purpose |
|:---|:---|
| `GalleryApplication.kt` | Application class, `@HiltAndroidApp` entry point |
| `GalleryApp.kt` | Root Composable |
| `MainActivity.kt` | Single Activity |
| `GalleryLifecycleProvider.kt` | Lifecycle observer for model management |
| `Analytics.kt` | Firebase Analytics event definitions |
| `FcmMessagingService.kt` | Firebase Cloud Messaging handler |
| `BenchmarkResultsSerializer.kt` | DataStore serializer for benchmark data |
| `SettingsSerializer.kt` | DataStore serializer for settings |
| `SkillSerializer.kt` | DataStore serializer for skills |
| `UserDataSerializer.kt` | DataStore serializer for user data |
| `CutoutsSerializer.kt` | DataStore serializer for display cutouts |

**data package:**

| File | Purpose |
|:---|:---|
| `Tasks.kt` | `Task` data class, `BuiltInTaskId` constants, `isLegacyTasks()` |
| `Model.kt` | `Model` data class |
| `ModelAllowlist.kt` | Remote allowlist fetching and parsing |
| `SkillAllowlist.kt` | Skill discovery and allowlist |
| `Categories.kt` | `CategoryInfo` definitions for home screen tabs |
| `Config.kt` | Runtime config aggregation |
| `ConfigValue.kt` | Typed config value with validation |
| `Consts.kt` | App-wide constants (`SAMPLE_RATE`, etc.) |
| `DataStoreRepository.kt` | DataStore access layer |
| `DownloadRepository.kt` | Download state management |
| `Types.kt` | Data-layer type definitions |

**common package:**

| File | Purpose |
|:---|:---|
| `ProjectConfig.kt` | HF OAuth constants (`clientId`, `redirectUri`, endpoints) |
| `Types.kt` | Core types: `AgentAction`, `SkillInfo`, `CallJsSkillResult`, `AudioClip`, etc. |
| `Utils.kt` | Image/audio processing, JSON parsing, HTTP utilities, `LOCAL_URL_BASE` |

**runtime package:**

| File | Purpose |
|:---|:---|
| `LlmModelHelper.kt` | Model lifecycle management (init, infer, cleanup) |
| `ModelHelperExt.kt` | Extension functions for model helpers |
| `aicore/AICoreModelHelper.kt` | Google AI Core integration |

**worker package:**

| File | Purpose |
|:---|:---|
| `DownloadWorker.kt` | WorkManager worker for background model downloads |

> [!NOTE]
> The `LOCAL_URL_BASE` constant in `Utils.kt` is set to `https://appassets.androidplatform.net`. This is used by `WebViewAssetLoader` to serve local skill assets over a virtual HTTPS origin, avoiding security restrictions on `file://` URLs in WebView.

## 9. Integration Points

### Hugging Face Integration

The app integrates with Hugging Face at two levels:

1. **Model hosting:** All model files are hosted on Hugging Face repositories. The `modelId` field in the allowlist maps to a HF repo (e.g., `google/gemma-3n-E2B-it-litert-lm`). The `commitHash` pins downloads to a specific commit for reproducibility.
2. **Authentication:** OAuth 2.0 via AppAuth for accessing gated models. The flow uses the authorization code grant.

**Download URL construction:**

```
https://huggingface.co/{modelId}/resolve/{commitHash}/{modelFile}
```

For example:
```
https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/7fa1d78.../gemma-4-E2B-it.litertlm
```

### Google AI Core

`AICoreModelHelper.kt` integrates with Google's on-device AI Core runtime. This is an optional accelerator path available on supported devices (primarily Pixel phones with Tensor chipsets). When available, AI Core provides hardware-optimized inference that can outperform the standard LiteRT-LM path.

> [!TIP]
> AI Core availability is device-dependent and not guaranteed. The app gracefully falls back to LiteRT-LM inference when AI Core is unavailable. No configuration is needed -- the runtime detects AI Core support automatically.

### Firebase

- **Firebase Analytics:** Event tracking for user interactions. No PII is collected.
- **Firebase Cloud Messaging (FCM):** `FcmMessagingService` handles token management and incoming messages. Used for optional push notification capabilities.

### Gemini API (via Skills)

The Restaurant Roulette featured skill calls the Gemini API (`generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent`) to generate restaurant recommendations. This is not a core app integration -- it is a skill-level dependency that requires the user to provide their own Gemini API key via the Secret Editor. The skill uses structured JSON output mode (`responseMimeType: 'application/json'`) with a response schema to ensure reliable parsing.

### Skill WebView Bridge

Skills communicate with the app through a JavaScript-to-Kotlin bridge:

1. **App to Skill:** The app calls `window['ai_edge_gallery_get_result'](data, secret)` via `WebView.evaluateJavascript()`.
2. **Skill to App:** The skill returns a JSON string that the app parses into `CallJsSkillResult` using Moshi.
3. **Asset loading:** Local skill assets (HTML, JS, images) are served via `WebViewAssetLoader` under the `appassets.androidplatform.net` domain.
4. **Network access:** Skills with `iframe: true` or external URLs can make network requests subject to Android's standard WebView network policies.

### Adding a New Model to the Allowlist

To add a new model:

1. Upload the quantized model file to a Hugging Face repository.
2. Note the commit hash, file name, and file size in bytes.
3. Determine the model's capabilities (`llmSupportImage`, `llmSupportAudio`, `llmSupportThinking`).
4. Test on target devices to establish `minDeviceMemoryInGb` and optimal `accelerators` ordering.
5. Add the model entry to the appropriate version's allowlist JSON in `model_allowlists/`.
6. Set `taskTypes` to the task IDs where this model should appear.
7. Optionally set `bestForTaskTypes` to prioritize this model for specific tasks.
8. Push the updated allowlist to the GitHub repository.

### Adding a New Skill

To create and distribute a new skill:

1. Create a directory with `SKILL.md`, `scripts/index.html`, and `scripts/index.js`.
2. Write the `SKILL.md` manifest describing the skill's purpose and expected input parameters.
3. Implement the `window['ai_edge_gallery_get_result']` function in `index.js`.
4. Test locally by importing the skill via the local import flow in the app.
5. To distribute as a built-in skill: add the directory under `Android/src/app/src/main/assets/skills/` and rebuild.
6. To distribute as a featured skill: add the directory under `skills/featured/` and update the featured skills index.

### Adding a New Custom Task

To add a new custom task to the app:

1. Create a new package under `customtasks/` (e.g., `customtasks/yourtask/`).
2. Implement the `CustomTask` interface with your task metadata, model lifecycle, and UI.
3. Create a Hilt module in the same package with `@Provides @IntoSet` binding.
4. Define which models your task uses via `modelNames` (matched from the allowlist) or manually populate `models`.
5. Choose or create a `CategoryInfo` for home screen placement.
6. Build and test -- Hilt auto-discovers your task and adds it to the home screen.

---

_Generated May 2026 from google-ai-edge/gallery at 2ccb603. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._