signs-RFC-001 — Sign-language avatar + bilingual translation across the Koder Stack

1. Goal

Make every Koder UI capable of presenting its content in Brazilian Sign Language (Libras) through a friendly, animated 3D avatar (Hand) that signs in real time, activated by a single Settings toggle and persisted across the user's apps via Koder ID.

The capability must:

• Accept Brazilian Portuguese (pt-BR) and American English (en-US) as source languages.
• Output Libras (only) — ASL is reserved for a future version.
• Work in mobile, desktop, TV, and web Koder UIs — i.e., a cross-cutting feature per policies/sdk-first.kmd.
• Be fully proprietary and self-hosted — no dependency on VLibras or any third-party service (Path C).
• Be architected for hyperscale from day one — no throw-away v0; the architecture that ships first is the architecture that scales to v∞.

2. Why now

• Accessibility-first products attract a meaningful user base (10M+ deaf/HoH Brazilians, ~5% of population per IBGE) and align with Koder's positioning as the platform that "speaks Brazil while shipping globally".
• Public-sector buyers (vital to Edictus, ERP track) require Libras compliance — a single SDK widget makes every Koder app eligible at zero per-product effort.
• Koder products are en-US first in UI source / marketing per policies/language.kmd, but operate primarily in Brazil where deaf users sign Libras. Hand serves the same UI to two source-language audiences in one architecture.
• Koru just shipped (2026-04-29) without using "Hand" as its brand precisely so this RFC could claim it.

3. Requirements

Functional

• R1. Single Settings toggle in any Koder UI activates Hand; persists across apps via Koder ID.
• R2. Translate visible UI text and on-demand passages to Libras in real time.
• R3. Render an animated 3D avatar overlay (draggable, resizable, dismissible).
• R4. Work on mobile (Flutter AndroidiOS), desktop (Flutter LinuxmacOS/Windows), TV (TizenOS+WebOS), and web (Flutter Web + landing-page widget).
• R5. Optional speech input (hold-to-speak) — defers to services/ai/voice for ASR.
• R6. Accept both pt-BR and en-US as source languages, with explicit lang tag on the API call (auto-detection as fallback).

Non-functional

• NF1. Latency: gloss-sequence start within ≤300ms after request; first gesture rendered within ≤500ms.
• NF2. Bundle size: SDK widget ≤8MB on mobile (gzipped); avatar assets streamable from CDN if larger.
• NF3. Offline graceful degradation: SDK has a small cached glossary (high-frequency phrases) so common UI labels keep signing without network.
• NF4. Privacy: translation requests are stateless and not logged with content; only opt-in usage telemetry per specs/errors/reporting.kmd.
• NF5. Compliance: Lei nº 10.4362002 + Decreto 5.6262005 cited in landing/legal copy; WCAG 2.2 AA where applicable.
• NF6. Self-hosted only. Like every other Koder backend (Flow, AI Voice, Foundation), Koder AI Signs runs entirely on Koder infrastructure. Calls to any external endpoint are forbidden — sovereignty (no third-party can take the service down) + privacy (translation requests never leave Koder infra).
• NF7. Hyperscale-first per policies/hyperscale-first.kmd — every architectural decision optimizes the long-term system over short-term ship speed; no throw-away v0.

4. Architecture

4.1 The pipeline (4 stages, swappable, instrumented)

[ source text + lang tag ]
    │
    ▼
┌────────────────────────────────────┐
│ §1 Normalization                   │  bilingual: pt-BR | en-US
│   - spell correction               │  rules per language
│   - abbreviation expansion         │  ("vc"→"você", "u"→"you")
│   - sentence segmentation          │
└──────────────┬─────────────────────┘
               │
               ▼
┌────────────────────────────────────┐
│ §2 Semantic parsing                │  ML — multilingual foundation model
│   - intent / entity / sentiment    │  (Llama-3.x · Gemma-3 · Qwen)
│   - cross-lingual representation   │  one model serves both src langs
└──────────────┬─────────────────────┘
               │
               ▼
┌────────────────────────────────────┐
│ §3 Glosa generation (HYBRID)       │  ┌──────────────────────────────┐
│   ┌─────────────────────────────┐  │  │ Confidence router            │
│   │ Rule-based dictionary       │◀─┼──│ · high conf → use rules      │
│   │ (curated, deterministic)    │  │  │ · low conf  → use ML         │
│   └─────────────────────────────┘  │  │ · ML hallucinates → fallback │
│   ┌─────────────────────────────┐  │  │   to fingerspelling          │
│   │ ML translator               │  │  └──────────────────────────────┘
│   │ (foundation + LoRA fine-tune)│  │
│   └─────────────────────────────┘  │
└──────────────┬─────────────────────┘
               │  glosa sequence (intermediate language, src-agnostic)
               ▼
┌────────────────────────────────────┐
│ §4 Animation generation            │  glosa → 3D timeline
│   - sign animation lookup          │  (Hand renders client-side)
│   - inflection / classifier rules  │  Libras spatial grammar
│   - timing / rhythm                │
└──────────────┬─────────────────────┘
               │
               ▼
[ Hand avatar signs ]

Why hybrid neural-symbolic, not pure ML or pure rules:

Production-grade NLP in specialized domains (medical, legal, sign-language) all use hybrid. Single-paradigm systems either don't scale (rules-only) or hallucinate dangerously (ML-only).

4.2 Foundation model strategy (Q3 — RESOLVED 2026-04-29)

We do not train from scratch — we fine-tune a multilingual foundation model that already understands pt-BR and en-US.

• Why not from scratch: training a 7B-param model on PT+EN+Libras from zero would cost ~\(500k-2M and 6+ months. Fine-tuning gives us 95% of the result for ~\)5-20k and weeks.

Decision: Gemma 4 + LoRA via Unsloth

Selection rationale:

Why Gemma 4:

• Apache 2.0 in 2026 (changed from Gemma-license) — cleanest legal posture, no clauses to track, no MAU thresholds.
• 100+ training languages, 30+ first-class — both pt-BR and en-US natively strong.
• E4B size (~4B effective params) — sweet spot for a specialized translation task; fits commodity GPU; inference latency aligned with NF1 (<300ms).
• Active Unsloth pipeline — official LoRA documentation; ~$10-16 + 8-12h on a single H100 produces a 50-200MB LoRA adapter that merges with the base model.
• Cross-lingual transfer — single fine-tune handles both pt-BR and en-US inputs because the base model understands both natively. A pt-BR training example partially teaches the en-US equivalent — corpus efficiency multiplier.

Architectural commitment vs. specific model: the architectural decision (foundation model + LoRA) is stable; the specific base (Gemma 4 today) is swappable. Re-evaluate at each major release of Gemma / Llama / Qwen — fine-tune effort to migrate is days, not months.

v1 spike before locking: train a small LoRA on a 5k-pair pt-BR↔glosa subset, evaluate human-rated quality vs. baseline. If Gemma 4 underperforms Llama 4 by >15% on the eval set, switch to Llama 4 (license tradeoff acceptable given we're <700M MAU by orders of magnitude).

4.3 Boundaries — modules to create

services/ai/signs/                       ← backend, owns the 4-stage pipeline
  ├── backend/         (Go HTTP/gRPC server: /v1/translate, /v1/speech)
  ├── stages/
  │   ├── normalize/   (Go — language-specific rules, deterministic)
  │   ├── semantic/    (Python — calls foundation model via gRPC)
  │   ├── glosa/       (Python — hybrid: rule lookup + ML inference)
  │   └── animate/     (Go — glosa → keyframes; run server-side OR client)
  ├── corpus/          (versioned PT+EN+glosa parallel corpus)
  ├── models/          (model registry: weights + version + eval metrics)
  ├── eval/            (held-out test set + regression suite)
  ├── annotation/      (web tool for Libras consultants to label/dispute)
  ├── docker-compose.yaml
  └── README.md

engines/sdk/koder_hand_kit/                ← Flutter SDK package
  ├── lib/
  │   ├── koder_hand_kit.dart              (public API)
  │   └── src/
  │       ├── hand_avatar.dart             (3D avatar widget — picks backend by platform)
  │       ├── hand_overlay.dart            (draggable floating window)
  │       ├── hand_button.dart             ("Ver em Libras" / "Watch in Libras")
  │       ├── hand_gate.dart               (auto-listen to UI text via accessibility tree)
  │       ├── gloss_player.dart            (consumes gloss sequence → animation timeline)
  │       ├── offline_dictionary.dart      (cached high-frequency entries for NF3)
  │       ├── face_overlay.dart            (Rive-based facial NMM blendshapes)
  │       └── backends/
  │           ├── filament_backend.dart    (mobile + desktop via platform channel)
  │           └── modelviewer_backend.dart (web + TV via <model-viewer>)
  ├── android/                             (Kotlin glue: Filament SurfaceView + glTF loader)
  ├── ios/                                 (Swift glue: Filament + Metal)
  ├── linux/macos/windows/                 (C++ glue: Filament desktop)
  ├── assets/                              (Hand glTF master + SMPL-X rig + Rive face)
  └── docs/

products/horizontal/hand/                  ← brand surface
  ├── README.md, koder.toml, icon.svg
  ├── landing/index.html                   (https://hand.koder.dev)
  └── (no app/ — Hand is embedded inside other apps via koder_hand_kit)

4.4 Separation of translation and rendering

Hard architectural boundary: the translator produces glosa (intermediate language), the renderer (Hand SDK) consumes glosa.

Why: gives us four levers we can pull independently:

This is the same pattern that lets Whisper improvements not affect Spotify clients.

4.5 Naming

• services/ai/signs — descriptive, follows AI service convention (voice, recsys, runtime, bot). Technical reference name.
• engines/sdk/koder_hand_kit — SDK package name; mirrors koder_kit family.
• Hand — public brand of the avatar (visible label "Avatar Hand" / "Hand — sign-language avatar"). Brand score 83 (Great), confirmed in earlier scoping. Visual identity is hands signing — semantic exactly matches the function.
• Settings labels:
  • en-US: "Sign-language avatar (Hand)"
  • pt-BR: "Avatar de Libras (Hand)"

Two-name pattern (engine + brand) follows the established engines/kodec (technical) ↔ Play/Tune (consumer-facing) pattern.

4.6 Scope ladder

The architecture exists in full from v0. What changes between versions is the maturity and coverage of each stage, not the architecture itself. No throw-aways.

Critically: rule-based dictionary is not a v0-only crutch. It remains in production forever as the high-confidence layer. Same applies to fingerspelling — never deprecated, always available for proper nouns and unknown terms.

4.7 Avatar tech stack (Q2 — RESOLVED 2026-04-29)

Hand is rendered by a per-platform best-of-class engine, all consuming the same glTF asset with SMPL-X rig. No single engine is forced to be sub-optimal everywhere; one asset workflow serves all surfaces.

Animator workflow                    Rendering per platform
─────────────────                    ──────────────────────
Blender / Maya
    │
    ├─ skeletal animation            ┌──────────────────────────┐
    │   on SMPL-X rig                │ Mobile (Android, iOS)    │
    │                                │   → Filament (Google)    │
    │                                │      via platform channel│
    └─ export glTF + GLB             │      ~6MB native lib     │
        (single source of truth)     ├──────────────────────────┤
              │                      │ Desktop (Lin/Mac/Win)    │
              │                      │   → Filament native      │
              │                      ├──────────────────────────┤
              ├─→ runtime ─→         │ Web (Flutter Web)        │
              │                      │   → <model-viewer> (Google)│
              │                      │      built on three.js   │
              │                      ├──────────────────────────┤
              │                      │ TV (TizenOS, WebOS)      │
              │                      │   → <model-viewer>       │
              │                      │      (TVs are web-first) │
              │                      └──────────────────────────┘
              │
              └─→ Face overlay
                  → Rive (excellent at facial blendshapes,
                          essential for Libras non-manual markers
                          which are part of the grammar, not
                          decoration)

Why this stack and not a single engine:

Filament is the production-mature backend (Sceneform, Google Earth, etc.). flutter_scene will likely match it once it leaves preview — at that point we re-evaluate and possibly migrate mobile/desktop. The glTF asset format and SMPL-X rig are forward-compatible with that migration; only the rendering glue changes.

Why SMPL-X specifically:

SMPL-X is the de facto rig for sign-language research — used by Hand Talk's pipeline, SignAvatars (ECCV 2024), and most academic papers in the space. Adopting SMPL-X aligns with the format ecosystem (BlenderMaya export, glTF compatibility, MediaPipe pose retargeting). Note: the SignAvatars dataset itself is non-commercial and excluded from our training corpus (see §4.8 Corpus); we use SMPL-X as a *ormatrig*independent of any particular dataset.

Why Rive for face overlay (and only the face):

Libras non-manual markers — eyebrow position, mouth shape, eye gaze direction, head tilt — are grammatical, not decorative. Filament can do facial blendshapes but Rive's facial-animation tooling and editor are markedly better and the runtime is tiny (~700KB). The compromise: face is rendered as a Rive surface composited over the Filament 3D body, sharing timing and parameters via the Hand SDK. Single avatar visually, two engines internally.

Future migration target: when flutter_scene ships stable, re-evaluate replacing Filament with it. Same glTF assets work directly; only the platform-channel glue is replaced.

4.8 Corpus strategy (Q4 — RESOLVED 2026-04-29)

Brazilian Libras corpus landscape is rich (UFPB, UFSC, UFPel, UFPE all have datasets). Our strategy builds a primary corpus from VLibrasBD (CC BY-SA 4.0, commercial-permitted) and supplements it with bootstrap MT for en-US plus paid annotation for v2 quality refinement.

Primary corpus — VLibrasBD (Mendeley Data, CC BY-SA 4.0):

• 127,349 aligned pt-BR ↔ Libras-glosa sentence pairs, built by 10 Libras interpreters
• ~72k general-purpose sentences + ~55k from Brazilian federal government content (gov.br services)
• DOI: 10.17632/ryj88ckjww — open-download from Mendeley Data
• License: CC BY-SA 4.0 — commercial use permitted; share-alike clause applies to derivative works (interpreted conservatively, our trained models inherit CC BY-SA)

Acceptance of share-alike: open-weight models align with Koder's self-hosted ethos. Our value proposition is the integrated stack (Hand SDK + apps + Koder ID + UX), not the model weights themselves. Trade is acceptable vs. ~$50-100k cost of annotating 127k pairs from scratch.

Supplementary corpora (license verification pending before training):

Bilingual strategy (en-US bootstrap):

VLibrasBD is pt-BR only. To get en-US → glosa coverage:

1. MT bootstrap — pass the PT side of VLibrasBD through DeepL/Google Translate to produce ~127k en-US ↔ glosa pairs. Cost: ~$200 in DeepL API.
2. Cross-lingual transfer — Gemma 4's existing multilingual capacity means a pt-BR → glosa fine-tune partially teaches en-US → glosa for free.
3. Paid annotation for v2 refinement — commission ~10k high-quality en-US → glosa pairs from professional Libras interpreters fluent in English. Estimate: ~\(5k (10k pairs ÷ 45 sentences/h × R\)100/h ≈ R$22k).

Animation bank strategy:

Since SignAvatars is non-commercial, the avatar's signing animations are produced through:

1. V-Librasil pose extraction — process the dataset's videos with MediaPipe Holistic + retarget to our SMPL-X rig (we extract poses, we don't redistribute the videos). License-permitting.
2. Own motion-capture session — commission a mocap studio session with deaf native signer for the initial 1,000 signs (~R$30-60k including studio time, signer fee, post-production).
3. Hand-keyframed animation — supplementary for rare signs, glossary expansion, edge cases.

Active learning loop (production):

• Model flags high-uncertainty inputs in production
• Flagged inputs queued in annotation pipeline
• Libras consultants label / dispute / approve
• Retrained periodically (monthly initially → weekly at scale)
• Output: continuous quality improvement; corpus grows organically without large upfront annotation push

Cost estimate — corpus year 1: ~$50-80k.

4.9 Data infrastructure (built from day 1, not day 1000)

These exist starting v0 even if at small scale, so v1's ML training plugs in cleanly:

• Corpus — versioned object storage with metadata (source, dialect, quality score, lang tag). pt-BR and en-US examples co-located, both targeting same glosa.
• Annotation pipeline — web tool (services/ai/signs/annotation/) for Libras consultants (deaf native signers we contract) to label, verify, dispute. Permissive of regional variants; tracks reviewer identity for academic citation.
• Active learning loop — model flags high-uncertainty inputs in production; humans label; retrained periodically (initially monthly, later weekly).
• Eval suite — held-out test set + regression suite + production feedback. BLEU-glosa as primary metric; human review (deaf native signer panel) as quality gate.
• Model registry — every trained model versioned with corpus snapshot + hyperparameters + eval results. Rollback automatic if production metrics regress.

4.10 Serving infrastructure — gRPC (Q8 — RESOLVED 2026-04-29)

Decision: follow existing Koder gRPC convention.

gRPC is already the established backend transport in the Koder Stack. Audit found:

• `servicesfoundationidengineprotokoderid{identity,keys,session,admin}v1