Search (Web Search/Scrape): foundations

accepted

Status

accepted — ratificada 2026-05-09 (mesmo dia da abertura) como parte da onda piloto de bootstrap servicesai. Implementação iniciada em `servicesaisearch; tickets em servicesaisearchbacklogpending/{001..005}`.

Summary

Web searchscraping pra agents — análogo Perplexity APITavilyFirecrawlExa.

Motivation

Foi referenciado no commit recente (/k-evolve koder-ai vs firecrawl+wispr+testsprite em 2026-05-09)! Agents sem web search = dump completo do mundo na sessão. Foundation crítica pra agentes úteis.

Scope

In

  • Web search (proxy + fallback engine)
  • Scrape (Firecrawl-like)
  • Citations
  • Dedup

Out (yet)

  • Deep research multi-step (escopo agents)

Initial design

Surfaces

  • backend/ — Go API + worker scrape
  • app/ — não aplicável v1

Key APIs

  • POST /v1/search/web — search
  • POST /v1/search/scrape — fetch + clean
  • POST /v1/search/news — news vertical

Dependencies

  • services/ai/gateway — LLM rerank
  • services/ai/cache — search results
  • infra/data/kdb-blob — scrape cache

Relation to existing sectors

  • Pré-requisito de agents úteis em produção
  • Consome cache pra reduzir custo

Self-hosted-first analysis (5 gates)

  • G1 Feature parity: zero
  • G2 Performance: N/A
  • G3 Stability: N/A
  • G4 Capability: SearxNG self-hosted + Firecrawl FOSS viáveis
  • G5 Critical-path readiness: destrava agents de produção

Open questions

  • Q1: Default engine — SearxNG self-hosted ou proxy comercial?
  • Q2: Robots.txt compliance enforce no service?

Next steps

  1. Ratificar esta RFC (1 round de comments).
  2. Criar sector dir services/ai/search/ com koder.toml, README.md, skeleton.
  3. Abrir tickets de implementação em services/ai/search/backlog/pending/.
  4. Registrar em meta/docs/stack/registries/self-hosted-pairs.md se substituir externo.