API Server
REST API serving the web UI and CLI. Handles search requests, repository management, and job coordination.
Architecture Overview
Code Search is built with a modular architecture designed for scalability and extensibility.
System Overview
Section titled “System Overview”┌─────────────────────────────────────────────────────────────────┐│ Web UI / CLI │└─────────────────────────────────────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────────┐│ API Server ││ ││ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ ││ │ Search │ │ Repos │ │ Jobs │ │ Connections │ ││ │ Handler │ │ Handler │ │ Handler │ │ Handler │ ││ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │└─────────────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼┌──────────────┐ ┌──────────────┐ ┌──────────────┐│ Zoekt │ │PostgreSQL/ │ │ Redis ││ (Search Index)│ │MySQL (Data) │ │ (Queue) │└──────────────┘ └──────────────┘ └──────────────┘ ▲ ▲ │ │┌─────────────────────────────────────────────────────────────────┐│ Indexer Service ││ ││ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ ││ │Scheduler │ │ Worker │ │ Cloner │ │ Index Builder │ ││ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │└─────────────────────────────────────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────────┐│ Code Hosts ││ ││ ┌──────────┐ ┌──────────────────┐ ┌────────────────────────┐ ││ │ GitHub │ │ GitHub │ │ GitLab │ ││ │ │ │ Enterprise │ │ │ ││ └──────────┘ └──────────────────┘ └────────────────────────┘ │└─────────────────────────────────────────────────────────────────┘Components
Section titled “Components”Indexer Service
Background service that discovers, clones, and indexes repositories from configured code hosts.
Zoekt
Fast trigram-based code search engine. Provides millisecond search across large codebases.
PostgreSQL / MySQL
Stores metadata about connections, repositories, jobs, and configuration.
Redis
Job queue and distributed locking for coordinating indexer workers.
Web UI
Next.js frontend providing search interface and management dashboard.
Design Principles
Section titled “Design Principles”Modular Architecture
Section titled “Modular Architecture”Each component has a single responsibility:
- API: Request handling and response formatting
- Indexer: Repository management and indexing
- Zoekt: Search execution
Horizontal Scalability
Section titled “Horizontal Scalability”Components can be scaled independently:
- Multiple API servers behind a load balancer
- Multiple indexer workers processing jobs
- Zoekt shards for large-scale deployments
Fault Tolerance
Section titled “Fault Tolerance”- Job queue persists across restarts
- Failed jobs are automatically retried
- Health checks for all components
Extensibility
Section titled “Extensibility”- Pluggable code host providers
- Configurable indexing strategies
- Custom search filters
Scaling Architecture
Section titled “Scaling Architecture”Single Instance (Default)
Section titled “Single Instance (Default)”One indexer with Zoekt sidecar, handling search, file browsing, and replace jobs.
Shared Storage (RWX)
Section titled “Shared Storage (RWX)”Multiple indexer workers sharing the same PersistentVolume:
- All workers process from the same Redis queue
- Requires ReadWriteMany storage (NFS, CephFS, EFS)
- All features work: search, file browsing, replace
Hash-Based Sharding
Section titled “Hash-Based Sharding”Each shard handles a subset of repositories:
┌─────────────────────────────────────────────────────────────────┐│ API Server ││ ││ ┌──────────┐ ┌──────────────────┐ ┌──────────────────────┐ ││ │ Search │ │ Federated Files │ │ Federated Replace │ ││ │(parallel)│ │ (proxy) │ │ (fan-out) │ ││ └──────────┘ └──────────────────┘ └──────────────────────┘ │└─────────────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼┌──────────────┐ ┌──────────────┐ ┌──────────────┐│ Indexer-0 │ │ Indexer-1 │ │ Indexer-2 ││ + Zoekt │ │ + Zoekt │ │ + Zoekt ││ (shard 0) │ │ (shard 1) │ │ (shard 2) │└──────────────┘ └──────────────┘ └──────────────┘- Repos distributed via consistent FNV hashing
- Search queries all Zoekt shards in parallel
- File browsing proxied to correct shard
- Replace jobs fanned out and results merged
See Sharding Configuration for details.
Technology Stack
Section titled “Technology Stack”| Layer | Technology |
|---|---|
| Frontend | Next.js, React, TailwindCSS |
| API | Go, Chi router |
| Database | PostgreSQL / MySQL |
| Queue | Redis |
| Search | Zoekt (trigram index) |
| Container | Docker |
| Orchestration | Kubernetes (optional) |
Communication Patterns
Section titled “Communication Patterns”Synchronous
Section titled “Synchronous”- Web UI → API: HTTP REST
- CLI → API: HTTP REST
- API → Zoekt: HTTP
- API → Indexer: HTTP (federated file/replace in sharded mode)
Asynchronous
Section titled “Asynchronous”- API → Indexer: Redis job queue
- Indexer → Code Hosts: Git clone/fetch
Next Steps
Section titled “Next Steps”- Components - Detailed component docs
- Data Flow - Request and indexing flows
- Indexing - How indexing works
- Sharding - Horizontal scaling