Skip to content

Latest commit

 

History

History
238 lines (188 loc) · 11 KB

README.md

File metadata and controls

238 lines (188 loc) · 11 KB

Self-hosting SixSentences

This is the small reference deployment for one community installation. It runs PostgreSQL, one API process, one all-lanes background worker, the open web client and Caddy. PostgreSQL remains only on an internal backend network. The API publishes no host port and Caddy reaches it over that backend; the API and worker also use a separate outbound network for operator-selected providers. Caddy is the only public service.

The deployment creates no account with any SixSentences-operated service. Mail, scholarly metadata, model processing, speech processing and optional document extraction use only endpoints and credentials supplied by the operator.

Requirements

  • Docker Engine with Docker Compose 2.33.1 or newer;
  • at least 4 GiB RAM plus space for the database, documents and backups;
  • for internet deployment, one DNS name pointing to the host and inbound TCP ports 80/443 plus UDP 443;
  • an SMTP account before enabling public registration; and
  • reviewed privacy information and provider terms before enabling any external PubMed, model, web-search or spoken-interview processing.

Quick start

One command writes a private local configuration, runs the preflight below, starts the stack health-gated, and creates the first account:

make up

It runs quickstart.sh, which is exactly the documented sequence with a health gate and the owner prompt attached; every step can still be run by hand. The same script serves a public deployment:

bash deploy/community/quickstart.sh --domain research.example.org

An existing environment file is reused, never overwritten, so re-running the script is a safe way to restart a deployment after a configuration change.

The workspace starts without downloading a literature corpus or contacting a model provider. Corpus-dependent literature searches remain unavailable until the operator imports a corpus; its status remains visible in operator diagnostics. API readiness requires the database, storage, queue and a live background worker. Both API and worker start after migration, without waiting on each other.

After the unified release workflow checks and attaches IMAGE_DIGESTS and its runtime evidence before publication, a release supplies an API image and a localhost web image. Pointing the deployment at them replaces the longest step of a first start:

RELEASE=v0.2.0-alpha.4  # use a published release with IMAGE_DIGESTS attached
export SIX_API_IMAGE=ghcr.io/sixsentences/community-api:$RELEASE
export SIX_WEB_IMAGE=ghcr.io/sixsentences/community-web:$RELEASE-localhost
bash deploy/community/quickstart.sh --pull

The web client bakes SIX_PUBLIC_ORIGIN and SIX_PUBLIC_API_URL at build time, so a published web image serves only the origin it was built for. A TLS deployment keeps building its own web image and may still pull the API image.

Checking a running deployment

The preflight proves a configuration is consistent before the stack starts. Afterwards, one command reports what each part can actually do — whether the database carries its tables, whether documents can be written, whether the configured mail host accepts a session, and which optional features are off and why:

docker compose --env-file .env.selfhost exec api six-community doctor

A feature that is switched off is reported as off rather than as broken, and no configured secret appears in the output, so the result can be pasted into an issue. quickstart.sh runs it once after the stack starts.

To watch the same answers from a monitor rather than read them, add --json:

docker compose --env-file .env.selfhost exec api six-community doctor --json

That prints one object per check — name, state (ok, off or failed) and detail — as the whole of standard output, and nothing else, so it can be piped straight into a parser. The exit code is the same in both modes: non-zero when any check failed. Those three field names are a stable interface; alerting on them is safe.

First account

Public registration stays off until mail is configured, so the first account is created on the command line — quickstart.sh prompts for it, and it can be run again at any time:

docker compose --env-file .env.selfhost run --rm api \
  six-community auth create-owner --email [email protected] --org "My Lab"

The password is prompted twice and is never accepted as a flag, so it reaches neither a shell history nor a process list.

Production TLS setup

Generate a private configuration without printing its secrets:

bash deploy/community/init-env.sh --domain research.example.org
chmod 600 .env.selfhost

Edit .env.selfhost and replace the example sender, legal-document location, metadata contact and optional provider settings. Keep registration and spoken interviews disabled until their respective mail and processing prerequisites are complete. The three generated secrets serve different purposes and must not be copied between fields.

Run the fail-closed check, then build and start:

bash deploy/community/preflight.sh .env.selfhost
docker compose --env-file .env.selfhost build
docker compose --env-file .env.selfhost up --detach --wait

Caddy obtains and renews public certificates automatically. The web client is built with SIX_PUBLIC_ORIGIN and SIX_PUBLIC_API_URL; changing either value requires rebuilding the web image. The public API lives below /api, while all other paths go to the web client.

Inspect state without exposing secret values:

docker compose --env-file .env.selfhost ps
docker compose --env-file .env.selfhost logs --tail 100 api worker web proxy
curl --fail https://research.example.org/api/health/ready

Local HTTP mode

init-env.sh --local emits loopback-only HTTP URLs, configures Caddy not to request certificates and sets the web image's explicit local-build opt-in. The Dockerfile accepts that exception only for localhost and 127.0.0.1; every other non-HTTPS origin still fails the image build. TLS deployments must keep the opt-in disabled.

External services

All external features are off when their credentials are empty.

  • SIX_SMTP_* configures transactional mail. Public registration additionally requires email verification and a non-example sender.
  • SIX_OPENALEX_* identifies scholarly-metadata requests.
  • SIX_PUBMED_ENABLED=true plus a valid contact SIX_PUBMED_EMAIL enables the optional native PubMed arm; SIX_PUBMED_API_KEY is optional and remains server-side. The browser sees only a boolean capability from /models. Enabling it sends the generated scholarly query, including year filters, to NCBI but sends no SixSentences account, workspace, project or protocol ID. Disclose that external query transfer in the deployment privacy information and make NCBI's Disclaimer and Copyright notice visible before rollout.
  • SIX_OPENROUTER_*, SIX_WEBSEARCH_*, SIX_LLM_ROUTING and the local budget configure optional model/search processing. Use a credential-free HTTPS base URL and archive the applicable processor assessment first.
  • Spoken interviews require a server-side SIX_GEMINI_API_KEY plus both explicit confirmation switches. Those switches do not replace participant notice, consent, retention and transfer review.
  • SIX_TYPESAFE_API_KEY only configures the isolated TypeSafe Jev shadow evaluator. It does not enable a product/API route or alter screening. Before using it, follow the public-data, legal and validation boundary.
  • SIX_GROBID_URL optionally selects an operator-controlled document parser.

Never put provider credentials in NEXT_PUBLIC_* values. Browser-visible build arguments contain origins, versions and an optional public OAuth client ID only. The full provider, provenance and rollout boundary is documented in Native multi-source retrieval.

Browser Capture

Build the extension in apps/browser-extension for this deployment's exact SIX_PUBLIC_ORIGIN and SIX_PUBLIC_API_URL. Load the generated directory unpacked, then copy the ID shown by Chrome or Edge. Set SIX_BROWSER_CAPTURE_REDIRECT_URIS to exactly https://<extension-id>.chromiumapp.org/ and restart the API before pairing. The variable is intentionally empty in generated configuration, so an unknown extension can never pair by default. See the extension README for the stable-ID option, local HTTP exception, permission boundary, and deterministic build.

Updates and scaling

Create and verify a backup before changing images. Re-run the preflight, build the new images and use docker compose up --detach --wait. The dedicated one-shot migrate service owns schema migration; restore with the same source revision before attempting a version upgrade.

Resizable pinboards add a size field to stored note JSON, without a new SQL migration. Existing notes read as size 1.0 without being rewritten, and old cached clients preserve saved sizes when editing. Once notes are saved by this version, do not run a pre-size API against that database: its strict schema rejects the new field. Prefer a forward fix; an old binary requires its matching pre-upgrade database/files backup, with subsequent deletions replayed. Never strip saved sizes to make a downgrade pass.

CI runs rehearse.py against a randomly named, loopback-only Compose project: fresh released API, upgrade to candidate, authenticated pinboard round-trip, database/file backup and restore, then a separate fresh candidate startup. It requires --confirm DISPOSABLE, accepts no existing env file, generates new credentials, strips inherited provider settings, and removes only its own synthetic volumes. It uses an already running Docker engine and never starts one. This gate does not send mail or make inference calls; normal operator registration, browser, provider, and offsite recovery acceptance remain separate.

The reference worker uses the all lane at low concurrency. Larger deployments should split worker lanes only after measuring PostgreSQL connections, memory, provider limits and queue latency. Do not expose PostgreSQL or port 8000 merely to scale the frontend.

Security boundary

  • Images run read-only and without Linux capabilities where their software permits it; persistent writes are limited to named volumes and bounded tmpfs.
  • The environment file must be a regular mode-0600/0400 file. Preflight rejects placeholder keys, unsafe public URLs, a backup path inside the checkout and public registration without verified mail.
  • Caddy access logging is intentionally not enabled by this reference config, because public participation capabilities may appear in URL paths.
  • Docker administrators can inspect container environments. Protect host access and use an external secret manager when the installation outgrows one host.
  • The compose file does not itself implement firewalling, OS updates, encrypted offsite backup, retention deletion or incident response; the operator owns those controls.

See BACKUP-RESTORE.md before accepting user data.