# Novu Admin Guide This guide is for the engineers, architects, and operators who **deploy, configure, secure, and extend** Novu. It documents the system architecture, the self-hosting topology, the complete environment-variable reference, security posture, operational procedures, and the internals of the monorepo deeply enough to reason about — or contribute to — the platform. For day-to-day usage (triggering workflows, the Inbox component, the CLI, providers), see the companion **Novu User Guide**. > [!NOTE] > This guide reflects the open-source repository `novuhq/novu` at commit `b6ab172`. Novu is an **open-core** product: the core is MIT-licensed; the `enterprise/` tree and `*/ee/*` folders are under a commercial license. Where a capability is enterprise-only, this guide says so. --- ## Table of Contents 1. [Executive Summary](#1-executive-summary) 2. [Architecture](#2-architecture) 3. [Repository Structure](#3-repository-structure) 4. [System Requirements](#4-system-requirements) 5. [Deployment (Self-Hosting)](#5-deployment-self-hosting) 6. [Configuration Reference](#6-configuration-reference) 7. [Data Stores & Schema](#7-data-stores--schema) 8. [The Workflow Engine & Queues](#8-the-workflow-engine--queues) 9. [Authentication & Authorization](#9-authentication--authorization) 10. [Security Hardening](#10-security-hardening) 11. [Observability & Operations](#11-observability--operations) 12. [Module Reference (Internals)](#12-module-reference-internals) 13. [Code Patterns & Conventions](#13-code-patterns--conventions) 14. [Testing Strategy](#14-testing-strategy) 15. [Extending Novu](#15-extending-novu) 16. [Potential Improvements](#16-potential-improvements) 17. [Quick Reference](#17-quick-reference) --- ## 1. Executive Summary Novu is a **notification & agent-communication infrastructure** delivered as a set of cooperating Node.js services backed by MongoDB, Redis, ClickHouse, and object storage. A client triggers a **workflow** via the REST API; the API enqueues work onto Redis-backed **BullMQ** queues; a **worker** service executes the workflow's steps (channel sends, digests, delays) through pluggable **providers**; a **websocket** service streams in-app notifications to the embeddable Inbox in real time. | Aspect | Summary | | --- | --- | | **Language** | TypeScript (Node.js 22), strict monorepo via **Nx** + **pnpm** workspaces. | | **Runtime services** | `api`, `worker`, `ws`, `dashboard`, plus optional `webhook` and `inbound-mail`. | | **Primary datastore** | MongoDB 8.x (via Mongoose, in `libs/dal`). | | **Queue / cache** | Redis (BullMQ for jobs, plus a cache service). | | **Analytics store** | ClickHouse (activity/metrics). | | **Object storage** | S3-compatible (attachments; LocalStack for local dev). | | **API framework** | NestJS 11 (CQRS commands/use-cases pattern). | | **Frontend** | React 19 + Vite + Radix UI + Tailwind (dashboard). | | **Auth** | Community: Passport (JWT, GitHub OAuth). Enterprise: Clerk / better-auth. | | **Licensing** | Open core — MIT core, commercial `enterprise/`. | > [!IMPORTANT] > All cross-channel delivery flows through the **worker** service, not the API. The API is a thin, authenticated front door that validates input and enqueues jobs. Scale the worker independently of the API. --- ## 2. Architecture ### 2.1 Runtime topology ```text ┌─────────────────────────────────────────────┐ Browser / Inbox ◀───▶│ ws (NestJS + socket.io) :3002 │ └───────────────▲─────────────────────────────┘ │ subscriber feed (Redis pub/sub) Your backend ──trigger──▶ ┌────────────┴───────────┐ POST /v1/events/trigger │ api (NestJS) :3000 │──┐ └────────────┬───────────┘ │ enqueue (BullMQ) Dashboard (React/Vite) ─▶ api │ ▼ :4000 │ ┌──────────────────┐ │ │ Redis (BullMQ) │ Inbound email (SMTP) ─▶ inbound-mail ──┤ └────────▲─────────┘ Provider webhooks ─▶ webhook ────────┘ │ consume ┌───────┴──────────┐ │ worker (NestJS) │──▶ Email/SMS/Push/Chat providers └───────┬──────────┘ ┌───────────────┬───────────────┬──────────────┴───────────────┐ ▼ ▼ ▼ ▼ MongoDB 8.x ClickHouse S3 (attachments) In-App (▶ ws ▶ Inbox) (entities) (analytics) ``` ### 2.2 Services | Service | Package | Port | Responsibility | | --- | --- | --- | --- | | **API** | `@novu/api-service` | 3000 | REST API: auth, trigger ingestion, CRUD for workflows/subscribers/integrations/topics, billing, health. Generates OpenAPI via `@nestjs/swagger`. | | **Worker** | `@novu/worker` | 3004 (health) | Consumes BullMQ queues; executes workflow steps, digests, delays; calls providers; writes execution details. | | **WS** | `@novu/ws` | 3002 | socket.io gateway streaming in-app notifications to the Inbox; Redis adapter for horizontal scale. | | **Dashboard** | `@novu/dashboard` | 4000 | React SPA (Vite) — the admin UI for workflows, integrations, activity, subscribers. | | **Webhook** | `@novu/webhook` | — | Receives inbound provider webhooks (delivery status, etc.) via Svix. | | **Inbound-mail** | `@novu/inbound-mail` | — | SMTP server (`smtp-server`) that parses inbound email into the conversation model. | ### 2.3 Shared libraries | Library | Role | | --- | --- | | `@novu/dal` | Data Access Layer — Mongoose schemas, repositories (repository pattern), indexes. | | `@novu/application-generic` | Shared backend building blocks: use-cases, services (cache, queues, storage, metrics, analytics), SSRF-safe HTTP, auth services, BullMQ wiring. | | `@novu/shared` | Cross-cutting types, DTOs, constants, provider catalogs, validation utilities (data/tags filters). | | `@novu/framework` | The code-first workflow engine + serve adapters (Bridge). | | `@novu/providers` | All channel provider implementations behind a common interface. | | `@novu/stateless` | Lightweight, stateless trigger primitives. | | `@novu/maily-core` / `@novu/maily-render` | The no-code email editor (Tiptap-based) and its HTML renderer. | ### 2.4 Edge workers (enterprise / Cloudflare) The `enterprise/workers/` tree contains Cloudflare Workers (Hono/Wrangler) used in Novu's hosted product: | Worker | Purpose | | --- | --- | | `scheduler` | Edge scheduling (Hono + valibot). | | `socket` | Edge websocket fan-out (`ws`, JWT). | | `step-resolver` | Executes code-defined steps at the edge in a dispatch namespace (HMAC-secured). | | `thalamus-observer` | Agent observability. | > [!NOTE] > The Cloudflare workers are part of the enterprise/hosted topology. A community self-host does **not** need them — the in-cluster `ws` and `worker` services cover the same responsibilities. --- ## 3. Repository Structure ```text novu/ ├── apps/ Runtime services (deployable) │ ├── api/ @novu/api-service (NestJS REST API) │ ├── worker/ @novu/worker (BullMQ consumers) │ ├── ws/ @novu/ws (socket.io gateway) │ ├── dashboard/ @novu/dashboard (React + Vite SPA) │ ├── webhook/ @novu/webhook (inbound provider webhooks) │ └── inbound-mail/ @novu/inbound-mail (SMTP server) ├── libs/ Internal backend libraries │ ├── dal/ Mongoose schemas + repositories │ ├── application-generic/ Shared use-cases & services │ ├── testing/ Test harness (sessions, seed data) │ ├── automation/ Code generators (e.g. new provider) │ ├── notifications/ System notification templates │ └── maily-core, maily-render, ... ├── packages/ Published npm packages (SDKs + CLI) │ ├── novu/ the `novu` CLI │ ├── framework/ @novu/framework (code-first workflows) │ ├── js, react, nextjs, Inbox SDKs │ │ react-native/ │ ├── api/ @novu/api (generated server SDK → libs/internal-sdk) │ ├── providers/ channel providers │ ├── shared/ shared types/DTOs/consts │ ├── stateless/ stateless triggers │ └── agent-toolkit/ ACI agent toolkit ├── enterprise/ Commercial-license code │ ├── packages/ ee-ai, ee-api, ee-auth, ee-billing, ee-translation │ └── workers/ Cloudflare edge workers ├── docker/ Self-host (community) + local dev compose ├── playground/ Example Next.js & NestJS apps └── nx.json, pnpm-workspace.yaml, biome.json ``` > [!TIP] > Build orchestration is **Nx** (`nx run-many`, `nx affected`). Linting/formatting is **Biome** (not ESLint/Prettier). The default git base branch is `next`. Conventional Commits are enforced (`type(scope): description`). --- ## 4. System Requirements | Component | Requirement | | --- | --- | | Node.js | `>=22 <23` (engines field). | | Package manager | pnpm `^11` (`only-allow pnpm` enforced via preinstall). | | MongoDB | 8.x (self-host compose uses `mongo:8.0.17`). | | Redis | Any modern Redis (compose uses `redis:alpine`). | | ClickHouse | 24.x (for analytics; `clickhouse/clickhouse-server:24.3-alpine` in local compose). | | Object storage | S3-compatible bucket (LocalStack for local dev). | | Docker | Docker Engine + Compose v2 (for self-host). | | Build tooling | `openssl`, `curl` (community installer). | ### 4.1 Minimum self-host footprint The community compose runs four app containers (`api`, `worker`, `ws`, `dashboard`) plus `mongodb` and `redis`. For a small deployment, plan for ~2 vCPU / 4 GB RAM as a starting point and scale the `worker` first under load. --- ## 5. Deployment (Self-Hosting) ### 5.1 One-line community install ```bash curl -fsSL https://raw.githubusercontent.com/novuhq/novu/next/docker/community/setup.sh | bash ``` The installer (`docker/community/setup.sh`): 1. Checks for `curl`, `docker`, `openssl`, and Compose v2. 2. Downloads `docker-compose.yml` and `.env.example`. 3. Generates secrets with `openssl rand -hex`: `JWT_SECRET` (32), `STORE_ENCRYPTION_KEY` (16), `NOVU_SECRET_KEY` (32). 4. `chmod 600 .env` and runs `docker compose up -d`. Resulting endpoints: ```text Dashboard : http://localhost:4000 API : http://localhost:3000 WebSocket : http://localhost:3002 ``` ### 5.2 The compose stack The community `docker/community/docker-compose.yml` defines: | Service | Image | Notes | | --- | --- | --- | | `redis` | `redis:alpine` | Job queue + cache. Healthcheck `redis-cli ping`. | | `mongodb` | `mongo:8.0.17` | Primary datastore, root creds from env, named volume `mongodb:/data/db`. | | `api` | `ghcr.io/novuhq/novu/api:3.17.0` | Waits for healthy mongo+redis. Health: `GET /v1/health-check`. | | `worker` | `ghcr.io/novuhq/novu/worker:3.17.0` | Consumes queues. Health on `:3004`. | | `ws` | `ghcr.io/novuhq/novu/ws:3.17.0` | socket.io gateway. | | `dashboard` | `ghcr.io/novuhq/novu/dashboard:3.17.0` | Served on `:4000`; built with `VITE_API_HOSTNAME` / `VITE_WEBSOCKET_HOSTNAME`. | > [!IMPORTANT] > The prebuilt images are published to **GitHub Container Registry** under `ghcr.io/novuhq/novu/:`. Pin the version tag (e.g. `3.17.0`) in production rather than tracking a moving tag, and bump deliberately. ### 5.3 Production deployment checklist - [ ] Generate strong, unique `JWT_SECRET`, `STORE_ENCRYPTION_KEY`, `NOVU_SECRET_KEY` and store them in a secrets manager (never in the repo). - [ ] Point `MONGO_URL` at a replica-set MongoDB with auth enabled. - [ ] Use a managed/secured Redis with `REDIS_PASSWORD` (and TLS via `REDIS_TLS` if available). - [ ] Set `FRONT_BASE_URL`, `API_ROOT_URL`, and dashboard `VITE_API_HOSTNAME`/`VITE_WEBSOCKET_HOSTNAME` to externally reachable HTTPS URLs. - [ ] Terminate TLS at a reverse proxy in front of `api`, `ws`, and `dashboard`. - [ ] Configure object storage (`S3_BUCKET_NAME`, `S3_REGION`, credentials) for attachments. - [ ] Set `IS_SELF_HOSTED=true`. - [ ] Enable rate limiting (`IS_API_RATE_LIMITING_ENABLED=true`) and idempotency (`IS_API_IDEMPOTENCY_ENABLED=true`) if desired. - [ ] Wire health checks (`/v1/health-check`) into your orchestrator's liveness/readiness probes. - [ ] Configure Sentry/New Relic if you use them. ### 5.4 Upgrades & rollback - **Upgrade:** bump the image tags for `api`/`worker`/`ws`/`dashboard` together to the same version and `docker compose up -d`. The worker runs MongoDB migrations on boot when `MONGO_AUTO_CREATE_INDEXES`/migration scripts apply; review release notes for ClickHouse migrations (`clickhouse:migrate:prod`). - **Rollback:** revert the image tags to the previous version. Because services share schemas, roll all four back together; avoid running mismatched versions. > [!WARNING] > Do not run a newer `api` against an older `worker` (or vice-versa) for long — they share the BullMQ job shape and MongoDB schema. Deploy all services as a set. --- ## 6. Configuration Reference All services are configured by environment variables (parsed/validated with `envalid`/`dotenv`). The tables below list the variables found in the source. ### 6.1 Core | Variable | Service(s) | Purpose | | --- | --- | --- | | `NODE_ENV` | all | `production` / `development` / `test`. | | `PORT` / `API_PORT` / `WS_PORT` / `WORKER_PORT` | per service | Listen ports (defaults: api 3000, ws 3002, worker 3004, dashboard 4000). | | `API_ROOT_URL` | api, worker | Public base URL of the API. | | `FRONT_BASE_URL` | api | Public base URL of the frontend/dashboard. | | `API_CONTEXT_PATH` / `WS_CONTEXT_PATH` | api, ws | Optional path prefix when reverse-proxied under a subpath. | | `IS_SELF_HOSTED` | api, libs | Marks a community self-hosted deployment. | ### 6.2 Secrets | Variable | Purpose | | --- | --- | | `JWT_SECRET` | Signs dashboard/user JWTs (community auth) and WS auth. | | `STORE_ENCRYPTION_KEY` | Encrypts stored provider credentials / secrets at rest. | | `NOVU_SECRET_KEY` | The API secret key used for server-side triggering. | | `NOVU_INTERNAL_SECRET_KEY` | Internal service-to-service secret. | | `SUBSCRIBER_WIDGET_JWT_EXPIRATION_TIME` | Lifetime of subscriber (Inbox) tokens. | > [!CAUTION] > `STORE_ENCRYPTION_KEY` encrypts integration credentials in the database. **Rotating or losing it makes stored provider credentials unreadable.** Back it up securely and rotate only with a documented re-encryption procedure. ### 6.3 Datastores | Variable | Purpose | | --- | --- | | `MONGO_URL` | MongoDB connection string. | | `MONGO_MIN_POOL_SIZE` / `MONGO_MAX_POOL_SIZE` | Connection pool bounds. | | `MONGO_MAX_IDLE_TIME_IN_MS` | Idle connection timeout. | | `MONGO_AUTH_MECHANISM` | Mongo auth mechanism. | | `MONGO_AUTO_CREATE_INDEXES` | Whether to auto-create indexes on boot. | | `REDIS_HOST` / `REDIS_PORT` / `REDIS_PASSWORD` | Redis (BullMQ) connection. | | `REDIS_DB_INDEX` / `REDIS_PREFIX` / `REDIS_TLS` | Redis db index, key prefix, TLS toggle. | | `REDIS_CACHE_SERVICE_HOST` / `REDIS_CACHE_SERVICE_PORT` | Separate cache Redis. | | `S3_BUCKET_NAME` / `S3_REGION` / `S3_LOCAL_STACK` | Object storage for attachments. | | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | S3 credentials. | ### 6.4 Worker tuning | Variable | Purpose | | --- | --- | | `WORKER_DEFAULT_CONCURRENCY` | Concurrent jobs per worker. | | `WORKER_DEFAULT_LOCK_DURATION` | BullMQ lock duration (ms). | | `ACTIVE_WORKERS` | Which worker queues this instance runs. | | `BROADCAST_QUEUE_CHUNK_SIZE` / `MULTICAST_QUEUE_CHUNK_SIZE` | Fan-out batch sizes for topic/broadcast triggers. | | `SQS_DEFAULT_CONCURRENCY` / `SQS_DEFAULT_VISIBILITY_TIMEOUT` / `SQS_DEFAULT_BATCH_SIZE` / `SQS_DEFAULT_WAIT_TIME_SECONDS` | SQS-mode tuning (alternative queue backend). | | `IN_MEMORY_CLUSTER_MODE_ENABLED` | In-memory cluster mode flag. | ### 6.5 Feature flags & behavior | Variable | Purpose | | --- | --- | | `IS_API_RATE_LIMITING_ENABLED` | Enable API rate limiting (throttler). | | `IS_API_IDEMPOTENCY_ENABLED` | Enable idempotent trigger handling. | | `IS_V2_ENABLED` | Enable v2 API surface. | | `IS_NEW_MESSAGES_API_RESPONSE_ENABLED` | New messages API response shape. | | `IS_EMAIL_INLINE_CSS_DISABLED` | Disable email inline-CSS step. | | `IS_USE_MERGED_DIGEST_ID_ENABLED` | Merged digest ID behavior. | | `NOVU_STRICT_AUTHENTICATION_ENABLED` | Strict auth mode. | | `NOVU_ENTERPRISE` / `CI_EE_TEST` | Enable enterprise code paths. | ### 6.6 Auth & integrations | Variable | Purpose | | --- | --- | | `GITHUB_OAUTH_CLIENT_ID` / `GITHUB_OAUTH_CLIENT_SECRET` | GitHub OAuth (community login). | | `CLERK_SECRET_KEY` | Clerk auth (enterprise). | | `EE_AUTH_PROVIDER` | Selects the enterprise auth provider. | | `SVIX_API_KEY` | Svix (webhook delivery infra). | | `MAIL_SERVER_DOMAIN` | Inbound mail / domain features. | | `NOVU_MANAGED_CLAUDE_API_KEY` | Managed agent (Claude) key. | | `STEP_RESOLVER_CF_ACCOUNT_ID` / `STEP_RESOLVER_CF_API_TOKEN` / `STEP_RESOLVER_CF_DISPATCH_NAMESPACE` | Cloudflare step-resolver deploy (enterprise). | | `SOCKET_IO_ADMIN_USERNAME` / `SOCKET_IO_ADMIN_PASSWORD_HASH` | socket.io admin UI auth. | ### 6.7 Observability | Variable | Purpose | | --- | --- | | `SENTRY_DSN` | Sentry error reporting. | | `NEW_RELIC_ENABLED` / `NEW_RELIC_APP_NAME` / `NEW_RELIC_LICENSE_KEY` | New Relic APM. | | `ENABLE_OTEL` | OpenTelemetry export. | | `MIXPANEL_TOKEN` / `SEGMENT_TOKEN` | Product analytics. | | `IS_ANALYTICS_LOGS_ENABLED` | Analytics logging toggle. | > [!NOTE] > The dashboard is a static SPA built with Vite; its only runtime env vars are `VITE_API_HOSTNAME` and `VITE_WEBSOCKET_HOSTNAME`, which are baked at build time and must point to browser-reachable URLs. --- ## 7. Data Stores & Schema ### 7.1 MongoDB (primary) All durable domain state lives in MongoDB, accessed through `libs/dal` using **Mongoose** with the **repository pattern**. Core entities (see `packages/shared/src/entities`): | Entity | Holds | | --- | --- | | `Organization`, `Environment`, `User` | Tenant, environment, and user accounts. | | `Subscriber`, `SubscriberPreference` | Recipients and their channel preferences. | | `NotificationTemplate` (workflow) | Workflow definitions and steps. | | `Notification`, `Job`, `Message` | A triggered notification, its per-step jobs, and rendered messages. | | `MessageTemplate` | Per-step channel content templates. | | `Integration` | Provider credentials/config per channel & environment. | | `Topic` | Subscriber groups for broadcast. | | `Layout` | Reusable email wrappers. | | `ExecutionDetails` | Per-step execution audit trail (shown in Activity Feed). | | `Change` | Versioned changes for promotion between environments. | | `Tenant`, `WorkflowOverride` | Multi-tenant per-tenant overrides. | > [!TIP] > When adding queries, add the matching MongoDB index in the repository. CodeRabbit's repo rules flag schema changes that need migrations and queries missing indexes — follow the repository pattern and keep reads indexed. Migrations live in `apps/api/migrations/` and run via the `migration` script. ### 7.2 Redis Two roles: **BullMQ job queues** (the heart of async processing) and a **cache service** (often a separate Redis instance via `REDIS_CACHE_SERVICE_*`). The `ws` service also uses a Redis adapter for socket.io fan-out across instances. ### 7.3 ClickHouse Analytics and high-volume activity/metrics data (charts, interaction trends, workflow-run counts) are stored in ClickHouse. Migrations are managed with `clickhouse-migrations` (`clickhouse:migrate:local` / `:prod`). ### 7.4 Object storage (S3) Attachments and large assets go to an S3-compatible bucket. LocalStack provides S3 locally (`docker/local/docker-compose.agent.yml`). --- ## 8. The Workflow Engine & Queues ### 8.1 Lifecycle of a trigger ```text POST /v1/events/trigger └─ api: authenticate (ApiKey) → validate → upsert subscriber(s) └─ enqueue "trigger" job (BullMQ) └─ worker: parse event → fan out per subscriber/topic └─ build notification jobs (one per step) └─ execute steps in order: ├─ channel step → resolve provider → send → record ExecutionDetails ├─ digest step → hold & aggregate over window ├─ delay step → schedule resume └─ inApp step → persist Message → ws → ``` ### 8.2 BullMQ queues The worker runs multiple queues (e.g. standard, workflow, subscriber-process, digest/unsnooze). Concurrency and lock duration are tuned with `WORKER_DEFAULT_CONCURRENCY` and `WORKER_DEFAULT_LOCK_DURATION`. Topic/broadcast fan-out is chunked (`*_QUEUE_CHUNK_SIZE`) to avoid flooding. > [!NOTE] > Novu also supports an **SQS** queue backend (the `SQS_DEFAULT_*` vars) as an alternative to Redis/BullMQ for managed deployments. Community self-hosts use BullMQ on Redis. ### 8.3 Digest & delay durability Digest and delay steps are **durable**: the worker persists the job state and resumes when the window/timer elapses. This survives restarts and does not hold open connections. Jobs missing required fields fail fast with `Job data is missing required fields: …`. ### 8.4 Idempotency & retries With `IS_API_IDEMPOTENCY_ENABLED`, duplicate triggers carrying the same idempotency key are de-duplicated. Failed channel sends are retried per the channel's policy; exhausted retries are recorded in `ExecutionDetails` and surfaced in the Activity Feed. --- ## 9. Authentication & Authorization ### 9.1 Two auth modes | Mode | Mechanism | | --- | --- | | **Community** | Passport strategies: JWT (`passport-jwt`), API key (`passport-headerapikey`), GitHub OAuth (`passport-github2`, `passport-oauth2`). `JWT_SECRET` signs tokens. | | **Enterprise** | Clerk and/or better-auth (`@novu/ee-auth`), selected via `EE_AUTH_PROVIDER` / `CLERK_*`. Supports SSO. | ### 9.2 API authentication Server-to-server requests authenticate with `Authorization: ApiKey ` (environment-scoped). The Bridge sync uses the same scheme plus an HMAC signature (`t=,v1=`) for request integrity. ### 9.3 Subscriber (Inbox) auth The Inbox authenticates a subscriber with the public Application Identifier plus, in production, an **HMAC `subscriberHash`** computed server-side from the Secret Key. Subscriber tokens expire per `SUBSCRIBER_WIDGET_JWT_EXPIRATION_TIME`. > [!WARNING] > Enable HMAC subscriber verification for the In-App integration in production. Without it, a `subscriberId` alone can read another user's feed. --- ## 10. Security Hardening Novu has several first-class security mechanisms visible in the source — treat these as the baseline posture. ### 10.1 SSRF protection (outbound requests) Because Novu makes outbound HTTP to user-supplied URLs (Bridge endpoints, webhook providers, provider APIs), it has a hardened outbound HTTP layer. The codebase **forbids** the naive one-shot `validateUrlSsrf` check and mandates `safeOutboundRequest` / `safeOutboundJsonRequest` (or `HttpClientService` with `enforceSsrfProtection: true`), which pins the connection to a validated IP and **re-validates every redirect** (defeating DNS-rebinding and redirect-based SSRF). Blocked requests raise `SsrfBlockedError` with reasons `INVALID_URL`, `CREDENTIALS_IN_URL`, `BLOCKED_HOSTNAME`, or `DNS_LOOKUP_FAILED`. > [!CAUTION] > This rule is enforced at lint time via Biome `noRestrictedImports`. When writing code that makes outbound calls, never import `validateUrlSsrf` — use the safe-outbound helpers so SSRF protection cannot be bypassed. ### 10.2 Secrets at rest Provider credentials are encrypted with `STORE_ENCRYPTION_KEY` before being stored in MongoDB. Keep this key in a secrets manager and back it up. ### 10.3 HTTP hardening The API uses `helmet` for security headers, `compression`, CORS configured per `WIDGET_BASE_URL`/`DOCS_BASE_URL`, and rate limiting via `@nestjs/throttler` (and `@upstash/ratelimit`) when `IS_API_RATE_LIMITING_ENABLED=true`. Input is validated with `class-validator` + `ajv` on every DTO. ### 10.4 Input & filter validation User-supplied workflow conditions are validated with strict bounds — `DataFilterValidationError` and `TagsFilterValidationError` enforce maximum group/value counts, scalar-only values, and mutually-exclusive AND/OR. This prevents pathological or abusive filter payloads. ### 10.5 Path traversal & file safety The `add-inbox` CLI and other file-touching code reject paths outside the working directory (`Access denied: Path outside working directory`). ### 10.6 Dependency hygiene `pnpm-workspace.yaml` carries an extensive `overrides` map pinning transitive dependencies above known-vulnerable versions, and `minimumReleaseAge` (4320 minutes ≈ 3 days) delays adopting brand-new releases to dodge supply-chain attacks. Renovate keeps deps current. > [!IMPORTANT] > The known issue **GH #10124** (ws crash on connection reading `subscriber.contextKeys.length`) is a reliability/security-adjacent bug fixed in newer versions — keep the `ws` image and client SDKs current. --- ## 11. Observability & Operations ### 11.1 Health checks Every service exposes `GET /v1/health-check` (used by the compose healthchecks and suitable for k8s liveness/readiness). Built with `@nestjs/terminus` and `@godaddy/terminus`. ```bash curl -fsS http://localhost:3000/v1/health-check ``` ```text { "status": "ok", "info": { "mongo": { "status": "up" }, "redis": { "status": "up" } } } ``` ### 11.2 Logging Structured logging uses **Pino** (`PinoLogger` from `@novu/application-generic` is mandated over NestJS's default `Logger` — enforced by Biome). Container logs are JSON with rotation (`max-size: 50m`, `max-file: 5`). ### 11.3 Metrics & tracing - **Sentry** (`SENTRY_DSN`) for error tracking across all services. - **New Relic** (`NEW_RELIC_*`) for APM. - **OpenTelemetry** (`ENABLE_OTEL`) for traces/metrics export. - Worker emits active-jobs metrics (`active-jobs-metric.service.ts`). ### 11.4 Routine operations | Task | How | | --- | --- | | View logs | `docker compose logs -f ` | | Restart a service | `docker compose restart worker` | | Scale workers | Run multiple `worker` containers; tune `WORKER_DEFAULT_CONCURRENCY` and `ACTIVE_WORKERS`. | | Run a DB migration | `pnpm --filter @novu/api-service migration