Stargate Docker advanced configuration¶

Backups¶

Automatic Backups¶

Daily backups run at 2:00 AM via cron (set up during install)
Backups stored in ./backups/ as timestamped .tar.gz files
Old backups (>7 days) are automatically cleaned up

What's Included in Backups¶

Full PostgreSQL dump (all databases with users and permissions)
Individual database dumps (for partial restore if needed)
Vault keys (vault-keys.json for unsealing)
Customer configuration (customer-config.sh with WireGuard key)
S/MIME CSR and certificates (any .crt, .pem, .cer files)
Backup manifest (manifest.json with metadata)

Manual Backup¶

./scripts/backup.sh

Creates a compressed archive in ./backups/YYYYMMDD_HHMMSS.tar.gz.

Restore from Backup¶

To restore on a new machine or after a purge:

## Copy the backup archive to the new machine, then:
./scripts/restore.sh backups/20260130_143022.tar.gz

The restore script will:

Stop any running services
Extract and validate the backup
Install Docker if needed
Restore customer configuration
Start infrastructure services (PostgreSQL, Vault, MinIO)
Restore the database
Unseal Vault with backed-up keys
Start application services

Partial Restore (single database)¶

If you only need to restore one database:

## Extract backup
tar -xzf backups/20260130_143022.tar.gz -C /tmp/

## Restore a specific database
cat /tmp/20260130_143022/database/mxengine.sql | docker exec -i stargate-postgres psql -U postgres -d mxengine

Updating Stargate¶

Update Deployment Scripts and Configuration¶

The Stargate deployment repository receives updates to scripts (onboard.sh, start.sh, health-check.sh, etc.), configuration templates, and documentation. To apply these updates:

## 1. Create a backup before updating
./scripts/backup.sh

## 2. Pull the latest changes from the repository
git pull

## 3. Restart services to pick up any script or config changes
./scripts/stop.sh
./scripts/start.sh

Note: git pull will not overwrite your customer-config.sh, .env, or secrets/ directory - these are in .gitignore. If you have local changes to tracked files (e.g. docker-compose.yml), git will warn you. In that case, stash your changes first with git stash, pull, then re-apply with git stash pop.

If the update includes changes to customer-config.sh.template, compare it with your existing config to see if new variables were added:

diff customer-config.sh customer-config.sh.template

Update Service Images¶

Update a Single Service¶

Edit the version in .env, then pull and recreate:

## Edit version in .env
sed -i 's/MXENGINE_VERSION=.*/MXENGINE_VERSION=v0.0.31/' .env

## Pull new image and recreate the service
docker compose pull mxengine
docker compose up -d --force-recreate mxengine

Quick Test (without editing .env)¶

Override the version directly:

MXENGINE_VERSION=v0.0.31 docker compose up -d --force-recreate mxengine

Update Multiple Services¶

## Edit versions in .env, then:
docker compose pull smimekeys-client policy idagent mxengine
docker compose up -d --force-recreate smimekeys-client policy idagent mxengine

Update All Services¶

## Pull all latest images
docker compose pull

## Recreate all services
docker compose up -d --force-recreate

Cleanup Old Images¶

After updates, remove unused images to free disk space:

docker image prune -f

Rollback¶

To rollback, edit .env to the previous version and recreate:

sed -i 's/MXENGINE_VERSION=.*/MXENGINE_VERSION=v0.0.30/' .env
docker compose up -d --force-recreate mxengine

Configuration¶

The .env file is automatically generated by install.sh from customer-config.sh. Domain, certificate, and WireGuard settings are updated by onboard.sh. To customize, edit customer-config.sh and re-run the appropriate script.

Key sections in the generated .env:

## PostgreSQL (auto-generated if empty in customer-config.sh)
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<auto-generated>

## Vault (auto-populated after initialization)
VAULT_TOKEN=<auto-generated>

## MinIO (auto-generated if empty in customer-config.sh)
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=<auto-generated>

## Application Versions
SMIMEKEYS_VERSION=v0.0.5
POLICY_VERSION=v0.0.5
IDAGENT_VERSION=v0.0.6-branch
MXENGINE_VERSION=v0.0.35

## Mail Configuration
MAIL_DOMAINS=example.com
MAIL_HOSTNAME=mail.example.com
MXENGINE_PUBLIC_ADDRESS=http://203.0.113.50:8084
OUTBOUND_SEALER_MX_DOMAIN=hintest.ch

## WireGuard
WG_LOCAL_IP=203.0.113.50
WG_INTERFACE_PORT=19818
WG_TRANSPORT_MODE=tcp

Do not edit .env directly. Changes will be overwritten by onboard.sh. Always edit customer-config.sh instead.

Service URLs¶

Service	URL/Port
smimekeys-client	http://localhost:8081
policy	http://localhost:8082
idagent	http://localhost:8083
mxengine HTTP	http://localhost:8084
mxengine SMTP	localhost:1587
Postfix SMTP	localhost:25
Postfix Reinjection	localhost:10026 (internal)
Vault UI	http://localhost:8200
MinIO Console	http://localhost:9001
PostgreSQL	localhost:5432

Health Checks¶

All services expose a /liveness endpoint:

curl http://localhost:8081/liveness  # smimekeys-client
curl http://localhost:8082/liveness  # policy
curl http://localhost:8083/liveness  # idagent
curl http://localhost:8084/liveness  # mxengine

Monitoring¶

Prometheus Metrics¶

All application services expose Prometheus metrics on port 2112 (internally), mapped to different host ports:

Service	Metrics Port	Metrics URL
smimekeys-client	2113	http://localhost:2113/metrics
idagent	2114	http://localhost:2114/metrics
policy	2115	http://localhost:2115/metrics
mxengine	2116	http://localhost:2116/metrics
node-exporter	9100	http://localhost:9100/metrics

Prometheus Scrape Config Example¶

scrape_configs:
  - job_name: 'stargate-smimekeys'
    static_configs:
      - targets: ['<host>:2113']
  - job_name: 'stargate-idagent'
    static_configs:
      - targets: ['<host>:2114']
  - job_name: 'stargate-policy'
    static_configs:
      - targets: ['<host>:2115']
  - job_name: 'stargate-mxengine'
    static_configs:
      - targets: ['<host>:2116']
  - job_name: 'stargate-node'
    static_configs:
      - targets: ['<host>:9100']

Quick Metrics Check¶

## Check all metrics endpoints
curl -s http://localhost:2113/metrics | head -20  # smimekeys-client
curl -s http://localhost:2114/metrics | head -20  # idagent
curl -s http://localhost:2115/metrics | head -20  # policy
curl -s http://localhost:2116/metrics | head -20  # mxengine
curl -s http://localhost:9100/metrics | head -20  # node-exporter

Log Collection (Promtail → Loki)¶

Promtail collects logs from application containers and ships them to Loki.

Containers monitored:

stargate-smimekeys-client
stargate-policy
stargate-idagent
stargate-mxengine

Configuration in .env:

## Loki push URL
LOKI_URL=https://loki.infra.vereign-cdn.com

## Hostname label for logs (auto-set to DEPLOYMENT_NAME)
PROMTAIL_HOSTNAME=stargate-acme

Labels added to logs:

environment=<DEPLOYMENT_NAME> - Identifies the deployment
host=<PROMTAIL_HOSTNAME> - Identifies the host (same as deployment name)
container=<container-name> - Container name
service=<service-name> - Service name (e.g., smimekeys-client, policy)
level=<log-level> - Extracted from JSON logs if available

Query logs in Grafana:

{environment="stargate-acme"} |= "error"
{environment="stargate-acme", service="mxengine"}
{environment="stargate-acme", level="error"}

Verify Promtail is working:

## Check Promtail status
docker logs stargate-promtail

## Check targets
curl -s http://localhost:9080/targets

Note: The VM's public IP must be whitelisted in Loki's ingress configuration.

Postfix Relay¶

The Postfix relay container automatically configures itself from DNS records and integrates with mxengine for mail processing.

Mail Flow Architecture¶

External Mail Server
         │
         ▼ (port 25)
┌─────────────────────────────────────────────────────┐
│ Postfix Relay (stargate-postfix-relay)              │
│                                                     │
│  Port 25 (main listener)                            │
│    │                                                │
│    ▼                                                │
│  content_filter = smtp:[mxengine]:1587              │
│    │                                                │
└────┼────────────────────────────────────────────────┘
     │
     ▼ (port 1587)
┌─────────────────────────────────────────────────────┐
│ MXEngine (stargate-mxengine)                        │
│                                                     │
│  Port 1587 (SMTP input)                             │
│    │                                                │
│    ▼                                                │
│  Sign/encrypt/process mail                          │
│    │                                                │
│    ▼                                                │
│  Deliver back to Postfix for relay                  │
│    │                                                │
└────┼────────────────────────────────────────────────┘
     │
     ▼ (port 10026)
┌─────────────────────────────────────────────────────┐
│ Postfix Relay (stargate-postfix-relay)              │
│                                                     │
│  Port 10026 (reinjection listener)                  │
│    │                                                │
│    ▼                                                │
│  transport_maps → relay to destination MX           │
│    │                                                │
└────┼────────────────────────────────────────────────┘
     │
     ▼ (port 25)
Destination Mail Server (via MX lookup)

Seal callback flow (inbound): When a remote sealer needs to deliver a sealed message, it calls MXENGINE_PUBLIC_ADDRESS (default: http://<SERVER_STATIC_IP>:8084). This is why port 8084 must be open for inbound traffic. The http:// protocol is correct — TLS is not required because the seal payload is already encrypted.

Configuration¶

Set MAIL_DOMAINS in your customer-config.sh:

## Required: Your mail domain(s), comma-separated for multiple
MAIL_DOMAINS=example.com

## Multiple domains:
MAIL_DOMAINS=example.com,example.org

The container will (for each domain):

Look up MX records to find the relay destination (where to deliver processed mail)
Parse SPF records recursively to find allowed sender networks (which IPs may send mail via port 25)
Auto-detect Docker networks for the port 10026 listener
Configure content_filter to route incoming mail through mxengine
Set up transport maps to relay processed mail to the destination MX

Mail Routing (Migrating from Old MGW)¶

Key difference from the old HIN-MGW: In the old MGW, you had to manually configure a target server per domain. In Stargate, Postfix automatically discovers where to deliver mail by looking up the MX records of each domain in DNS. Manual per-domain routing is also available via DOMAIN_RELAY_MAP if needed (see below).

How it works:

When the Stargate Postfix container starts, it queries the DNS MX records for each domain listed in MAIL_DOMAINS. It filters out its own hostname and uses the remaining MX entries as the delivery target. So for each domain, Postfix knows exactly which Exchange server to forward the processed mail to - based purely on DNS.

What you need to do:

For each of your domains, make sure there is an MX record in DNS pointing to the corresponding Exchange (or other mail) server:

domain1.com    MX 10  exchange1.domain1.com
domain2.com    MX 10  exchange2.domain2.com
domain3.com    MX 10  exchange3.domain3.com

This works for any number of domains - each domain can point to a different mail server, and Postfix will route accordingly.

If Stargate is the only MX record for a domain, Postfix will filter it out and have no delivery target. In that case, add a second MX record pointing to your mail server. Give Stargate a higher priority number (= lower priority) so it acts as the inbound gateway, and give your mail server a lower number (= higher priority) so Postfix uses it as the delivery target:

example.com    MX 10  exchange.example.com      ← delivery target (mail server)
example.com    MX 20  stargate.example.com      ← inbound gateway (Stargate)

Alternative - single relay for all domains:

If all your domains deliver to the same mail server, you can use RELAYHOST in customer-config.sh instead of MX records:

RELAYHOST=[smtp.office365.com]

Note: RELAYHOST sends all mail to a single host and does not support per-domain routing. For setups with multiple domains and different mail servers, use DOMAIN_RELAY_MAP or the MX-based approach.

Alternative - explicit per-domain relay mapping (sender-based):

If you prefer not to manage MX record priorities, you can configure explicit per-domain relay targets in customer-config.sh:

DOMAIN_RELAY_MAP="domain1.ch:[exchange1.domain1.ch]:25,domain2.ch:[exchange2.domain2.ch]:25"

Each entry maps a sender (envelope-From) domain to a specific relay host and port. After mxengine signs/encrypts the message, Postfix routes it via the listed relay based on the sender's domain - this is what implements the "relay back through M365" pattern. Mail from senders not listed falls back to RELAYHOST (if set) or MX lookup of the recipient.

Precedence (highest to lowest):

DOMAIN_RELAY_MAP - explicit per-sender-domain target (if the sender's domain is listed)
RELAYHOST - global fallback for all unmapped senders
MX lookup - automatic discovery from DNS for the recipient (default)

After updating MX records, restart the Postfix container so it picks them up:

docker compose restart postfix-relay

Ports¶

Port	Purpose
25	Main SMTP listener (external connections)
10026	Reinjection port (mxengine → postfix, internal only)
1587	MXEngine SMTP input (postfix → mxengine, internal only)

Manual Overrides¶

If DNS lookups fail or you need custom configuration:

## Skip MX lookup - specify relay host directly (all domains use the same host)
RELAYHOST=[smtp.office365.com]

## Per-domain relay targets (sender-based; for relay-back through M365/Exchange)
DOMAIN_RELAY_MAP="domain1.ch:[exchange1.domain1.ch]:25,domain2.ch:[exchange2.domain2.ch]:25"

## Skip SPF lookup - specify allowed networks directly
POSTFIX_MYNETWORKS=10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

Using Exchange? See Exchange-integration.md for the full Exchange Online / On-Premises connector and transport rule setup.

Verification¶

## Check Postfix status
docker exec stargate-postfix-relay postfix status

## View main configuration
docker exec stargate-postfix-relay postconf | grep -E 'relayhost|mynetworks|relay_domains|content_filter'

## View transport maps
docker exec stargate-postfix-relay postconf transport_maps
docker exec stargate-postfix-relay postmap -q '*' hash:/etc/postfix/transport

## Check master.cf (port 10026 listener)
docker exec stargate-postfix-relay grep -A5 "10026" /etc/postfix/master.cf

## Check logs
docker logs stargate-postfix-relay

## Test connection to port 25
telnet localhost 25

## Test internal port 10026 (from mxengine container)
docker exec stargate-mxengine nc -zv postfix-relay 10026

Rebuild After Changes¶

If you modify wrapper.sh or Dockerfile:

docker compose build postfix-relay
docker compose up -d postfix-relay

Troubleshooting¶

Mail not being processed by mxengine:

Check content_filter is set: docker exec stargate-postfix-relay postconf content_filter
Should show: content_filter = smtp:[mxengine]:1587
Verify mxengine is reachable: docker exec stargate-postfix-relay nc -zv mxengine 1587

Mail stuck after mxengine processing:

Check mxengine outbound config: OUTBOUND_SMTP_HOST=postfix-relay, OUTBOUND_SMTP_PORT=10026
Verify port 10026 listener: docker exec stargate-postfix-relay ss -tlnp | grep 10026
Check mynetworks on port 10026 includes Docker network (172.x.x.x/16)

Greylisting errors (450 4.7.1):

This is normal! The destination server is temporarily rejecting mail
Postfix automatically retries after ~5 minutes
Check queue: docker exec stargate-postfix-relay mailq

Microsoft blocking IP (S3140):

Your server's IP has poor reputation with Microsoft
Request delisting at: https://sender.office.com
May take 24-48 hours to take effect

DNS Lookup Failures:

Set DNS_SERVER=8.8.8.8 to use a specific DNS server
Use RELAYHOST and POSTFIX_MYNETWORKS to skip DNS lookups

Connection Refused on port 25:

Ensure port 25 is not blocked by firewall
Check if another service is using port 25: ss -tlnp | grep :25

WireGuard (Agent-to-Agent Communication)¶

IDAgent uses WireGuard to establish secure encrypted tunnels between Stargate instances for delivering sealed messages.

How It Works¶

Each Stargate instance uses its server's real static public IP as the WireGuard tunnel address. This guarantees uniqueness across all deployments without manual coordination.

┌──────────────────────────────────────────┐       ┌──────────────────────────────────────────┐
│ Your Stargate (203.0.113.50)             │       │ HIN Test (5.102.144.182)                 │
│                                          │       │                                          │
│  IDAgent (203.0.113.50:19818)            │◄─────►│  IDAgent (5.102.144.182:19818)           │
│     │                                    │  WG   │     │                                    │
│     ▼                                    │ Tunnel│     ▼                                    │
│  Sealed message delivery via WG tunnel   │ (TCP) │  Receive sealed message                  │
│                                          │       │                                          │
└──────────────────────────────────────────┘       └──────────────────────────────────────────┘

Configuration¶

WireGuard settings in customer-config.sh:

## ==============================================================================
## Server IP — used as WireGuard tunnel address and MXEngine callback URL
## ==============================================================================
SERVER_STATIC_IP="203.0.113.50"       # Your server's real static public IP

## ==============================================================================
## WireGuard local settings (typically left at defaults)
## ==============================================================================
WG_PRIVATE_KEY=""                     # Auto-generated by IDAgent, then saved back to config
WG_INTERFACE_PORT="19818"             # Default WireGuard port
WG_TRANSPORT_MODE="tcp"               # "tcp" (default) or "udp"

## ==============================================================================
## WireGuard peer (pre-filled for HIN Test — override for a different peer)
## ==============================================================================
WG_PEER_NAME="hin-test"
WG_PEER_PUBLIC_KEY="ol2zlG40M7+Rn81V9RUFmkIQV2ILLmEJHZww7HfoLxA="
WG_PEER_ENDPOINT="5.102.144.182:19818"
WG_PEER_IP="5.102.144.182"            # Peer's WireGuard tunnel address
WG_PEER_PORT="9090"                   # Peer's HTTP API port (over the tunnel)
WG_PEER_ALLOWED_IPS="5.102.144.182/32"
WG_PEER_EXTERNAL_ID="hintest.ch"      # Used for routing decisions
WG_PEER_DESCRIPTION="Connection to HIN Test IDAgent"

WG_LOCAL_IP is auto-derived from SERVER_STATIC_IP. You do not need to set it separately.

WG_PEER_IP vs WG_PEER_PORT: WG_PEER_IP is the remote peer's tunnel address (used for WireGuard routing). WG_PEER_PORT is the HTTP port the remote IDAgent listens on for API calls over the tunnel (default 9090). These are independent values.

Peer Connection Setup¶

The WireGuard peer connection is managed by onboard.sh (which runs the idagent-init container). The customer-config.example template comes with HIN Test peer defaults pre-filled, so the connection is set up automatically during install.sh.

The idagent-init container:

Waits for IDAgent to start and generate its WireGuard keypair
Open the logs of the IDAgent docker compose logs idagent and copy the wireguard public key: value example: V2Qvr...IB1A2wQCApmHY=
Get the public IP address for this machine
Get the domain name
The information from step 2, 3 and 4 should be provided to Vereign (kalin.canov@vereign.com)
For any other connection you would like to establish, through the Wireguard tunnel, you will need to provide to the other party the information from step 2, 3 and 4, and also receive/store their info. To store new peer after you received the info you should run the following curl command

curl --location 'localhost:8083/v1/connections' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
  "allowedIps": "<IP of new peer>/32",
  "description": "<short description>",
  "endpoint": "<IP of new peer>:19818",
  "externalId": [
    "<domain of new peer>"
  ],
  "name": "<Name of new peer>",
  "presharedKey": "",
  "publicKey": "<public key of new peer>",
  "status": "completed",
  "transport": "tcp",
  "wireguardIp": "<IP of new peer>",
  "wireguardPort": 10080
}'

Verification¶

## Check IDAgent WireGuard interface
docker exec stargate-idagent wg show

## Check connection in database
docker exec stargate-postgres psql -U postgres -d idagent \
  -c "SELECT connection_id, name, endpoint, wireguard_ip, transport, status FROM connections;"

## Check connection external IDs (used for routing)
docker exec stargate-postgres psql -U postgres -d idagent \
  -c "SELECT connection_id, external_id FROM connection_external_ids;"

## Test WireGuard connectivity (check tunnel status from host)
docker logs stargate-idagent 2>&1 | grep -i "handshake\|peer.*added\|started listening"

## Check IDAgent logs for tunnel activity
docker logs stargate-idagent | grep -i wireguard

Troubleshooting¶

No WireGuard interface:

Check IDAgent logs: docker logs stargate-idagent
Verify WG_LOCAL_IP is set in .env (auto-derived from SERVER_STATIC_IP — should be this server's static public IP)

Peer not reachable:

Verify remote endpoint is accessible: nc -zv <endpoint_host> <endpoint_port>
Check firewall allows TCP+UDP port 19818
Verify public keys match on both ends
If TCP has issues, try setting WG_TRANSPORT_MODE="udp" in customer-config.sh

Connection not in database:

Run idagent-init manually: docker compose run --rm idagent-init
Check idagent-init logs: docker logs stargate-idagent-init

Policy Sync¶

The policy-sync service automatically syncs OPA/Rego policies from a Git repository to the PostgreSQL database.

How It Works¶

┌─────────────────────┐      ┌─────────────────────┐      ┌─────────────────────┐
│ Git Repository      │      │ policy-sync         │      │ PostgreSQL          │
│                     │      │                     │      │                     │
│ policies/           │─────►│ Clone/Pull repo     │─────►│ policy database     │
│   alpha/            │      │ Parse .rego files   │      │ policies table      │
│   outbound/         │      │ Upsert to database  │      │                     │
│   ...               │      │ (runs every 1h)     │      │                     │
└─────────────────────┘      └─────────────────────┘      └─────────────────────┘

Configuration¶

Settings in customer-config.sh:

## Git repository containing policies (pre-configured with HIN Stargate policies)
POLICY_SYNC_REPO_URL="https://github.com/Health-Info-Net-AG/Stargate-policies.git"

## Optional: Authentication for private repos
POLICY_SYNC_REPO_USER=""
POLICY_SYNC_REPO_PASS=""

## Optional: Specific branch (default: main)
POLICY_SYNC_REPO_BRANCH=""

## Optional: Subfolder within repo containing policies
POLICY_SYNC_REPO_FOLDER=""

## Sync interval (default: 1h)
POLICY_SYNC_INTERVAL="1h"

Verification¶

## Check policy-sync status
docker logs stargate-policy-sync

## View synced policies
docker exec stargate-postgres psql -U postgres -d policy \
  -c "SELECT name, policy_group, filename, to_timestamp(updated_at) as updated FROM policies ORDER BY name;"

## View specific policy content
docker exec stargate-postgres psql -U postgres -d policy \
  -c "SELECT rego FROM policies WHERE name='deliveryStrategy' AND policy_group='alpha';"

Manual Trigger¶

To force an immediate sync:

docker restart stargate-policy-sync

Vault¶

Access Vault UI¶

Open http://localhost:8200
Use the root token from secrets/vault-keys.json or .env file

Vault Mounts¶

The following KV-v2 secret engines are created:

secret-smimekeys-client
secret-policy
secret-idagent
secret-mxengine

Manual Vault Operations¶

## Check status
docker exec stargate-vault vault status

## List mounts
docker exec -e VAULT_TOKEN=<token> stargate-vault vault secrets list

## Write a secret
docker exec -e VAULT_TOKEN=<token> stargate-vault vault kv put secret-smimekeys-client/test key=value

Databases¶

PostgreSQL databases created:

smimekeys_client
policy
idagent
mxengine

Connect to PostgreSQL¶

docker exec -it stargate-postgres psql -U postgres

## Or connect externally
psql -h localhost -U postgres -d smimekeys_client

Policies (Rego)¶

MXEngine uses OPA/Rego policies stored in PostgreSQL to determine mail delivery strategy.

Recommended: Use policy-sync to automatically sync policies from a Git repository. See Policy Sync section.

View Current Policy¶

## List all policies
docker exec stargate-postgres psql -U postgres -d policy \
  -c "SELECT id, name, policy_group, filename, to_timestamp(updated_at) as updated FROM policies;"

## View policy content
docker exec stargate-postgres psql -U postgres -d policy \
  -c "SELECT rego FROM policies WHERE name='deliveryStrategy';"

Policy Location¶

MXEngine config: POLICY_OUTBOUND: "outbound/delivery"
Database: policy database, policies table
Managed by: policy-sync service (syncs from Git repository)

Logs¶

## All services
docker compose logs -f

## Specific service
docker compose logs -f smimekeys-client
docker compose logs -f vault

Troubleshooting¶

Certificate issuance failed / WireGuard tunnel not established¶

This is the most common issue after initial installation. The S/MIME certificate cannot be issued because the WireGuard tunnel to the HIN CA is not established.

Symptoms:

onboard.sh shows: ⚠ Certificate issuance failed (WireGuard tunnel may not be established yet)
smimekeys-client logs show: issue certificate error: certcatunnel: error sending request: idagent: ... context deadline exceeded

Root causes (check in order):

Peer not registered on HIN CA - Your WireGuard public key must be registered on the HIN side. Provide HIN with:

# Get your WireGuard public key
docker compose logs idagent | grep "public key"

Along with your DEPLOYMENT_NAME, SERVER_STATIC_IP, and WG_INTERFACE_PORT (if changed from 19818).

Firewall blocking port 19818 - Ensure 19818/TCP is open both inbound and outbound on the Stargate server.
Wrong MAIL_HOSTNAME - If still set to mail.example.com (the template default), update it in customer-config.sh.

After the issue is resolved:

./scripts/onboard.sh --regenerate-cert

See Step 5: WireGuard Peer Registration for the full process.

Vault is sealed after restart¶

Run the start script which handles unsealing:

./scripts/start.sh

Cannot pull images¶

Login to the registry:

docker login registry.vereign.io

Service won't start¶

Check logs:

docker compose logs <service-name>

Reset everything¶

./scripts/purge.sh
./scripts/install.sh

Files Structure¶

stargate/
├── docker-compose.yml      # Main compose file
├── .env                    # Environment variables (generated by install.sh)
├── customer-config.sh      # Customer-specific settings (copied from customer-config.example)
├── customer-config.example # Template for customer configuration
├── README.md               # This file
├── config/
│   ├── vault/
│   │   └── vault.hcl       # Vault configuration
│   ├── postfix/
│   │   ├── Dockerfile       # Postfix relay container build
│   │   └── wrapper.sh       # Postfix entrypoint script
│   └── promtail/
│       └── promtail-config.yaml  # Promtail log shipping config
├── init/
│   └── postgres/
│       └── 01-create-databases.sql
├── scripts/
│   ├── install.sh              # First-time installation
│   ├── onboard.sh              # Domain onboarding (run after install or to add domains)
│   ├── start.sh                # Start services + unseal Vault
│   ├── stop.sh                 # Stop containers (preserves data)
│   ├── backup.sh               # Full backup (DB, Vault, config, certs)
│   ├── restore.sh              #  Restore from backup archive
│   ├── purge.sh                # Delete all data (destructive!)
│   ├── health-check.sh         # Comprehensive health check of all services
│   ├── init-vault.sh           # Vault initialization (used by vault-init container)
│   ├── init-idagent.sh         # WireGuard peer connection setup (used by idagent-init container)
│   └── gather-app-versions.sh  # Collects app versions for node-exporter metrics
├── secrets/                # Created on first run (gitignored)
│   ├── vault-keys.json     # Vault unseal keys (BACK THIS UP!)
│   └── signing-key.csr     # S/MIME certificate signing request
└── backups/                # Full backups (gitignored)
    └── *.tar.gz

Quick Health & Log Checks¶

Run the comprehensive health check:

./scripts/health-check.sh

## With verbose output (shows WireGuard details, liveness responses):
./scripts/health-check.sh -v

This checks:

All container statuses (running, healthy)
Liveness endpoints (smimekeys-client, policy, idagent, mxengine)
Vault seal status
PostgreSQL connectivity and all 4 databases
MinIO health
WireGuard tunnel status and peer handshakes
Postfix (running, port 25, port 10026, mail queue)
Prometheus metrics endpoints
Disk and memory usage

For manual log inspection:

# Check logs (last 10 lines)
docker logs stargate-smimekeys-client --tail 10
docker logs stargate-policy --tail 10
docker logs stargate-idagent --tail 10
docker logs stargate-mxengine --tail 10

# Follow logs in real-time
docker logs -f stargate-mxengine

# Check all container statuses
docker ps --format 'table {{.Names}}\t{{.Status}}'