Troubleshooting & Diagnostics¶

A structured guide to diagnosing a Stargate appliance from the command line: what to check, where the logs are, and safe recovery actions.

Where the scripts are

The helper scripts live in the deployment directory under scripts/. Commands below use the full path for VM images: /usr/share/stargate-deployment/docker-compose/scripts/. If you installed elsewhere, substitute your own install directory (the folder that contains docker-compose.yml and scripts/).

docker compose ... commands must be run from the deployment directory:

cd /usr/share/stargate-deployment/docker-compose   # adjust to your install path

Where the writable state is

The docker-compose/ tree above (scripts, docker-compose.yml, config templates) is read-only at runtime. All writable state - .env, customer-config.sh, secrets/, generated TLS/Keycloak/APISIX config, backups, and every service's own data - lives under /var/data instead (STARGATE_DATA_DIR overrides this root for testing). In particular: config/secrets are under /var/data/vereign/ and backups under /var/data/backups/. See Data Layout for the full breakdown. Prefer the wrapper scripts (./scripts/start.sh, ./scripts/stop.sh, ...) over a bare docker compose up -d, which won't pick up /var/data/vereign/.env on its own.

1. Start here: the health check¶

One command summarizes the whole appliance:

QuickVerbose

/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh

/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh -v

It reports pass/fail for: containers (running/healthy), liveness endpoints (smimekeys, policy, irisagent, mxengine), Vault seal status, PostgreSQL connectivity + databases, SeaweedFS, the WireGuard tunnel + peer handshakes, Stalwart MTA (ports 25 / 10026), Prometheus metrics endpoints, and disk / memory.

Tip

Run this first. A single FAIL line usually points you straight at the section below.

2. Where the logs are¶

Layer	Command	What it shows
Boot / first-install / auto-start	`sudo journalctl -u stargate -n 200 --no-pager`	The systemd service that runs `start.sh` on boot and the first-boot install
Updates (VM appliance)	`hingateway-status`, `docker logs stargate-ops-agent --tail 100` and `journalctl -u greenboot-healthcheck`	Booted, staged and rollback OS versions; backup, download and errors of an update (only until the next reboot); result of the boot health check
`update.sh` runs	`cat /var/data/vereign/update.log`	Output of every `update.sh` run, newest at the end: plain Compose updates, dashboard updates on older pre-bootc VMs, and runs after changing `customer-config.sh`
A single service	`docker logs stargate-<service> --tail 100`	e.g. `stargate-dashboard`, `stargate-mxengine`, `stargate-keycloak`
Follow one service live	`docker logs -f stargate-mxengine`	Real-time
All containers live	`docker ps -a --format '{{.Names}}' \\| xargs -I{} sh -c 'docker logs --timestamps -f {} 2>&1 \\| sed "s/^/[{}] /"'`	Merged, prefixed by container
Web log viewer	Dozzle at `https://<SERVER_IP>:8190` (Keycloak login)	Browse all container logs in a UI

To hand logs to HIN support, use the upload script: it collects the last N log lines from every service (plus host and version info), uploads them, and prints a link to share - see Provide logs to support:

/usr/share/stargate-deployment/docker-compose/scripts/send-logs-to-support.sh --tail 5000     # last 5000 lines from each service
# other options:  --since 1h   |   --until 5m   |   --all   (no argument = --tail 500)

The upload is capped at 20 MB, so on a busy appliance prefer --tail/--since over --all.

3. Containers not running or restarting¶

docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Read the Status column:

Status	Meaning	Action
`Up ... (healthy)`	Running fine	-
`Up ...` (no health)	Running; no healthcheck defined	Check its `docker logs` if you suspect trouble
`Restarting`	Crash-looping	`docker logs stargate-<svc>` - fix the root error (config, secret, dependency)
`Exited (0)`	One-shot init finished OK (e.g. `*-init`, `vault-data-fixer`)	Normal
`Exited (1+)`	Failed	`docker logs stargate-<svc>` - the last lines show why
`Created`	Never started - a dependency didn't come up	Check what it `depends_on` (usually Postgres/Vault); fix that first

Restart a single service (safe, non-destructive):

docker compose up -d <service>          # recreate one service
docker compose restart <service>        # just restart it

Startup order

Services wait on their dependencies (depends_on + healthchecks). During a full restart, brief connection refused / database system is starting up lines while Postgres/Vault come up are normal and clear within a minute.

4. Diagnose by symptom¶

Dashboard or Keycloak won't load / can't log in¶

Both are fronted by Caddy: Dashboard on :443, Keycloak on :8180.
Check the chain: docker logs stargate-caddy, stargate-dashboard, stargate-keycloak, stargate-apisix.
Keycloak must be healthy before the dashboard works: docker compose ps keycloak.
TLS warning in the browser is expected (self-signed cert) - accept and proceed.
Login redirects failing usually mean the public URL doesn't match how you reach the box - verify KEYCLOAK_PUBLIC_URL / DASHBOARD_PUBLIC_URL in .env point at the IP/host you actually use.

WireGuard tunnel down / certificate issuance fails¶

This is the most common issue - certificates fail when the tunnel is down, so always fix the tunnel first.

/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh -v      # shows WireGuard peer + handshake status
docker logs stargate-irisagent | grep -iE "handshake|peer|cert|wireguard"

Confirm the firewall allows 19818 (UDP and TCP) inbound/outbound.
Verify the peer is registered on the HIN side (support step) - you supply WG public key, DEPLOYMENT_NAME, SERVER_STATIC_IP, WG_INTERFACE_PORT.
Once the tunnel shows a recent handshake, retry certificate issuance from the dashboard.

Vault sealed or init failed¶

docker compose exec vault vault status        # look for "Sealed: false"
docker logs stargate-vault-init

Vault must be unsealed for smimekeys/mxengine/policy to work. Keys live in /var/data/vereign/secrets/vault-keys.json.
If vault-init exited non-zero, the keys file may be missing/corrupt - check its logs; re-running /usr/share/stargate-deployment/docker-compose/scripts/init-vault.sh re-attempts unseal.

Do not delete /var/data/vereign/secrets/vault-keys.json

Losing it means losing access to all stored secrets. Keep a backup.

PostgreSQL / database connectivity¶

docker compose exec postgres pg_isready -U postgres
docker logs stargate-postgres --tail 50

Transient the database system is starting up (57P03) right after a restart is normal - services reconnect automatically.
Persistent auth failures usually mean POSTGRES_PASSWORD in .env drifted from the data volume - see the update/secrets notes, and avoid editing it by hand.

Mail not flowing¶

Inbound arrives on :25 (Stalwart). Many cloud providers block port 25 by default:
```
nc -zv <this-server-ip> 25          # from an external host
docker logs stargate-stalwart --tail 100
```
If 25 is blocked, request an exception from your provider. - Outbound / sealing goes Stalwart → mxengine (:8084 HTTP, SMTP :1587): docker logs stargate-mxengine. - Mail loops show as the same message cycling - check that your domain's MX does not resolve back to this appliance's own IP. - See Mail relay setup and DNS setup for the expected routing.

An update failed¶

On the VM appliance, an update from the dashboard is run by the ops-agent in three phases: a backup (backup.sh), the download of the target OS image (bootc switch), and the reboot into it. Start with:

docker logs stargate-ops-agent --tail 100     # the update orchestrator: backup, download, errors
hingateway-status                             # booted, staged and rollback OS versions

The ops-agent container is recreated on every boot, so its log covers an update only until the VM restarts. After a reboot, use hingateway-status and journalctl -u greenboot-healthcheck instead.

Failed while backing up. The ops-agent cancels the update when the pre-update backup fails, so nothing has been changed. Check the error in the ops-agent log and the free space on the Data Disk (df -h /var/data), then start the update again.
Failed while downloading. The dashboard's update status shows the error as bootc switch: ...; the ops-agent log only records os select failed with the exit status. Usually the VM cannot reach the image registry (check outbound internet access and DNS) or the OS disk is full (df -h /sysroot). The running version is not affected; start the update again once the cause is fixed.
Downloaded, but the version did not change. The new version is activated only by a reboot. If you chose Later, restart the VM (sudo systemctl reboot) and check hingateway-status again. If the VM did reboot and still shows the old version as booted, it rolled back automatically - see the next item.
Automatic rollback after a failed boot. After booting a new version, health checks run - among others, whether Docker is running, the Data Disk is mounted and the Stargate stack is healthy, plus the operating system's default checks. If a required check fails, the boot is marked RED and the VM boots the previous version again on its own. journalctl -u greenboot-healthcheck shows which check failed. Fix the cause before you try the update again; rebooting alone repeats the same rollback. If the new version had already changed a database schema, or the failed update was the first upgrade onto an image with Data Disk support (the stack then shows an empty /var/data, but your data is untouched), read the rollback warnings before you do anything else.
The new version boots but is unhealthy. Check the boot health check with journalctl -u greenboot-healthcheck and the stack with /usr/share/stargate-deployment/docker-compose/scripts/health-check.sh -v. To return to the previous version, use Roll back on the dashboard or run hingateway-rollback && sudo systemctl reboot - read the rollback warnings first.
After a successful update, confirm the application versions: /usr/share/stargate-deployment/docker-compose/scripts/gather-app-versions.sh (or check docker compose ps image tags).
If a single service is stuck after an update, recreate only that service: cd /usr/share/stargate-deployment/docker-compose && sudo docker compose --env-file /var/data/vereign/.env up -d --no-deps --force-recreate <service>. sudo is needed because .env is readable only by root, and --no-deps keeps Compose from also recreating the service's dependencies, such as Vault, which would come back sealed. If /var/data/vereign/docker-compose.override.yml (or .yaml) exists, add -f docker-compose.yml -f <override file> after --env-file /var/data/vereign/.env so the override stays applied. Services that belong to a profile need it as well: add --profile dozzle for dozzle and oauth2-proxy, --profile keri-idagent for idagent and --profile keri-watcher for watcher.

On a plain Compose install (a git clone you manage yourself) there is no ops-agent update: you run update.sh yourself, and its output is appended to /var/data/vereign/update.log, newest run at the end. See Plain Compose install for the update steps.

Warning

Do not try to repair an update on the VM appliance with git or update.sh --release. /usr/share/stargate-deployment is part of the read-only OS image and there is no repository checkout to reset; versions change only by switching the OS image.

Older VMs (before the bootc image)¶

VMs installed from an image older than the bootc appliance have a git checkout under /usr/share/stargate-deployment instead of a read-only OS image. The VM appliance steps above do not apply to them. The long-term fix is to move them to a bootc appliance as described in Migrating a Compose Deployment to a bootc Appliance. Until then, the following applies to these older VMs only. On them the ops-agent runs update.sh, whose output is appended to /var/data/vereign/update.log (newest run at the end) - check it first when an update fails.

Update starts but nothing happens (updating from an older version). If the ops-agent log stops at pulling deployment repo ... and the update never proceeds, the repository on the VM most likely has local edits to a tracked file (commonly a hand-patched docker-compose.yml). That makes the ops-agent's git checkout refuse to run, so the update stalls. Force-reset the repository to the latest revision, then re-run the update. Git is the single source of truth; this discards local edits to tracked files only - customer-config.sh, .env, and secrets/ live under /var/data/vereign/, outside the repository checkout, and are always preserved:

cd /usr/share/stargate-deployment
sudo git fetch origin
sudo git checkout -f main
sudo git reset --hard origin/main
cd docker-compose
sudo /usr/share/stargate-deployment/docker-compose/scripts/update.sh

update.sh regenerates .env, pulls the images, and recreates the affected services. If the release changed configuration files, run sudo /usr/share/stargate-deployment/docker-compose/scripts/start.sh afterwards so every container picks up the new files. When it completes, retry the update from the dashboard; it will now proceed.

Warning

Do not use git pull here. On a working tree with local edits it aborts with "local changes would be overwritten", which forces a git stash / merge-conflict / manual-recovery detour. The git checkout -f + git reset --hard sequence above avoids that entirely and is the safe, repeatable way to bring the repo current.

VMs on v0.5.1¶

The following steps update a VM from v0.5.1 to v0.5.3. On v0.5.x, .env, customer-config.sh and secrets/ are inside the checkout in /usr/share/stargate-deployment/docker-compose/, not under /var/data/vereign/. Log in to the VM with the Linux administrator account.

Edit /usr/share/stargate-deployment/docker-compose/.env and set the ops-agent version to v0.0.3.
Edit /usr/share/stargate-deployment/docker-compose/customer-config.sh and set the ops-agent version to v0.0.3 there as well.
Fetch the release tags: cd /usr/share/stargate-deployment && sudo git fetch --tags origin
Check out the target release: sudo git checkout v0.5.3. Do not check out main: it contains the current scripts, which expect the newer /var/data layout.
Update the ops-agent container: cd docker-compose && sudo docker compose up -d ops-agent
Log in to the Dashboard.
Navigate to Settings.
In the Update section at the bottom of the page, enter the target version (v0.5.3) and start the update process.

After that, move the VM to a bootc appliance as described in Migrating a Compose Deployment to a bootc Appliance.

Dozzle (log viewer) not reachable¶

URL is https://<SERVER_IP>:8190; it requires a Keycloak login (same realm as the dashboard) via oauth2-proxy.
It only runs when DOZZLE_ENABLED="true". Check: docker compose ps dozzle oauth2-proxy.
Ensure the firewall allows :8190 inbound. See Monitoring and Logs.

Onboarding: entering the activation code returns an error¶

If the activation code is rejected, check in order:

Wrong code - it wasn't copied in full (a truncated copy-paste, an extra space, or a missing character). Re-copy the complete code and re-enter it.
Code already used - it was already consumed by a previous onboarding. Request a fresh code.
WireGuard / connectivity to HIN - the irisagent tunnel isn't up, so the code can't be validated against HIN. See WireGuard tunnel down above (/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh -v, docker logs stargate-irisagent).
No domain tied to the registration - the customer's HIN registration has no domain associated with it, so there is nothing to activate. This is resolved on the HIN side.

Onboarding: activation code accepted, but no domains are listed¶

Most likely cause: the WireGuard tunnel is not established - usually a wrong public IP or a firewall port not open. Without the tunnel the appliance cannot fetch the domain list from HIN.

Verify SERVER_STATIC_IP in customer-config.sh matches the real public IP.
Confirm 19818 (UDP and TCP) is open inbound/outbound.
Confirm the peer is registered on the HIN side for this IP (support step).
docker logs stargate-irisagent should show a recent handshake; if not, fix the tunnel first (see WireGuard tunnel down above).

Changing the server IP address (initial setup only)¶

If the server IP was wrong or unset at first boot, reset cleanly and reinstall. Save the configuration first, because purge.sh also deletes customer-config.sh:

/usr/share/stargate-deployment/docker-compose/scripts/backup.sh --config-only  # saves customer-config.sh
/usr/share/stargate-deployment/docker-compose/scripts/purge.sh                 # destroys ALL data - see warning below
/usr/share/stargate-deployment/docker-compose/scripts/restore.sh --config-only /var/data/backups/<timestamp>_config-only.tar.gz
nano /var/data/vereign/customer-config.sh    # set SERVER_STATIC_IP=<NEW IP>
reboot                                       # installs again with the new IP

backup.sh prints the exact restore command, and the archive stays in /var/data/backups, which purge.sh does not delete. The reboot runs the installation and also reloads the locally built images that purge.sh removes. On a plain Docker installation, run install.sh instead of rebooting.

The TLS certificate and several service URLs are derived from the IP at first boot, so a purge + reinstall regenerates them for the new address. The restored file keeps the generated passwords and the WireGuard key. If the gateway is already registered at HIN, the registration has to be updated with the new IP.

Only before onboarding

purge.sh permanently deletes all data - databases, Vault, and the S/MIME keys. This is safe only on a fresh, not-yet-onboarded appliance. Never run purge.sh to change the IP of a live/onboarded gateway - it causes data loss and mail that can no longer be decrypted. For a production IP change, contact support.

5. Storage & disk¶

df -h /                              # is the disk full?
docker system df                     # space used by images / containers / volumes
du -sh /var/lib/docker/volumes/*     # per-volume usage (Postgres, SeaweedFS, Loki, ...)

Container logs are capped (json-file, 100 MB × 5 per container) so they shouldn't fill the disk, but images and volumes can.
Reclaim space safely: docker image prune -af (removes unused images only). Avoid docker system prune --volumes - it deletes data volumes.
Object storage is SeaweedFS (stargate-seaweedfs): docker logs stargate-seaweedfs --tail 50.

6. VM resources¶

free -h                              # memory (min 8 GB)
nproc                                # CPUs (min 4)
docker stats --no-stream             # per-container CPU/RAM
uptime                               # load average

Host metrics are also exported for Prometheus on :9100/metrics (see Monitoring). If the box is swapping or pegged, expect healthchecks to flap and updates to be slow.

7. Network & ports¶

Quick reachability check for the key inbound ports:

for p in 25 443 8180 8190 19818; do nc -zv <this-server-ip> $p; done

Port	Service	Direction
`25`	Stalwart SMTP (inbound mail)	inbound
`443`	Dashboard (HTTPS)	inbound
`8180`	Keycloak	inbound
`8190`	Dozzle (optional)	inbound
`19818`	WireGuard (UDP and TCP)	in/outbound

Outbound access is needed to the container registry, the S/MIME CA (over the WireGuard tunnel), and any remote Loki you configured. See the full port table on the home page and Applications overview.

The Docker network overlaps the local network¶

The services talk to each other on an internal Docker network. Docker picks its address range when the stack is created, and avoids the networks the VM is directly connected to at that moment. If the range still contains addresses from your network, the appliance cannot reach those machines (for example a mail or DNS server), and they cannot reach it. The health check reports this under Docker network:

[FAIL] stargate-network 172.18.0.0/16 overlaps host routes: 172.18.5.0/24 dev ens18 - see Troubleshooting: The Docker network overlaps the local network

The IP was changed after the first boot - Docker chose the range while the VM still had its first address. On a gateway that is not onboarded yet, use the purge + reinstall from Changing the server IP address above; Docker then picks a free range by itself. On an onboarded gateway, recreate the network as shown below.
The overlapping network is behind a router (a static route, or a network at another site) - Docker does not avoid such networks. The health check sees every route on the host, but not networks reached through the default gateway. First give Docker a free range: add this to /etc/docker/daemon.json, keeping any settings already in the file, and use a range your network does not use:
```
{
  "default-address-pools": [
    { "base": "10.200.0.0/16", "size": 24 }
  ]
}
```
Then run systemctl restart docker and recreate the network as shown below.

Recreating the network keeps all data in /var/data. Stop the service first, otherwise a running Dozzle keeps the old network in use:

systemctl stop stargate
cd /usr/share/stargate-deployment/docker-compose
docker compose --env-file /var/data/vereign/.env down
systemctl start stargate
/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh

8. Recovery actions¶

Ordered least- to most-disruptive:

docker compose up -d <service>       # recreate one stuck service
sudo systemctl restart stargate      # restart the whole stack (via start.sh)
/usr/share/stargate-deployment/docker-compose/scripts/stop.sh  &&  /usr/share/stargate-deployment/docker-compose/scripts/start.sh

Backups & destructive recovery

/usr/share/stargate-deployment/docker-compose/scripts/backup.sh and /usr/share/stargate-deployment/docker-compose/scripts/restore.sh handle data backup/restore. /usr/share/stargate-deployment/docker-compose/scripts/purge.sh deletes all data (databases, Vault, storage) for a clean reinstall - use only as a last resort and only with a current backup. Details: Docker Advanced configuration.

9. When to contact support¶

If the health check still shows failures after the steps above, open a ticket via Support / Contact us and include:

The appliance version (/usr/share/stargate-deployment/docker-compose/scripts/gather-app-versions.sh) and customer name.
The health-check output (/usr/share/stargate-deployment/docker-compose/scripts/health-check.sh -v).
A log bundle link from /usr/share/stargate-deployment/docker-compose/scripts/send-logs-to-support.sh (see Provide logs to support).
What you were doing when it broke, and any screenshots.

Setup updated keycloak¶

Note: This instructions is valid if you are running on VM image v0.5.1 and then you updated to newer version.

After the latest Keycloak update, a breaking change causes authenticated users to be unexpectedly redirected to the login page when navigating to specific application routes (e.g., Peers, Peer Certificates).

To resolve this, the following manual configuration must be completed in the Keycloak UI.

Resolution Steps¶

Open Keycloak in your browser at https://<VM IP address>:8180/admin/master/console/
- The :8180 port is required - Keycloak is served on port 8180. Opening the IP with no port reaches the dashboard (:443) instead, which redirects you to the stargate-realm login where the admin user does not exist (this is the usual cause of "invalid username or password" / "wrong realm" here).
- Log in against the master realm (the /admin/master/console/ path selects it) with username admin (the KEYCLOAK_ADMIN_USER value - lowercase) and the KEYCLOAK_ADMIN_PASSWORD value from the machine's /var/data/vereign/.env (read it from the Linux console).
In the admin console, switch the realm selector (top-left) from master to stargate:
Go to Clients → dashboard
Navigate to the Client scopes tab → click dashboard-dedicated
Select Configure a new mapper → Audience
Set the following configurations:
- Name: apisix-audience
- Included client audience: apisix (select from dropdown)
- Included custom audience: (leave empty)
- Add to access token: On
- Add to token introspection: On
- Add to ID token / lightweight token: Off
Click Save

Resetting a Stargate User Password in Keycloak¶

Follow the steps below to reset the password of a Stargate user through the Keycloak administration console.

Connect to the HIN Gateway VM
Open the VM console, or connect to the HIN Gateway VM via SSH.
Retrieve the Keycloak administrator credentials
Open the /var/data/vereign/.env file
Locate the following variables:
- KEYCLOAK_ADMIN_USER
- KEYCLOAK_ADMIN_PASSWORD
Open the Keycloak Admin Console
In a browser, navigate to: http://<VM-IP>:8180/admin/master/console
Replace <VM-IP> with the IP address of the HIN Gateway VM.
Sign in to Keycloak
Enter the administrator username and password retrieved from the .env file.
keep the same administrator password if you change it you must store it securely
Select the Stargate realm
From the realm selector in the Keycloak Admin Console, select Stargate.
Find the user
Navigate to Users.
Search for and select the Stargate user whose password needs to be reset.
Reset the user's password
Select the option to reset the user's password.
Enter the new password and confirm the change.

Do not change or reset the Keycloak administrator password as part of this procedure. Any changes of admin password can lead to potention lockdown of Keycloak without anyone be able to access it!

Request Activation code¶

In case you want to install or re-install HIN Gateway instance you will need Activation code , which can be provided by HIN Support

Reset Activation code breaks peer connection

Resetting the activation code for an active instance clears the customer WireGuard peer connection.

If you request this reset for the wrong customer instance, it will delete and break that active instance connection. Before requesting a reset, double-check the exact instance and confirm all domains hosted on it.

Reboot on VM v0.6.1¶

Please keep in mind if you do a REBOOT an instance which is running on v0.6.1 it will revert back to the previous non working Stalwart version <0.0.23.

We recommend to update the instance to newer version from Settings->System version section!