Troubleshooting¶

Read this before trusting a diagnostic that returned nothing

Two properties of /var/log/messages on FreeBSD make a correct command print nothing at all and exit non-zero, which is indistinguishable from "the daemon never logged that": grep silently treats the whole file as binary if any program on the host wrote one non-ASCII byte, and newsyslog rotates the file about once an hour, so a daemon 80 minutes old already has its startup line in messages.0.bz2.

Reading the log first. Every other page here assumes it.

The CLI said…¶

Symptom
`Cannot connect to the SatL daemon at unix:///var/run/satl.sock. Is satld running?`	the daemon
`Permission denied` on the socket for a non-root user	the daemon
`storage preflight failed` / `ZFS root dataset … does not exist`	the daemon
`satld` exits after an edit to `satld.toml`	the daemon
`This node is not a swarm manager.`	the cluster
`cannot …: this cluster has no raft leader right now`	the cluster
`container … has already run and cannot be started again`	containers
`HostConfig.Privileged is not supported by SatL: …`	containers
`satl service ps <name>` prints an empty table and exits 0	the cluster
`root CA bundle does not match the join token`	TLS and joins
`malformed join token: …`	TLS and joins
`This node is already part of a swarm.`	TLS and joins
`node … has been removed from the cluster`	TLS and joins

The container…¶

Symptom
never leaves `Pending`; `ERROR` says `no suitable node (…)`	containers
is `REJECTED` naming the linuxulator or an init system	containers
exits immediately with empty logs	containers
ignores `--memory` / `--cpus`	containers
shows `(health: starting)` forever, then fails	containers
leaves its ZFS dataset behind after `satl rm`	containers
cannot pull its image	containers
resolves no names at all	node-local networking

The network on one node…¶

Symptom
the container reaches nothing outbound, but its published port answers	node-local networking
a published port refuses connections	node-local networking
`curl localhost:<port>` fails on the node that publishes it	node-local networking
one node answers correctly, then wrongly, in bursts	node-local networking
`epair`s or bridges are left behind after a failure	node-local networking

The network between nodes…¶

Symptom
a task cannot reach any remote task on the network	the overlay
one pair of tasks fails and everything else works	the overlay
everything works, throughput is poor, packet counts are doubled	the overlay
large transfers stall while pings answer	the overlay
a task loses the network some time after a configuration change	the overlay
a service name does not resolve, or resolves to the wrong service	the overlay
the first overlay network on a fresh host fails to program	the overlay

The cluster…¶

Symptom
`satl node ls` names a dead node `Leader`	the cluster
a node reads `Down`	the cluster
`satl service ls` says `8/6`	the cluster
a rolling update stopped halfway and says `paused`	the cluster
a crash-looping task stopped being replaced	the cluster
a drained node came back and stayed empty	the cluster
editing a node label moved running containers	the cluster
a worker refuses to start after its state was moved	the daemon
`satld` was killed and its containers kept running	the daemon
everything fails at once, some time after nothing changed	TLS and joins
`refused an internal TLS connection` on a manager	TLS and joins
a root CA rotation does not finish	TLS and joins

The log…¶

Symptom
a `grep` that should match prints nothing and exits 1	reading the log
the line you want is not there — "did this ever happen?"	reading the log
two timestamps, or two `{`, on one line	reading the log
`M-^` sequences in a line	reading the log
a background loop's span nested under another span	reading the log
the daemon started but the banner is full of warnings	the daemon

Not a failure, just a difference from Docker? See Differences from Docker. Ready to report something? What to collect first.