Skip to main content
Version: 1.x

Host Troubleshooting

Using logs​

The first place to check when troubleshooting the host is logs. wasmCloud host logs include the following:

  1. Logs from the host runtime
  2. Logs from components
  3. Logs from providers

Finding the logs​

If you followed the installation guide, then you likely started the host by running wash up. By default, the logs will be printed to the screen, but if you ran the host using --detached, the logs will be written to ~/.wash/downloads/wasmcloud.log.

At the start of this logfile you'll see something like:

2023-12-19T21:36:49.708714Z  INFO async_nats::options: event: connected
2023-12-19T21:36:49.708910Z  INFO async_nats::options: event: connected
2023-12-19T21:36:49.711351Z  INFO wasmcloud_host::wasmbus: bucket already exists. Skipping creation. bucket=LATTICEDATA_default
2023-12-19T21:36:49.712048Z  INFO wasmcloud_host::wasmbus: bucket already exists. Skipping creation. bucket=CONFIGDATA_default
2023-12-19T21:36:49.714279Z  INFO wasmcloud_host::wasmbus: wasmCloud host started host_id="NBNHZROISSPTB4U77L53USB47S5A7HWETPCVAENGYLTWHMKR2B7Q37VT"

Logs are appended to the end of this file as they are generated, so for the latest logs, you'll want to check the end of the file. On Unix systems, you can use the tail command to see the last several lines of the file, and you can use tail -f to stream new logs as they are written.

Changing the log level​

To configure the log level, you can use the --log-level flag, or set the WASMCLOUD_LOG_LEVEL environment variable, when starting the host.

The wasmCloud host supports the following log levels:

  • error
  • warn
  • info (default)
  • debug
  • trace

If you want to configure the log level for a specific library, you can set the RUST_LOG environment variable. For example, to change the log level of async-nats to warn while keeping the host at debug, you could run:

bash
RUST_LOG=async_nats=warn,debug wasmcloud --log-level debug
info

Note that all components and providers inherit the same log level settings as the host itself.

Log format​

By default, the wasmCloud host emits logs as unstructured text. However, structured logs can be enabled via the --enable-structured-logging flag, or by setting the WASMCLOUD_STRUCTURED_LOGGING_ENABLED environment variable to true. Structured logs are convenient when forwarding logs to an aggregation tool.

Structured logs look like the following:

{"timestamp":"2023-12-19T22:16:06.389492Z","level":"INFO","fields":{"message":"event: connected"},"target":"async_nats::options"}
{"timestamp":"2023-12-19T22:16:06.390689Z","level":"INFO","fields":{"message":"event: connected"},"target":"async_nats::options"}
{"timestamp":"2023-12-19T22:16:06.401737Z","level":"INFO","fields":{"message":"bucket already exists. Skipping creation.","bucket":"LATTICEDATA_default"},"target":"wasmcloud_host::wasmbus"}
{"timestamp":"2023-12-19T22:16:06.402034Z","level":"INFO","fields":{"message":"bucket already exists. Skipping creation.","bucket":"CONFIGDATA_default"},"target":"wasmcloud_host::wasmbus"}
{"timestamp":"2023-12-19T22:16:06.406203Z","level":"INFO","fields":{"message":"wasmCloud host started","host_id":"NC3CUT4IP43REAEQDE33652YNK4Z7KFE7KI3XGUMHVNSP2TNY3CFAH4M"},"target":"wasmcloud_host::wasmbus"}

Cleaning the slate​

In the course of troubleshooting, you may wish to completely clear your existing wasmCloud environment and start with a fresh installation. You can follow these steps to do so, but note that the process will remove all existing data associated with wasmCloud including application models stored on the lattice.

Drain and uninstall wash​

Run wash drain to clear caches (including OCI artifacts and provider binaries) and then uninstall wash in the same way you installed it. (The example below assumes an install via Cargo.)

shell
wash drain
# Uninstall wash the same way you installed it
cargo uninstall wash-cli
# Ensure the wash directory is clear
rm -r ~/.wash

Clear NATS streams and buckets​

Several streams and key-value buckets used by wasmCloud persist on NATS even if wash is uninstalled, since the model store is decoupled from any given host.

You can interact directly with NATS using the NATS CLI. Find instructions to install the CLI on the NATS CLI GitHub repo.

You can list all streams on your NATS instance with:

shell
nats stream list -a

The -a flag includes system streams in the list. The listed streams will include:

  • wadm_commands
  • wadm_events
  • wadm_mirror
  • wadm_notify
  • KV_CONFIGDATA_default
  • KV_wadm_manifests
  • KV_wadm_state
  • wadm_status
  • KV_LATTICEDATA_default

To delete a NATS stream:

shell
nats stream del STREAM_NAME

For example: nats stream del wadm_status

You can list all key-value buckets on your NATS instance with:

shell
nats kv list

The listed key-value buckets will include:

  • CONFIGDATA_default
  • wadm_manifests
  • wadm_state
  • LATTICEDATA_default

To delete a NATS key-value bucket:

shell
nats kv del BUCKET_NAME CONTEXT_NAME

Unless you've added or edited a context, the CONTEXT_NAME value is host_default.

For example: nats kv del wadm_state host_default

For a total reset, delete all buckets and streams. These buckets and streams will be automatically regenerated when you run wash up from your new wash installation.

Reinstall wash​

Install wash again according to the instructions for your installation method.