Standard Logging
Services should write logs using functions debug, error, warn, info exposed from @banno/node-template:
info('ExampleLogMessage', {
data: {
dataItem: 'one',
dataItem: 'two',
},
})
error('ExampleErrorMessage', {
error: err,
})
These logs will be exposed through Datadog (example) and help track down the execution of our code in Development, Staging, and Production. Check out the “Datadog” and “Datadog 101” pages in the Observability guides for more info.
NOTE: Debug logs will only be exposed in Development and Staging.
NOTE: Datadog logs cost money. Be judicious when adding new logs or adding data to logs.
You can also track logs in your local development while testing by changing the value of:
process.env.LOGGING_PRETTY_PRINT_LEVEL = 'silent'
to the log level you want in setup.js under /test/helpers.
Metrics
Services are monitored and alerted using Prometheus metrics. The metrics are normally provided on the first exposed port at a /metrics endpoint, but it can be configured to anything in the app’s Marathon config. See the old platform-server config for a simple example.
The healthChecks property in the config is used to alert the team to critical problems with the running service. Be sure to set the alert_channel label to auto-prism to send the alerts to our Slack channel. See the infrastructure docs for details.
Graphs of the metrics can be found in the Grafana dashboards:
Container Monitoring
You can see the services deployed and their configuration in the GKE dashboard. You can also use a command line tool like k9s.