Getting Started with Canary Checker
Canary checker is a monitoring system for executing synthetic tests, providing a built-in user interface, CLI and multi-cluster and multi-instance aggregation. Canary checker is designed with multi-tenancy in mind.
You are able to write your own tests and execute them to continually verify that your applications and clusters are working the way you expect.
In this guide, you'll see how to use canary-checker to test a Postgres database in several ways, using the CLI.
Prerequisities
For the purposes of this guide, you need a PostgreSQL instance running in Kubernetes. See the following guide on how to install PostgreSQL in your Kubernetes Cluster via Helm.
Installing the CLI
For this guide, Docker Desktop is used to create a Kubernetes cluster on MacOS. You can create your Cluster on the environment of your choice.
The canary-checker CLI will allow you to quickly and simply execute checks that you have defined, via a single CLI command.
Info
To install the CLI for preferred environment, See the Canary-checker Installation guide for more information.
To verify whether the CLI has been installed correctly, run canary-checker run -h from your terminal. You should see the following output:
Execute checks and return
Usage:
canary-checker run <canary.yaml> [flags]
Flags:
--csv output results in csv format
-d, --data string Template out each spec using the JSON or YAML data in this file
-h, --help help for run
-j, --junit output results in junit format
-n, --namespace string Namespace to run canary checks in
-o, --output-file string file to output the results in
Global Flags:
--db string Connection string for the postgres database (default "DB_URL")
--db-trace Trace database queries
--expose-env Expose environment variables for use in all templates. Note this has serious security implications with untrusted canaries
--json-logs Print logs in json format to stderr
--log-fail Log every failing check (default true)
--log-pass Log every passing check
-v, --loglevel count Increase logging level
--shared-library stringArray Add javascript files to be shared by all javascript templates
Creating a synthetic check for PostgreSQL
You'll define a check against our database that will connect to it, run a query against it and verify the results. Additionally, you'll see how canary-checker responds when the results are incorrect.
To get started, let’s created a directory to house our checks called postgres-canaries.
In that directory, create a file named postgres-canary-local.yaml, which will house the definition for our first check.
Add the following resource definition to our file. This is a Canary that will run the SELECT current_schemas(true) SQL query against our PostgreSQL instance every 30 seconds. It will also verify that the returned result is 1.
Replace the username and password with your PostgreSQL username and password, and save the file.
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: postgres-succeed
spec:
interval: 30
postgres:
- connection: "postgres://$(username):$(password)@127.0.0.1:5432/postgres?sslmode=disable"
name: postgres schemas check
auth:
username:
value: postgres
password:
value: yourpassword
query: SELECT current_schemas(true)
display:
template: |
{{- range $r := .results.rows }}
{{- $r.current_schemas}}
{{- end}}
results: 1
The above Canary runs the query against our database, which has been port forwarded to port 5432 running on our development machine.
Run the Canary using the following command:
canary-checker run postgres-canary-local.yaml
You should see that canary-checker ran the test that you defined, and validated the results:
2022-08-20T13:03:49.264+0200 INFO Checking postgres-canary-local.yaml, 1 checks found
2022-08-20T13:03:49+02:00 PASS [postgres] default/postgres-succeed/postgres schemas check duration=139 {pg_catalog,public}
2022-08-20T13:03:49.413+0200 INFO 1 passed, 0 failed in 154ms
Let’s modify the results field in our Canary definition to be 2 instead of 1, and run the check again.
2022-08-20T13:05:38.469+0200 INFO Checking postgres-canary-local.yaml, 1 checks found
2022-08-20T13:05:38+02:00 FAIL [postgres] default/postgres-succeed/postgres schemas check duration=130 {pg_catalog,public} Query return 1 rows, expected 2
2022-08-20T13:05:38.607+0200 INFO 0 passed, 1 failed in 145ms
The above check was a simple query that validated something that exists by default in a PostgreSQL database. However, a common use case for synthetic testing might be to validate the existence of some business-specific information.
Taking for instance there's a database table called Users, where you store the user information for your application.
You might want to run a Canary to validate whether a specific entry exists in the Users table.
To do this, create another Canary definition in our postgres-canaries directory called postgres-canary-local-does-admin-user-exist.yaml.
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: postgres-succeed
spec:
interval: 30
postgres:
- connection: "postgres://$(username):$(password)@127.0.0.1:5432/postgres?sslmode=disable"
name: postgres schemas check
auth:
username:
value: postgres
password:
value: yourpassword
query: SELECT * from Users
resultsFunction: '[[ if index .results 0 "username | eq "admin" ]]true[[else]]false[[end]]'
displayTemplate: '[[ index .results 0 ]]'
If you run your Canary right away, using the canary-checker run CLI command, you'll see that it fails, because you have not created the Users table yet.
canary-checker run ../postgres-canaries/postgres-canary-local-does-admin-user-exist.yaml
2022-09-08T13:18:31.547+0200 INFO Checking ../postgres-canaries/postgres-canary-local-does-admin-user-exist.yaml, 1 checks found
2022-09-08T13:18:31+02:00 FAIL [postgres] default/postgres-succeed/postgres schemas check duration=126 failed to query db: pq: relation "users" does not exist
2022-09-08T13:18:31.677+0200 INFO 0 passed, 1 failed in 134ms
Let’s create and insert the required data into our database with the following SQL.
CREATE TABLE users(
id int,
username varchar(200),
PRIMARY KEY(id)
);
insert into users (id, username) values (1, 'admin')
canary-checker run ../postgres-canaries/postgres-canary-local-does-admin-user-exist.yaml
2022-09-08T13:21:39.540+0200 INFO Checking ../postgres-canaries/postgres-canary-local-does-admin-user-exist.yaml, 1 checks found
2022-09-08T13:21:39+02:00 PASS [postgres] default/postgres-succeed/postgres schemas check duration=100
2022-09-08T13:21:39.647+0200 INFO 1 passed, 0 failed in 112ms
Installing canary-checker as a Kubernetes operator
So far, you've been running canary-checker using the CLI, but it's recommended you install it in your cluster and deploy a few Canaries with it.
To do this, you can use the operator for canary-checker.
From your terminal, run the following command to install canary-checker (Ensure that you have the prerequisites installed on your cluster first).
kubectl apply -f https://github.com/flanksource/canary-checker/releases/download/v0.38.154/release.yaml
Once the operator has been installed, you should be able to run kubectl get canary to see any canaries that you've deployed into our namespace.
To get started using the operator, let’s deploy a simple HTTP canary to our namespace.
Create a file called http_pass.yaml containing the below resource definition.
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
name: http-pass
spec:
interval: 30
http:
- endpoint: https://httpstat.us/200
thresholdMillis: 3000
responseCodes: [201, 200, 301]
responseContent: ""
maxSSLExpiry: 7`
You can then deploy this canary into our namespace using:
kubectl apply -f http_pass.yaml
canary.canaries.flanksource.com/http-pass created
You can then check the status of our canary by running:
TODO - Add the status
Wrapping up
In this guide, you've seen how to get started with canary-checker and run a few synthetic tests against PostgreSQL running in Kubernetes. You've also seen how you can deploy canary-checker as a Kubernetes operator and deploy a Canary into your Kubernetes cluster to continuously monitor your systems.
In the next guide, You'll take a look at how to model an application and Kubernetes cluster using SystemTemplates, as well as how to link components together - eventually linking components to canaries.