Write Integration Tests with Ephemeral Resources using Ory Dockertest
You have probably encountered this before. You are writing a function that relies on a third-party service (e.g. PostgreSQL) and as every good developer, you would like to verify the correctness of your code with some tests!
func main() {
db, err := sql.Open("mysql",
"user:password@tcp(127.0.0.1:3306)/hello")
if err != nil {
log.Fatal(err)
}
defer db.Close()
var (
id int
name string
)
rows, err := db.Query("select id, name from users where id = ?", 1)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&id, &name)
if err != nil {
log.Fatal(err)
}
log.Println(id, name)
}
err = rows.Err()
if err != nil {
log.Fatal(err)
}
}
Now, there are several strategies to do test your program. The most common advice you will find is to use mocking and Golang has an excellent tool for it called gomock. However, at Ory we found that mocking is not very useful when testing third-party services because it primarily helps to verify the command flow of your code - what happens on an error? Or if the code returns an empty slice?
With third-party services, you want to test the correctness of the integration:
- Is your SQL statement correct?
- Does the table exist?
- Does the column update work as you expect?
- Was the key/value pair deleted in Redis?
Let us explore an efficient solution to this problem, utilizing Docker and Ory’s Dockertest to spin up real, short-living databases and third-party integrations.
Refresh your Testing Knowledge
Before we go ahead, let’s take a refresher on the different archetypes of testing. If testing is second nature to you, feel free to skip to the next section!
Unit Tests
A “unit” is the smallest section of an application that can be isolated for testing. Unit tests make sure each unit meets its design and behaves as intended. If you are building an application that grows over time and is adding new features or plugins, you want to make sure it still works when you make changes to existing functionality. A minimal test example for unit tests might look as follows:
package main
import "testing"
func Sum(x int, y int) int {
return x + y
}
func TestSum(t *testing.T) {
total := Sum(5, 5)
if total != 10 {
t.Errorf("Sum was incorrect, got: %d, want: %d.", total, 10)
}
}
However, unit tests are not useful when testing integrations with external services such as databases. Let’s take a look at a pseudocode example:
package main
import (
"database/sql"
"log"
)
type Storage interface {
GetUserName(id int) (string, error)
}
type SQLStorage struct {
db *sql.DB
}
func NewSQLStorage() *SQLStorage {
db, _ := sql.Open("mysql", "user:password@tcp(127.0.0.1:3306)/hello")
// ...
return &SQLStorage{db: db}
}
func (s *SQLStorage) GetUserName(id int) (string, error) {
var name string
rows, err := s.db.Query("select name from users where id = ?", id)
if err != nil {
return "", err
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&name)
if err != nil {
return "", err
}
return name, nil
}
return "", rows.Err()
}
While it is easy to mock Storage.GetUserName(). Mocking however will not help in testing whether SQLStorage.GetUserName's SQL code actually works! That’s why we need integration tests.
Integration Tests
Integration testing verifies that different parts of the system work together and talk to each other. You are testing how a code snippet depends on another to run and pass a value to it. Integration tests are more complicated than unit tests because they require you to set up a mock environment where all parts of your application can be tested together. Also, developers can make mistakes when producing test responses from a mock database. A better way is to spin up real instances of database & third-party services through Docker and get real responses from a real system: You are checking whether your code integrates correctly with other parts of your architecture!
Other Testing Approaches
Even though we focus on integration tests in this article, let’s quickly look at the remaining two test approaches as well:
Functional Tests
Functional or system testing tests the function of a system as a black box. It only looks at what the system does while being ignorant of its inner workings. This type of testing checks if all the features are implemented and work as expected. A functional test would answer e.g. whether registration or login works!
End-to-End Tests
End-to-end tests test the application flow from start to end, ensuring the function of all sub-systems, network dependencies, services, and databases— including third-party service providers. E2E tests are best conducted in an environment as close to production as possible. This can be done by creating a real duplicate of your database and third-party integrations. You would test full end-to-end user experiences - from signing in, to adding a product to the shopping cart, to checking out.
Enter Ory’s Dockertest
Tests are the best way to increase the trust you have in your code. Integration tests are important, but they are also hard to set up. You need to start a database, check that all the code works together, and then tear it down again. Docker (or rather virtualization) is a useful tool for these scenarios. Create a short-living instance of your external dependency (e.g. MySQL) without the need for network connectivity or access to a production or staging system. Using virtualization, you can easily spin up these dependencies in your CI/CD system to test your code before you ship it.
A common concept is to set up Docker Compose configuration files and start the database before you run your test. This can work, but has some drawbacks:
- You can not easily test parts of your code in isolation - the database runs “globally” for your tests;
- You need to remember to clean up the databases after your test runs;
- On test failures it is common for the test pipeline to exit early, leaving unused databases in your Docker environment.
Ory’s Dockertest solves these issues by allowing you to start any Docker Image on demand from your code, and also tears them down on program exit!
Installing Ory’s Dockertest is straightforward. Run in your Golang project’s root:
go get github.com/ory/dockertest/v3
There are a few core concepts in Ory’s Dockertest. First, you have the concept of a pool. Please note that we are using Go’s testing.Main function to set up Ory’s Dockertest. This allows us to clean up any stale containers after the tests are completed - regardless of the test outcome!
Let’s start by creating a new Dockertest pool:
package foobar
import (
"log"
"testing"
"github.com/ory/dockertest/v3"
)
func TestMain(m *testing.M) {
// uses a sensible default on windows (tcp/http) and linux/osx (socket)
pool, err := dockertest.NewPool("")
if err != nil {
log.Fatalf("Could not connect to docker: %s", err)
}
}
The pool ensures that the connection to Docker is alive and well. If your Docker Daemon is not running, this will fail! If you have trouble connecting Ory’s Dockertest to your Docker environment, check out the troubleshoot section.
Once connected, just start the image you need (here MySQL v5.7) using:
// pulls an image, creates a container based on it and runs it
resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
if err != nil {
log.Fatalf("Could not start resource: %s", err)
}
The container will need some time to come alive! This might take longer or shorter and depends on a few factors:
- Is the image already downloaded and available on your machine?
- How long the software takes to start and become available - databases usually take a bit longer to come alive.
Ory’s Dockertest has an exponential backoff retry convenience helper which you can use to reconnecting to the dependencies until it comes live:
// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
if err := pool.Retry(func() error {
var err error
db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
if err != nil {
return err
}
return db.Ping()
}); err != nil {
log.Fatalf("Could not connect to database: %s", err)
}
To clean up the container after your test has finished, use the following code:
// Runs your tests
code := m.Run()
// You can't defer this because os.Exit doesn't care for defer
if err := pool.Purge(resource); err != nil {
log.Fatalf("Could not purge resource: %s", err)
}
// Exits
os.Exit(code)
}
Putting it all together:
package foobar
import (
"database/sql"
"fmt"
"log"
"os"
"testing"
_ "github.com/go-sql-driver/mysql"
"github.com/ory/dockertest/v3"
)
var db *sql.DB
func TestMain(m *testing.M) {
// uses a sensible default on windows (tcp/http) and linux/osx (socket)
pool, err := dockertest.NewPool("")
if err != nil {
log.Fatalf("Could not connect to docker: %s", err)
}
// pulls an image, creates a container based on it and runs it
resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
if err != nil {
log.Fatalf("Could not start resource: %s", err)
}
// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
if err := pool.Retry(func() error {
var err error
db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
if err != nil {
return err
}
return db.Ping()
}); err != nil {
log.Fatalf("Could not connect to database: %s", err)
}
code := m.Run()
// You can't defer this because os.Exit doesn't care for defer
if err := pool.Purge(resource); err != nil {
log.Fatalf("Could not purge resource: %s", err)
}
os.Exit(code)
}
func TestSomething(t *testing.T) {
// db.Query()
}
Starting and Connecting to an Ephemeral PostgreSQL Instance
Starting a PostgreSQL Container follows the same principles we already established. It is also possible to define additional configuration which helps with clean up and retries.
resource, err := pool.RunWithOptions(&dockertest.RunOptions{
Repository: "postgres",
Tag: "11",
Env: []string{
"POSTGRES_PASSWORD=secret",
"POSTGRES_USER=user_name",
"POSTGRES_DB=dbname",
"listen_addresses = '*'",
},
}, func(config *docker.HostConfig) {
// set AutoRemove to true so that stopped container goes away by itself
config.AutoRemove = true
config.RestartPolicy = docker.RestartPolicy{Name: "no"}
})
if err != nil {
log.Fatalf("Could not start resource: %s", err)
}
// Tell docker to hard kill the container in 120 seconds
resource.Expire(120)
hostAndPort := resource.GetHostPort("5432/tcp")
databaseUrl := fmt.Sprintf("postgres://user_name:secret@%s/dbname?sslmode=disable", hostAndPort)
pool.MaxWait = 120 * time.Second
// if err = pool.Retry(func() error {
// db, err = sql.Open("postgres", databaseUrl)
// if err != nil {
// ...
Cassandra, Redis, Kafka, …
Ory’s Dockertest works with all Docker Images! There are many examples available on GitHub. To name a few:
Dockertest in the Ory Ecosystem
Ory is an open-source, cyber security company. Ory’s Dockertest is just one example of how Ory approaches software development. If this article caught your attention and interest, check out our other projects on GitHub as well!
Conclusion
Testing production code includes testing the interaction of all components within an environment that resembles production as closely as possible. By spinning up Docker containers a test environment is created that is real and fully controlled. Controlling the flow from your Go code gives you a high degree of freedom and flexibility! Ory’s Dockertest can automate this process in a CI pipeline not only for integration but also for unit, functional, and e2e tests!
More examples, code, and development over at GitHub: Ory Dockertest.
Thank you for taking the time to read this article! You can find out more about the Ory Community on our website. While you’re here, sign up for the Ory newsletter to receive updates on Ory’s Dockertest and Ory’s other projects!