Recently I worked on a project where I used Google Cloud Spanner, the fully managed relational database with unlimited scale, strong consistency, and up to 99.999% availability. When started creating tests, I had some difficulty to find out a good testing strategy.
In Go, a common pattern for creating tests around external libraries is to use or define an interface which will be implemented by a mock, and to use the mock in your tests.
For a sql database, you can use a library like github.com/DATA-DOG/go-sqlmock which defines implementations for the interfaces defined in databases/sql. This means that whenever you use a
sql.DB, you can replace it with the mock from the
Unfortunately, with Cloud Spanner it is not that straight-forward. There are no exported interfaces in the Spanner library. I found an issue on Github mentioning the exact problem. This comment explains how annoying it is to have to define your own interfaces and wrapper to get to a workable situation. A couple of comments later advice is given: “Don’t overuse mocks."
In one of the replies, it is mentioned that:
As a general rule, the best way to write unit tests with these clients is to use a fake server.
The fake server for Spanner is spannertest. You get greeted by the friendly message:
Here’s a list of features that are missing or incomplete.
Alright, I get that this is unexplored territory, but let’s try to use it anyway.
Well, this is awkward. The fake spanner server does not support the
ReplaceStruct operation yet. Should I rewrite my application to not use these nice features, just because the fake spanner server does not support them? I don’t think so.
Testing against the Emulator
When you develop an application that uses Spanner, you probably develop against a locally running Spanner Emulator. Or course this is in beta too, but it’s better than the alpha or experimental packages that seem to be extremely common with Googles products.
The emulator can be started using
or by running a docker container using
A colleague mentioned having good experiences with testcontainers/testcontainers-go. It is a Go library that lets you interact with docker from Go.
Starting the Spanner Emulator docker container can be done using:
It starts the container and checks the logs to wait for that message that the server is listening. The exposed port will be mapped to a random available port, so you should set the
SPANNER_EMULATOR_HOST environment variable with the right port.
There are some drawbacks to this though. On my machine it takes about 4 seconds for the container to start, so you don’t want to start a docker container for each of your tests. However, if you want to run a set of integration tests, than this is a suitable solution. The only thing you need to remember is your tests do not start with a clean database, unless you create methods of either removing the data a test inserted, or by clearing all records from the database at the start of your tests.
Testing in Cloud Build
In this project, Cloud Run is used to run the tests and build the docker images. While you can spin up Spanner in Cloud Build, I don’t think it’s a good idea to spend $700 per month for a single instance to use in your tests.
Let’s try to use
testcontainers-go in our tests in Cloud Run as well. In the tests, we need to interact with the docker daemon to start the Spanner Emulator. Unfortunately, you don’t have direct access to the docker daemon, and when you try to mount
/var/run/docker.sock in your build container, you get an error:
Understandably, you cannot interact with the docker daemon directly. So is there a solution to this problem? What if we could start the Spanner Emulator before running the tests?
The Cloud Build config files make it look as if you are directly interacting with docker, but unfortunately only a few command line arguments is supported, which does not include
-d, the argument to start a container daemonized.
This Stackoverflow post shows us that you can use the
docker/compose build step to start a daemonized container. After tinkering, I came to the following set up:
When running the tests, I check if the environment variable
SPANNER_EMULATOR_AVAILABLE is set to
true, when it is the tests can be run. When the environment variable is not set, for example when running the tests locally, it will spin up the Spanner Emulator using
It is not as straight-forward to test an application that uses Cloud Spanner as I would like it to be. However, I’m happy with the current setup that uses
testcontainers-go. I will take a look at the RPC Replay if I find the time. Please let me know if this helped you or if you have found a better method.