Our team developed a service that reads from a Kafka topic, interacts with a Postgres database using basic CRUD operations, and calls APIs on an external service. As part of our increased focus on automated testing, getting to a higher level of testing code coverage required us to tie in the external components of our system architecture. Since we don’t control these components in production, we included these components in our CI/CD pipeline to give us a higher level of confidence that our integrated code is working as expected. We also wanted to run automated functional tests against our service, and that too required running these services in the pipeline. We use Drone as our CI/CD tooling and decided to learn how to use services in Drone to solve this problem. This short blog post runs through an example of how we started up a database service and a Kafka service in Drone and ran a test against these service, that might help others in a similar situation.

Database Service

In order to fire up a Postgres database service in Drone, we added this to our .drone.yml specification:

services:
  database:
    image: postgres:10.6
    environment:
      POSTGRES_USER: drone_db_user
      POSTGRES_PASSWORD: drone_db_pass

The environment variables for the container establish an initial user and password when the service starts.

Our service is written in golang and uses a package called pop for database interactions, which has a command line tool soda that creates the database and instantiates the tables in the database. We therefore needed to install these packages on our Drone system and invoke the soda commands in our pipeline once the database service was up and running. Our first pass at a pipeline in the .drone.yml file put these pieces together and executed a simple test to write to and read from the database:

pipeline:
  build-binary-linux:
    image: ${CUSTOM_REGISTRY}/${ORG}/alpine-go111-build:latest
    environment:
      - GO111MODULE=on
      - CGO_ENABLED=0
      - GOOS=linux
      - GOARCH=amd64
      - DB_ENVIRONMENT=localdocker
      - DB_USER=drone_db_user
      - DB_PASS=drone_db_pass
      - DB_NAME=dronetesting_db
      - DB_HOST=database
    commands:
      - go get -v  github.com/gobuffalo/pop
      - go install github.com/gobuffalo/pop/soda
      - cd ./db
      - soda drop -e dronetesting
      - soda create -e dronetesting
      - soda migrate -e dronetesting up
      - cd ..
      - go test -timeout 240s -p 1 github.com/${ORG}/service/cmd -run Test_DBConnection -v

There are several things to notice in this pipeline. First, the commands to install pop and soda in the Docker container that executes the test and to create the database and associated tables are exactly the same commands a developer runs on the command line on their system to set up a local database using these packages. Second, the environment variables DB_* are used in our program’s database configuration to connect to the correct database service, which is why the user and password values match those created when the database service was started. In this example, we used plaintext for the username and password since our tests are not writing any sensitive data to the database and the database only exists for the duration of the execution of the pipeline, but for added security, these could be stored as secrets. Third, and critically, DB_HOST must be the same as the name given to the database service. Depending on the language and packages you are using, the details of how to create tables and populate them with data will differ, but we have employed a similar strategy for creating and connecting to a database for testing a Rust application so know this approach adapts to other environments.

Kafka Service

We wanted to test that our program could read an IP from a Kafka topic, do some processing on it, and write the IP to a table in the database. To be able to effectively utilize a Kafka endpoint, we needed to create and configure a small running instance of Kafka and its required software component ZooKeeper within our CI/CD build, as integration testing relies on true functional capabilities of each external software dependency. This article gives a brief description of the Kafka architecture and explains its components. After some trial-and-error, we found a successful strategy for bringing up the Kafka service, connecting it to our pipeline, and running a test that executes our test scenario.

In the services section of the .drone.yml file, we added a Kafka service:

 kafka:
    image: spotify/kafka:latest
    ports:
      - "9092:9092"
      - "2181:2181"
    environment:
      - TOPICS=test

This container runs an instance of ZooKeeper on port 2181 and Kafka on port 9092. Writing to and reading from a Kafka service requires a broker – the connection to the service itself – and a topic – the queue the messages are written to and read from. Adding the environment variable TOPICS causes the ZooKeeper instance running in this service to create a Kafka topic named test when the service starts up. This is the topic we will write to and read from. Our application uses the environment variables KAFKA_BROKER, KAFKA_TOPIC, and KAFKA_GROUP to set up the reader, so our pipeline needed to have those variables set up to connect to the service. As with the database Drone setup, the KAFKA_BROKER variable uses the Drone Kafka service name (kafka) to establish the connection with that service. Below is our final pipeline to run the test:

pipeline:
  build-binary-linux:
    image: ${CUSTOM_REGISTRY}/${ORG}/alpine-go111-build:latest
    environment:
      - GO111MODULE=on
      - CGO_ENABLED=0
      - GOOS=linux
      - GOARCH=amd64
      - DB_ENVIRONMENT=localdocker
      - DB_USER=drone_db_user
      - DB_PASS=drone_db_pass
      - DB_NAME=dronetesting_db
      - DB_HOST=database
      - KAFKA_BROKER=kafka:9092
      - KAFKA_TOPIC=test
      - KAFKA_GROUP_ID=1
    commands:
      - go get -v github.com/gobuffalo/pop
      - go install github.com/gobuffalo/pop/soda
      - cd ./db
      - soda create -e dronetesting
      - soda migrate -e dronetesting up
      - cd ..
      - go test -timeout 240s -p 1 github.com/${ORG}/service/cmd -run Test_KafkaConnection -v

Automated Test

There were two additional things we learned related to the test itself that were critical to getting the entire automated test working. First, the code we wanted to test that created the Kafka reader needed to be started in a separate process. This ensured the reader was not blocking the execution of the test itself. Second, writing test data to the topic needed to occur after the process running the reader was completely up. This ensured that the service was up, topic created, and the connection available.

func Test_KafkaConnection(t *testing.T) {
    var brokerList []string
    
    broker := utils.Config.WHConfig.KafkaBroker
    topic := utils.Config.WHConfig.KafkaTopic
    
    // populate the string slice for the broker list
    if strings.Contains(broker, ",") {
        brokerList = strings.Split(broker, ",")
    } else {
        brokerList = []string{broker}
    }
    
    writer := kafka.NewWriter(kafka.WriterConfig{
        Brokers:  brokerList,
        Topic:    topic,
        Balancer: &kafka.LeastBytes{},
    })
    
    defer writer.Close()
    
    // Start the service in non-blocking fashion, and give it time to come up
    go func() {
        startService()
    }()
    time.Sleep(30 * time.Second)
    
    // Write the message to the topic
    ip := "some IP provided"
    writer.WriteMessages(context.Background(),
        kafka.Message{
            Key:   []byte("Event"),
            Value: generateMessage(ip),
        })
        
    // Pause to allow time for service to read from topic, process,
    // and write to database
    time.Sleep(10 * time.Second)
    
    // Verify IP ended up in database
    dbConn, err := utils.NewDBConnection("localdocker")
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }
    
    lookupIP, err := utils.LookupIP(dbConn, ip)
    assert.Nil(t, err, "LookupIP error")
    assert.NotNil(t, lookupIP)
}

Here is the output of the services when this pipeline is run in drone:

Database Service in Drone

Kafka Service in Drone

References

Pam Vermeer is an eMIP Senior Engineering Manager and Pat Moberg is a Lead Engineer, both on the Digital Security Team. Pam and Pat both have experience across a variety of programming languages, and have a passion for improving and increasing automated testing in whatever development ecosystem they are using.