feat: provide a tool to generate examples from code by mdelapenya · Pull Request #603 · testcontainers/testcontainers-go

mdelapenya · 2022-11-02T07:05:57Z

What does this PR do?

It moves the existing examples' code (cockroachDB, nginx, pulsar and redis) to separate example Go modules that will be automatically included into the docs site.

Besides, we have provided a command line tool to generate the scaffolding for the code of a new example. This tool will generate:

a Go module for the example, including:
- go.mod and go.sum files
- a Go package named after the example, in lowercase
- a Go file for the creation of the container, using a dedicated struct.
- a Go test file for running a simple test for your container, consuming the above struct.
- a Makefile to run the tests in a consistent manner
- a tools.go file including the build tools (i.e. gotestsum) used to build/run the example.
a markdown file in the docs/examples directory including the snippets for both the creation of the container and a simple test.

This tool is not doing the following:

If the example already exists, it will exit without updating the existing files.
You have to manually add the markdown entry in the docs to the mkdocs.yml file in the root directory of the project. It will generate the navigation menu for the docs website.

In this PR we are adding a Make goal to run all the examples' tests, in order to check if they are still valid and not outdated. The make goal, which calls each example's Makefile, is used in the CI pipeline.

Why is it important?

Examples embedded into the docs could get outdated and they could not even compile. With this approach, the tests will be part of the test suite.

We expect the code generation tool will simplify how to start an example, creating the scaffolding for it.

mdelapenya · 2022-11-02T12:16:14Z

Weird: the tests fail because of a difference of milliseconds in the create date:

38s
Run make test-examples
Running example tests...
make -C examples test
make[1]: Entering directory '/home/runner/work/testcontainers-go/testcontainers-go/examples'
make -C cockroachdb test
make[2]: Entering directory '/home/runner/work/testcontainers-go/testcontainers-go/examples/cockroachdb'
make test-cockroachdb
make[3]: Entering directory '/home/runner/work/testcontainers-go/testcontainers-go/examples/cockroachdb'
Running cockroachdb tests...
go run gotest.tools/gotestsum \
	--format short-verbose \
	--rerun-fails=5 \
	--packages="./..." \
	--junitfile TEST-cockroachdb.xml
go: downloading github.com/jackc/pgx/v4 v4.17.2
go: downloading github.com/jackc/pgconn v1.13.0
go: downloading github.com/jackc/pgtype v1.12.0
go: downloading golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa
go: downloading github.com/jackc/chunkreader/v2 v2.0.1
go: downloading github.com/jackc/pgio v1.0.0
go: downloading github.com/jackc/pgpassfile v1.0.0
go: downloading github.com/jackc/pgproto3/v2 v2.3.1
go: downloading github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b
=== RUN   TestIntegrationDBInsertSelect
2022/11/02 11:54:32 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 20.10.18+azure-2
  API Version: 1.41
  Operating System: Ubuntu 20.04.5 LTS
  Total Memory: 6944 MB
2022/11/02 11:54:32 Starting container id: 59544397cbd3 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:32 Waiting for container id 59544397cbd3 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:32 Container is ready id: 59544397cbd3 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:37 Starting container id: 1c0657415303 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:38 Waiting for container id 1c0657415303 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:38 Container is ready id: 1c0657415303 image: cockroachdb/cockroach:latest-v21.1
    cockroachdb_test.go:99: Saved task is not the same:
          cockroachdb.task{
          	ID:          "1415f232-710c-40af-a415-d119dfb9c3d0",
          	Description: "Update resumé",
          	DateDue:     nil,
        - 	DateCreated: s"2022-11-02 11:54:38.498883824 +0000 UTC m=+6.247304161",
        + 	DateCreated: s"2022-11-02 11:54:38.498883 +0000 UTC",
        - 	DateUpdated: s"2022-11-02 11:54:38.498883824 +0000 UTC m=+6.247304161",
        + 	DateUpdated: s"2022-11-02 11:54:38.498883 +0000 UTC",
          }
--- FAIL: TestIntegrationDBInsertSelect (6.46s)
FAIL TestIntegrationDBInsertSelect (6.46s)
FAIL .

DONE 1 tests, 1 failure in 17.610s

=== RUN   TestIntegrationDBInsertSelect
2022/11/02 11:54:41 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 20.10.18+azure-2
  API Version: 1.41
  Operating System: Ubuntu 20.04.5 LTS
  Total Memory: 6944 MB
2022/11/02 11:54:41 Starting container id: fdbcd1bdd8a2 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:41 Waiting for container id fdbcd1bdd8a2 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:41 Container is ready id: fdbcd1bdd8a2 image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:42 Starting container id: da1e07d8a1d9 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:42 Waiting for container id da1e07d8a1d9 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:42 Container is ready id: da1e07d8a1d9 image: cockroachdb/cockroach:latest-v21.1
    cockroachdb_test.go:99: Saved task is not the same:
          cockroachdb.task{
          	ID:          "d5b11f4a-b1d0-46d4-92c5-c0fdc87c326d",
          	Description: "Update resumé",
          	DateDue:     nil,
        - 	DateCreated: s"2022-11-02 11:54:42.615584654 +0000 UTC m=+1.118725029",
        + 	DateCreated: s"2022-11-02 11:54:42.615584 +0000 UTC",
        - 	DateUpdated: s"2022-11-02 11:54:42.615584654 +0000 UTC m=+1.118725029",
        + 	DateUpdated: s"2022-11-02 11:54:42.615584 +0000 UTC",
          }
--- FAIL: TestIntegrationDBInsertSelect (1.33s)
FAIL TestIntegrationDBInsertSelect (re-run 1) (1.33s)
FAIL .

DONE 2 runs, 2 tests, 2 failures in 21.718s

=== RUN   TestIntegrationDBInsertSelect
2022/11/02 11:54:45 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 20.10.18+azure-2
  API Version: 1.41
  Operating System: Ubuntu 20.04.5 LTS
  Total Memory: 6944 MB
2022/11/02 11:54:45 Starting container id: c6366d9a062b image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:46 Waiting for container id c6366d9a062b image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:46 Container is ready id: c6366d9a062b image: docker.io/testcontainers/ryuk:0.3.4
2022/11/02 11:54:46 Starting container id: 7e8edd34adf2 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:46 Waiting for container id 7e8edd34adf2 image: cockroachdb/cockroach:latest-v21.1
2022/11/02 11:54:46 Container is ready id: 7e8edd34adf2 image: cockroachdb/cockroach:latest-v21.1
    cockroachdb_test.go:99: Saved task is not the same:
          cockroachdb.task{
          	ID:          "ec01ebba-f8e9-4a58-9246-087cce19e013",
          	Description: "Update resumé",
          	DateDue:     nil,
        - 	DateCreated: s"2022-11-02 11:54:46.737085172 +0000 UTC m=+1.109263215",
        + 	DateCreated: s"2022-11-02 11:54:46.737085 +0000 UTC",
        - 	DateUpdated: s"2022-11-02 11:54:46.737085172 +0000 UTC m=+1.109263215",
        + 	DateUpdated: s"2022-11-02 11:54:46.737085 +0000 UTC",
          }
--- FAIL: TestIntegrationDBInsertSelect (1.31s)

eddumelendez

nice!

* main: docs: networking basics (#612) docs: wording in project name (#610) feat: Auth config for build images (#602) chore: sync governance files (#608)

* main: feat: add golangci-lint (#569)

* main: docs: fix syntax (#615)

This reverts commit 28c94c7.

mdelapenya · 2022-11-10T11:16:08Z

examples/pulsar/pulsar.go

+			wait.ForHTTP("/admin/v2/clusters").WithPort("8080/tcp").WithResponseMatcher(matchAdminResponse).WithStartupTimeout(defaultTimeout),
+			wait.ForLog("Successfully updated the policies on namespace public/default").WithStartupTimeout(defaultTimeout),
+		).WithStartupTimeout(defaultTimeout),


@clive-jevons please take a look at this startup timeout

mdelapenya · 2022-11-10T11:16:20Z

examples/pulsar/pulsar_test.go

+)
+
+func TestPulsar(t *testing.T) {
+	ctx, cancel := context.WithTimeout(context.Background(), defaultTimeout)


@clive-jevons same here

This reverts commit 65ee9b5.

* main: chore: use a fixed version of MySQL (#617)

…ckage""" This reverts commit f50e404.

mdelapenya · 2022-11-10T18:57:22Z

Superseded by #618

mdelapenya added 6 commits November 2, 2022 06:24

chore: convert pulsar example in executable code

0fcd7bd

chore: convert nginx example in executable code

9d5fbfe

fix: handle error in tests

30171e2

chore: convert redis example in executable code

51a0690

chore: convert cockroachDB example in executable code

9cd249f

chore: simplify including the entire file as example

9be5620

mdelapenya requested a review from a team as a code owner November 2, 2022 07:05

mdelapenya added the documentation Docs, docs, docs. label Nov 2, 2022

mdelapenya self-assigned this Nov 2, 2022

mdelapenya requested review from HofmeisterAn, eddumelendez and kiview November 2, 2022 07:05

mdelapenya added 7 commits November 2, 2022 09:36

chore: bump dependencies in the e2e module

1f396e5

chore: add a common Make goal for running Go tests

44434df

chore: push go mod goals to the common file

2e37ff3

chore: add gotestsum as tools to the examples

d8bf2e8

chore: support running the example files with Make

7c48ee0

chore: use new goal in CI pipelines

f6272d4

feat: add a tool to bootstrap an example

5f0fcc2

mdelapenya changed the title ~~docs: convert examples into executable tests~~ feat: provide a tool to generate examples from code Nov 2, 2022

mdelapenya added 9 commits November 2, 2022 16:45

chore: extract code generation to a function

f110af4

chore: pass the examples dir to the generate function

bfa9581

chore: pass the examples docs dir to the generate function

2328361

fix: prepend examples dir when creating the examples files

f32d9bb

chore: extract templates to a constant

b3ab1b9

chore: save toolsgo under its own package

e66df3f

chore: add unit tests for the code generation tool

7ab47d5

chore: run unit tests for the examples

9eba684

chore: simplify how to compare tasks

70e2ed5

eddumelendez previously approved these changes Nov 7, 2022

View reviewed changes

mdelapenya added 4 commits November 9, 2022 11:09

Merge branch 'main' into examples

ed0121e

* main: docs: networking basics (#612) docs: wording in project name (#610) feat: Auth config for build images (#602) chore: sync governance files (#608)

chore: support setting the Docker image for the example

f55c0fb

docs: document the command line flags

6305f98

fix: simplify path generation for examples

47f2a43

mdelapenya dismissed eddumelendez’s stale review via 47f2a43 November 9, 2022 11:52

mdelapenya added 5 commits November 9, 2022 17:15

fix: remove unused package in template

a618362

fix: use a title for the example in functions

fead327

chore: add comments to functions

e07d6c6

chore: simplify passing the example to the generator

df051de

Merge branch 'main' into examples

ea9cff0

* main: feat: add golangci-lint (#569)

mdelapenya mentioned this pull request Nov 9, 2022

docs: fix syntax #615

Merged

mdelapenya added 6 commits November 10, 2022 09:25

Merge branch 'main' into examples

e3f9ca1

* main: docs: fix syntax (#615)

chore: increase test timeout

c8e4433

fix: remove parallel execution at the root package

28c94c7

fix: do not use %w in Errorf in tests

052f4d7

fix: use a timeout in pulsar example

04c3dba

Revert "fix: remove parallel execution at the root package"

65ee9b5

This reverts commit 28c94c7.

mdelapenya commented Nov 10, 2022

View reviewed changes

mdelapenya added 2 commits November 10, 2022 12:30

Revert "Revert "fix: remove parallel execution at the root package""

f50e404

This reverts commit 65ee9b5.

Merge branch 'main' into examples

ffb4f90

* main: chore: use a fixed version of MySQL (#617)

mdelapenya added the enhancement New feature or request label Nov 10, 2022

mdelapenya added 3 commits November 10, 2022 15:41

Revert "Revert "Revert "fix: remove parallel execution at the root pa…

da332c0

…ckage""" This reverts commit f50e404.

chore: remove all t.Parallel

554f1e7

chore: try printing error in reaper

ad626a2

mdelapenya mentioned this pull request Nov 10, 2022

feat: provide a tool to generate examples from code #618

Merged

mdelapenya closed this Nov 10, 2022

mdelapenya deleted the examples branch November 23, 2022 16:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: provide a tool to generate examples from code#603

feat: provide a tool to generate examples from code#603
mdelapenya wants to merge 43 commits intotestcontainers:mainfrom
mdelapenya:examples

mdelapenya commented Nov 2, 2022 •

edited

Loading

Uh oh!

mdelapenya commented Nov 2, 2022

Uh oh!

eddumelendez left a comment

Uh oh!

mdelapenya Nov 10, 2022

Uh oh!

mdelapenya Nov 10, 2022

Uh oh!

mdelapenya commented Nov 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mdelapenya commented Nov 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Uh oh!

mdelapenya commented Nov 2, 2022

Uh oh!

eddumelendez left a comment

Choose a reason for hiding this comment

Uh oh!

mdelapenya Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya commented Nov 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mdelapenya commented Nov 2, 2022 •

edited

Loading