Skip to content

use t.TempDir instead of ioutil.TempDir in unit tests#10321

Closed
ianwoolf wants to merge 2 commits intoprometheus:mainfrom
ianwoolf:pr-remote-ioutil_TempDir-2
Closed

use t.TempDir instead of ioutil.TempDir in unit tests#10321
ianwoolf wants to merge 2 commits intoprometheus:mainfrom
ianwoolf:pr-remote-ioutil_TempDir-2

Conversation

@ianwoolf
Copy link
Copy Markdown
Contributor

Signed-off-by: ianwoolf [email protected]

Updates #9594

LeviHarrison
LeviHarrison previously approved these changes Feb 24, 2022
Copy link
Copy Markdown
Contributor

@LeviHarrison LeviHarrison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! And welcome to the project!

@LeviHarrison LeviHarrison enabled auto-merge (squash) February 24, 2022 15:48
dir, err := ioutil.TempDir("", "test_concurrency")
require.NoError(t, err)
dir := t.TempDir()
defer os.RemoveAll(dir)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, I thought the test_windows failure was a flake (as it usually is), but there is actually a problem. defer os.RemoveAll(dir) should be removed as well, as t.TempDir() takes care of deleting the files.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emm... I didn't notice the the tmp dir was automatically removed.

@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch from 438747e to 39afd26 Compare February 25, 2022 02:36
@LeviHarrison
Copy link
Copy Markdown
Contributor

I think there's just one last thing: the os import needs to be removed.

# github.com/prometheus/prometheus/promql [github.com/prometheus/prometheus/promql.test]
promql/engine_test.go:19:2: imported and not used: "os"

@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch from 39afd26 to 4ae36a0 Compare February 25, 2022 06:48
@ianwoolf
Copy link
Copy Markdown
Contributor Author

ianwoolf commented Feb 25, 2022

I think there's just one last thing: the os import needs to be removed.

# github.com/prometheus/prometheus/promql [github.com/prometheus/prometheus/promql.test]
promql/engine_test.go:19:2: imported and not used: "os"

fix

@ianwoolf ianwoolf closed this Feb 25, 2022
@ianwoolf ianwoolf reopened this Feb 25, 2022
@LeviHarrison LeviHarrison enabled auto-merge (squash) February 25, 2022 12:00
@LeviHarrison
Copy link
Copy Markdown
Contributor

OK, so I think I've figured out why test_windows keeps failing with the following error: testing.go:967: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestQueryConcurrency3635900781\001\queries.active: The process cannot access the file because it is being used by another process. This error happens because the query logger is still using the active query file when the temporary directory is removed. That file should be closed before the temporary directory is removed, and we do stop the query engine (which should close the file), just using a defer statement. That worked before this change when the temporary directory was also removed using a defer statement. But now, t.TempDir removes the directory using a cleanup function, which is called before the deferred functions are. Therefore, the directory is attempted to be removed before the query engine is closed, causing the error.

In order to fix this, these two deferred functions should be registered as cleanup functions instead:

https://github.com/ianwoolf/prometheus/blob/4ae36a0575b4a4bc243862231a0cf4e613d7970c/promql/engine_test.go#L53

https://github.com/ianwoolf/prometheus/blob/4ae36a0575b4a4bc243862231a0cf4e613d7970c/promql/engine_test.go#L58
(I don't think this one's actually necessary but we might as well)

That should fix it.

@LeviHarrison LeviHarrison changed the title promql: use t.TempDir instead of ioutil.TempDir on tests promql: use t.TempDir instead of ioutil.TempDir in unit tests Feb 25, 2022
@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch from 4ae36a0 to 7a7d891 Compare February 26, 2022 14:18
@ianwoolf
Copy link
Copy Markdown
Contributor Author

ianwoolf commented Feb 27, 2022

OK, so I think I've figured out why test_windows keeps failing with the following error: testing.go:967: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestQueryConcurrency3635900781\001\queries.active: The process cannot access the file because it is being used by another process. This error happens because the query logger is still using the active query file when the temporary directory is removed. That file should be closed before the temporary directory is removed, and we do stop the query engine (which should close the file), just using a defer statement. That worked before this change when the temporary directory was also removed using a defer statement. But now, t.TempDir removes the directory using a cleanup function, which is called before the deferred functions are. Therefore, the directory is attempted to be removed before the query engine is closed, causing the error.

In order to fix this, these two deferred functions should be registered as cleanup functions instead:

https://github.com/ianwoolf/prometheus/blob/4ae36a0575b4a4bc243862231a0cf4e613d7970c/promql/engine_test.go#L53

https://github.com/ianwoolf/prometheus/blob/4ae36a0575b4a4bc243862231a0cf4e613d7970c/promql/engine_test.go#L58 (I don't think this one's actually necessary but we might as well)

That should fix it.

only registering the refer astesting.ClearUp doesn't work. Looks like has to wait for the query logger to be closed.

I add Close for ActiveQueryTracker to close the os.File and call it in test. now i get an error as follow

=== RUN   TestQueryConcurrency
    testing.go:967: TempDir RemoveAll cleanup: remove C:\Users\circleci\AppData\Local\Temp\TestQueryConcurrency191334965\001\queries.active: Access is denied.
--- FAIL: TestQueryConcurrency (0.04s)

and i pass the test on mac and linux, it still fails on windows.

The error log is not is being used by another process any more, so closing the query log is worked. But i don't understant why Access is denied.
I open an issue and pr to discuss it, turn this issue to WIP

@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch from 0f862c6 to e9f136a Compare February 27, 2022 12:54
@ianwoolf ianwoolf changed the title promql: use t.TempDir instead of ioutil.TempDir in unit tests WIP: use t.TempDir instead of ioutil.TempDir in unit tests Feb 27, 2022
@ianwoolf ianwoolf changed the title WIP: use t.TempDir instead of ioutil.TempDir in unit tests use t.TempDir instead of ioutil.TempDir in unit tests Sep 12, 2022
@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch 2 times, most recently from d4d4213 to d77232a Compare April 14, 2023 07:20
@codesome codesome removed their request for review April 26, 2023 20:51
@roidelapluie roidelapluie self-assigned this Jul 18, 2023
@roidelapluie
Copy link
Copy Markdown
Member

cc @ianwoolf can you please rebase this pull request? Thanks!

@ianwoolf ianwoolf force-pushed the pr-remote-ioutil_TempDir-2 branch from d77232a to 4cb3a13 Compare July 19, 2023 16:53
@ianwoolf
Copy link
Copy Markdown
Contributor Author

cc @ianwoolf can you please rebase this pull request? Thanks!

done :)

@bboreham
Copy link
Copy Markdown
Member

Discussed in the bug scrub; apologies we dropped the ball in July. This still looks worthwhile, however at the last CI run Windows tests failed and something needs a DCO signed-off line.

@ianwoolf could you add the signed-off, rebase again and we can try to figure out the Windows test?

@aknuds1
Copy link
Copy Markdown
Contributor

aknuds1 commented Sep 21, 2024

I think this PR can now be safely closed, since after rebasing on main and fixing conflicts, there are no changes left.

@aknuds1 aknuds1 added the stale label Sep 21, 2024
@bboreham
Copy link
Copy Markdown
Member

bboreham commented Oct 8, 2024

Thanks!

@bboreham bboreham closed this Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants