Skip to content

Commit e33e048

Browse files
authored
[fix](be) Stop two SNII unit tests from trampling shared state (apache#66961)
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Two cases under `be/test/storage/index/snii/` reach outside their own test and break unrelated cases. Both are test-only; no production code changes. **1. A fixed `/tmp` scratch path (`metered_file_reader_test.cpp`)** `MakeRampFile()` built its 256-byte ramp file at the constant path `/tmp/snii_metered_ramp.bin`. On a shared build machine that file belongs to whoever ran the suite first; every later user gets `EACCES` from `::open`, and all seven `SniiMeteredFileReader` cases fail at the first assertion with no hint that a foreign file is the cause: ``` metered_file_reader_test.cpp:42: Failure Value of: w.open(path).ok() Actual: false Expected: true ``` Every other scratch path in this subtree already namespaces itself with the pid and a counter, and the nearest neighbour in the same directory (`file_reader_caller_buffer_test.cpp`) also `std::remove()`s the file afterwards. This PR adopts both halves of that convention: the ramp file becomes a `RampFile` RAII object whose name carries the pid and a counter and whose destructor removes it. Removing matters as much as naming — a leftover file is what causes the collision in the first place, and pids get recycled. **2. A permanently lowered `RLIMIT_NOFILE` (`spimi_term_buffer_test.cpp`)** `SniiSpimiTermBuffer.SpillOpenIoFailureLatched` drives a spill's `::open` to `EMFILE` by lowering `RLIMIT_NOFILE` to a hardcoded 64 and then opening `/dev/null` until the table is full. Two things go wrong once the binary already holds more than 64 descriptors: 1. The cap assumes the process holds fewer than 64 descriptors, so the hog loop can still take one. Sampling `/proc/<pid>/fd` during a run shows the SNII cases alone keep the process at **14** descriptors, but adding the CLucene inverted index cases to the same binary takes it to **273** — they hold hundreds of index files open at once. Above the cap the very first `::open` already fails, `hogs` stays empty, and the case stops at `ASSERT_FALSE(hogs.empty())`. What decides this is *which* cases share the binary, not how many. 2. That assertion is fatal, so it returns from the test body — and the code that restores `RLIMIT_NOFILE` sits after it and never runs. The whole process stays capped at 64 descriptors for every later case. The result is not a descriptor leak but a cascade: the next four `SniiSpimiTermBuffer` cases fail instantly because they cannot open a spill file, and the first death test after them aborts the entire run when gtest's `pipe()` hits `EMFILE`: ``` CHECK failed: gtest-death-test.cc, line 1400: pipe(pipe_fd) != -1 Aborted ``` The fix holds the tight limit and the hog descriptors in an inner scope whose `Defer` restores `RLIMIT_NOFILE` and closes the descriptors on every exit path, fatal assertion included; and derives the cap from `::dup(STDIN_FILENO)`, the lowest free descriptor, instead of a fixed 64, so the case no longer depends on which tests ran before it. The case still drives the spill open to `EMFILE` and still asserts the error is latched. **Before / after**, with `--filter='*Snii*:*snii*:*SNII*:CollectionStatisticsTest.*:*InvertedIndex*:*inverted_index*'`: | | before | after | |---|---|---| | `SniiMeteredFileReader` | 7 failed | pass | | `SniiSpimiTermBuffer*` | 5 failed | pass | | death test | aborted the run | pass | | total | run truncated by the abort | 1693 passed, 0 failed |
1 parent 02fbf37 commit e33e048

2 files changed

Lines changed: 77 additions & 42 deletions

File tree

‎be/test/storage/index/snii/io/metered_file_reader_test.cpp‎

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,10 @@
1818
#include "storage/index/snii/io/metered_file_reader.h"
1919

2020
#include <gtest/gtest.h>
21+
#include <unistd.h>
2122

2223
#include <cstdint>
24+
#include <cstdio>
2325
#include <string>
2426
#include <vector>
2527

@@ -35,27 +37,41 @@ using doris::snii::io::Range;
3537

3638
namespace {
3739

38-
// Writes 256 bytes (byte[i] = i) to a temp file and returns its path.
39-
std::string MakeRampFile() {
40-
const std::string path = "/tmp/snii_metered_ramp.bin";
41-
LocalFileWriter w;
42-
EXPECT_TRUE(w.open(path).ok());
43-
std::vector<uint8_t> data(256);
44-
for (int i = 0; i < 256; ++i) {
45-
data[i] = static_cast<uint8_t>(i);
40+
// Owns one scratch file holding 256 bytes (byte[i] = i) and removes it on scope
41+
// exit. The name carries the pid and a counter, like the other scratch paths in
42+
// this directory: a fixed /tmp name left behind by another user on a shared
43+
// machine makes ::open fail with EACCES and takes every test here down with it.
44+
class RampFile {
45+
public:
46+
RampFile() {
47+
static int counter = 0;
48+
path_ = "/tmp/snii_metered_ramp_" + std::to_string(::getpid()) + "_" +
49+
std::to_string(counter++) + ".bin";
50+
LocalFileWriter w;
51+
EXPECT_TRUE(w.open(path_).ok());
52+
std::vector<uint8_t> data(256);
53+
for (int i = 0; i < 256; ++i) {
54+
data[i] = static_cast<uint8_t>(i);
55+
}
56+
EXPECT_TRUE(w.append(Slice(data)).ok());
57+
EXPECT_TRUE(w.finalize().ok());
4658
}
47-
EXPECT_TRUE(w.append(Slice(data)).ok());
48-
EXPECT_TRUE(w.finalize().ok());
49-
return path;
50-
}
59+
~RampFile() { std::remove(path_.c_str()); }
60+
61+
const std::string& path() const { return path_; }
62+
63+
private:
64+
std::string path_;
65+
};
5166

5267
} // namespace
5368

5469
// Single reads: first read to a block is a cache miss (1 round, 1 GET, 1 block of
5570
// remote bytes); a second read to the same 16-byte block is a hit (no new round).
5671
TEST(SniiMeteredFileReader, SingleReadCacheAccounting) {
72+
const RampFile ramp;
5773
LocalFileReader inner;
58-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
74+
ASSERT_TRUE(inner.open(ramp.path()).ok());
5975
MeteredFileReader m(&inner, /*block_size=*/16);
6076

6177
std::vector<uint8_t> out;
@@ -84,8 +100,9 @@ TEST(SniiMeteredFileReader, SingleReadCacheAccounting) {
84100

85101
// A read spanning 3 contiguous blocks is one round and one coalesced GET.
86102
TEST(SniiMeteredFileReader, SpanMultipleBlocksCoalesced) {
103+
const RampFile ramp;
87104
LocalFileReader inner;
88-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
105+
ASSERT_TRUE(inner.open(ramp.path()).ok());
89106
MeteredFileReader m(&inner, 16);
90107

91108
std::vector<uint8_t> out;
@@ -98,8 +115,9 @@ TEST(SniiMeteredFileReader, SpanMultipleBlocksCoalesced) {
98115

99116
// A batch of reads to non-adjacent blocks: one serial round, one GET per run.
100117
TEST(SniiMeteredFileReader, BatchNonAdjacent) {
118+
const RampFile ramp;
101119
LocalFileReader inner;
102-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
120+
ASSERT_TRUE(inner.open(ramp.path()).ok());
103121
MeteredFileReader m(&inner, 16);
104122

105123
std::vector<Range> ranges = {{.offset = 0, .len = 4},
@@ -117,8 +135,9 @@ TEST(SniiMeteredFileReader, BatchNonAdjacent) {
117135

118136
// A batch of reads to adjacent blocks coalesces into a single GET.
119137
TEST(SniiMeteredFileReader, BatchAdjacentCoalesced) {
138+
const RampFile ramp;
120139
LocalFileReader inner;
121-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
140+
ASSERT_TRUE(inner.open(ramp.path()).ok());
122141
MeteredFileReader m(&inner, 16);
123142

124143
std::vector<Range> ranges = {{.offset = 0, .len = 4},
@@ -134,8 +153,9 @@ TEST(SniiMeteredFileReader, BatchAdjacentCoalesced) {
134153

135154
// reset_metrics clears both counters and the resident cache (cold query).
136155
TEST(SniiMeteredFileReader, ResetClearsCacheAndCounters) {
156+
const RampFile ramp;
137157
LocalFileReader inner;
138-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
158+
ASSERT_TRUE(inner.open(ramp.path()).ok());
139159
MeteredFileReader m(&inner, 16);
140160

141161
std::vector<uint8_t> out;
@@ -150,8 +170,9 @@ TEST(SniiMeteredFileReader, ResetClearsCacheAndCounters) {
150170
}
151171

152172
TEST(SniiMeteredFileReader, InvalidRangeDoesNotPolluteMetrics) {
173+
const RampFile ramp;
153174
LocalFileReader inner;
154-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
175+
ASSERT_TRUE(inner.open(ramp.path()).ok());
155176
MeteredFileReader m(&inner, 16);
156177

157178
std::vector<uint8_t> out;
@@ -165,8 +186,9 @@ TEST(SniiMeteredFileReader, InvalidRangeDoesNotPolluteMetrics) {
165186
}
166187

167188
TEST(SniiMeteredFileReader, InvalidBatchRangeDoesNotPolluteMetrics) {
189+
const RampFile ramp;
168190
LocalFileReader inner;
169-
ASSERT_TRUE(inner.open(MakeRampFile()).ok());
191+
ASSERT_TRUE(inner.open(ramp.path()).ok());
170192
MeteredFileReader m(&inner, 16);
171193

172194
std::vector<std::vector<uint8_t>> outs;

‎be/test/storage/index/snii/writer/spimi_term_buffer_test.cpp‎

Lines changed: 36 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131

3232
#include "common/status.h"
3333
#include "storage/index/snii/writer/term_posting_test_utils.h"
34+
#include "util/defer_op.h"
3435

3536
using doris::snii::writer::MemoryReporter;
3637
using doris::snii::writer::SpimiTermBuffer;
@@ -514,34 +515,46 @@ TEST(SniiSpimiTermBuffer, SpillOpenIoFailureLatched) {
514515
// Tiny threshold so the very first token triggers a spill_to_run().
515516
SpimiTermBuffer buf(/*has_positions=*/false, /*spill_threshold_bytes=*/1);
516517

517-
// Cap the soft limit low so we can exhaust the fd table cheaply, then hold it.
518518
struct rlimit saved {};
519519
ASSERT_EQ(getrlimit(RLIMIT_NOFILE, &saved), 0);
520-
struct rlimit tight = saved;
521-
tight.rlim_cur = 64; // small, but >= the few gtest/std fds already open
522-
tight.rlim_cur = std::min(tight.rlim_cur, saved.rlim_max);
523-
ASSERT_EQ(setrlimit(RLIMIT_NOFILE, &tight), 0);
524-
525-
// Open /dev/null until the table is full: every free fd below the limit is now
526-
// taken, so the next ::open (the spill's) cannot get one -> EMFILE.
527-
std::vector<int> hogs;
528-
for (;;) {
529-
int fd = ::open("/dev/null", O_RDONLY);
530-
if (fd < 0) {
531-
break; // table exhausted
532-
}
533-
hogs.push_back(fd);
534-
}
535-
ASSERT_FALSE(hogs.empty());
520+
{
521+
std::vector<int> hogs;
522+
// Restore on EVERY exit path, including a fatal assertion below. Leaving the
523+
// process at the tight limit would break every later test in the binary, and
524+
// gtest's death tests would abort the whole run when their pipe() hits EMFILE.
525+
doris::Defer restore([&] {
526+
for (int fd : hogs) {
527+
::close(fd);
528+
}
529+
EXPECT_EQ(setrlimit(RLIMIT_NOFILE, &saved), 0);
530+
});
536531

537-
buf.add_token("z", 0, 0); // triggers a spill whose RunWriter::open must fail
532+
// Cap just above the descriptors this process already holds instead of a fixed
533+
// number: how many are open depends on which tests ran first, and a cap below
534+
// that leaves no free descriptor for the hog loop to take. dup() returns the
535+
// lowest free descriptor, so every number under the cap is still available.
536+
const int lowest_free = ::dup(STDIN_FILENO);
537+
ASSERT_GE(lowest_free, 0);
538+
ASSERT_EQ(::close(lowest_free), 0);
539+
struct rlimit tight = saved;
540+
tight.rlim_cur = std::min<rlim_t>(static_cast<rlim_t>(lowest_free) + 8, saved.rlim_max);
541+
ASSERT_EQ(setrlimit(RLIMIT_NOFILE, &tight), 0);
542+
543+
// Open /dev/null until the table is full: every free fd below the limit is now
544+
// taken, so the next ::open (the spill's) cannot get one -> EMFILE.
545+
for (;;) {
546+
int fd = ::open("/dev/null", O_RDONLY);
547+
if (fd < 0) {
548+
break; // table exhausted
549+
}
550+
hogs.push_back(fd);
551+
}
552+
ASSERT_FALSE(hogs.empty());
538553

539-
// Release the hog fds and restore the limit before asserting (so gtest I/O works).
540-
for (int fd : hogs) {
541-
::close(fd);
554+
buf.add_token("z", 0, 0); // triggers a spill whose RunWriter::open must fail
542555
}
543-
ASSERT_EQ(setrlimit(RLIMIT_NOFILE, &saved), 0);
544-
556+
// The hog fds are released and the limit restored here, before asserting, so gtest
557+
// I/O works.
545558
EXPECT_FALSE(buf.status().ok()) << "spill open() failure must latch an error";
546559
}
547560

0 commit comments

Comments
 (0)