Skip to content

Commit 2b1fcae

Browse files
authored
chore: Hide test-only internal public APIs (#25695)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> See the previous PR for ratioanle - #25495 This PR adds `#[doc(hidden)]` and comments for test-only public APIs Note in the previous PR we've discussed using feature to hide them, but I tried and find it hard to maintain - #25678 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 9ee892b commit 2b1fcae

10 files changed

Lines changed: 99 additions & 11 deletions

File tree

‎datafusion/catalog/src/memory/table.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,12 @@ pub use datafusion_datasource::memory::PartitionData;
7171
#[derive(Debug)]
7272
pub struct MemTable {
7373
schema: SchemaRef,
74-
// batches used to be pub(crate), but it's needed to be public for the tests
74+
/// # Public Only for Internal Use:
75+
///
76+
/// This is not a public API and is for internal use only; see [API policy] for details.
77+
///
78+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
79+
#[doc(hidden)]
7580
pub batches: Vec<PartitionData>,
7681
constraints: Constraints,
7782
column_defaults: HashMap<String, Expr>,

‎datafusion/core/src/lib.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -855,7 +855,19 @@ pub mod physical_plan {
855855
}
856856

857857
// Reexport testing macros for compatibility
858+
/// # Public Only for Internal Use:
859+
///
860+
/// This is not a public API and is for internal use only; see [API policy] for details.
861+
///
862+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
863+
#[doc(hidden)]
858864
pub use datafusion_common::assert_batches_eq;
865+
/// # Public Only for Internal Use:
866+
///
867+
/// This is not a public API and is for internal use only; see [API policy] for details.
868+
///
869+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
870+
#[doc(hidden)]
859871
pub use datafusion_common::assert_batches_sorted_eq;
860872

861873
/// re-export of [`datafusion_sql`] crate
@@ -895,10 +907,22 @@ pub mod variable {
895907
pub use datafusion_expr::var_provider::{VarProvider, VarType};
896908
}
897909

910+
/// # Public Only for Internal Use:
911+
///
912+
/// This is not a public API and is for internal use only; see [API policy] for details.
913+
///
914+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
898915
#[cfg(not(target_arch = "wasm32"))]
916+
#[doc(hidden)]
899917
pub mod test;
900918

901919
mod schema_equivalence;
920+
/// # Public Only for Internal Use:
921+
///
922+
/// This is not a public API and is for internal use only; see [API policy] for details.
923+
///
924+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
925+
#[doc(hidden)]
902926
pub mod test_util;
903927

904928
#[cfg(doctest)]

‎datafusion/datasource/src/mod.rs‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -413,8 +413,6 @@ impl From<ObjectMeta> for PartitionedFile {
413413

414414
/// Generates test files with min-max statistics in different overlap patterns.
415415
///
416-
/// Used by tests and benchmarks.
417-
///
418416
/// # Overlap Factors
419417
///
420418
/// The `overlap_factor` parameter controls how much the value ranges in generated test files overlap:
@@ -450,6 +448,13 @@ impl From<ObjectMeta> for PartitionedFile {
450448
/// File 2: [40, 140]
451449
/// File 3: [60, 160]
452450
/// File 4: [80, 180]
451+
///
452+
/// # Public Only for Internal Use:
453+
///
454+
/// This is not a public API and is for internal use only; see [API policy] for details.
455+
///
456+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
457+
#[doc(hidden)]
453458
pub fn generate_test_files(num_files: usize, overlap_factor: f64) -> Vec<FileGroup> {
454459
let mut files = Vec::with_capacity(num_files);
455460
if num_files == 0 {
@@ -500,8 +505,14 @@ pub fn generate_test_files(num_files: usize, overlap_factor: f64) -> Vec<FileGro
500505
vec![FileGroup::new(files)]
501506
}
502507

503-
// Helper function to verify that files within each group maintain sort order
504-
/// Used by tests and benchmarks
508+
/// Verifies that files within each group maintain sort order.
509+
///
510+
/// # Public Only for Internal Use:
511+
///
512+
/// This is not a public API and is for internal use only; see [API policy] for details.
513+
///
514+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
515+
#[doc(hidden)]
505516
pub fn verify_sort_integrity(file_groups: &[FileGroup]) -> bool {
506517
for group in file_groups {
507518
// Known-empty files contribute no rows and may not have min/max

‎datafusion/expr/src/lib.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,12 @@ pub mod preimage;
8888
pub mod ptr_eq;
8989
#[cfg(not(feature = "sql"))]
9090
pub mod sql;
91+
/// # Public Only for Internal Use:
92+
///
93+
/// This is not a public API and is for internal use only; see [API policy] for details.
94+
///
95+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
96+
#[doc(hidden)]
9197
pub mod test;
9298
pub mod tree_node;
9399
pub mod type_coercion;

‎datafusion/functions-aggregate-common/src/tdigest.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,12 @@ impl TDigest {
206206
v.clamp(min, max)
207207
}
208208

209-
// public for testing in other modules
209+
/// # Public Only for Internal Use:
210+
///
211+
/// This is not a public API and is for internal use only; see [API policy] for details.
212+
///
213+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
214+
#[doc(hidden)]
210215
pub fn merge_unsorted_f64(&self, unsorted_values: Vec<f64>) -> TDigest {
211216
let mut values = unsorted_values;
212217
values.sort_by(|a, b| a.total_cmp(b));

‎datafusion/physical-expr/src/intervals/mod.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,11 @@
1818
//! Interval arithmetic and constraint propagation library
1919
2020
pub mod cp_solver;
21+
/// # Public Only for Internal Use:
22+
///
23+
/// This is not a public API and is for internal use only; see [API policy] for details.
24+
///
25+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
26+
#[doc(hidden)]
2127
pub mod test_utils;
2228
pub mod utils;

‎datafusion/physical-plan/src/joins/hash_join/mod.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,14 @@
1818
//! [`HashJoinExec`] Partitioned Hash Join Operator
1919
2020
pub use exec::{HashJoinExec, HashJoinExecBuilder, PreparedHashJoinBuild};
21-
pub use partitioned_hash_eval::{HashExpr, HashTableLookupExpr, SeededRandomState};
21+
/// # Public Only for Internal Use:
22+
///
23+
/// This is not a public API and is for internal use only; see [API policy] for details.
24+
///
25+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
26+
#[doc(hidden)]
27+
pub use partitioned_hash_eval::HashTableLookupExpr;
28+
pub use partitioned_hash_eval::{HashExpr, SeededRandomState};
2229

2330
mod exec;
2431
mod inlist_builder;

‎datafusion/physical-plan/src/joins/mod.rs‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,15 @@ use arrow::array::BooleanBufferBuilder;
2424
pub use asof_join::{AsOfJoinExec, AsOfMatchExpr};
2525
pub use cross_join::CrossJoinExec;
2626
use datafusion_physical_expr::PhysicalExprRef;
27+
/// # Public Only for Internal Use:
28+
///
29+
/// This is not a public API and is for internal use only; see [API policy] for details.
30+
///
31+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
32+
#[doc(hidden)]
33+
pub use hash_join::HashTableLookupExpr;
2734
pub use hash_join::{
28-
HashExpr, HashJoinExec, HashJoinExecBuilder, HashTableLookupExpr,
29-
PreparedHashJoinBuild, SeededRandomState,
35+
HashExpr, HashJoinExec, HashJoinExecBuilder, PreparedHashJoinBuild, SeededRandomState,
3036
};
3137
pub use nested_loop_join::{NestedLoopJoinExec, NestedLoopJoinExecBuilder};
3238
use parking_lot::Mutex;
@@ -52,8 +58,12 @@ mod array_map;
5258
mod join_filter;
5359
/// Hash map implementations for join operations.
5460
///
55-
/// Note: This module is public for internal testing purposes only
56-
/// and is not guaranteed to be stable across versions.
61+
/// # Public Only for Internal Use:
62+
///
63+
/// This is not a public API and is for internal use only; see [API policy] for details.
64+
///
65+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
66+
#[doc(hidden)]
5767
pub mod join_hash_map;
5868

5969
use array_map::ArrayMap;

‎datafusion/physical-plan/src/lib.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,4 +115,10 @@ pub mod udaf {
115115
pub use datafusion_physical_expr::aggregate::AggregateFunctionExpr;
116116
}
117117

118+
/// # Public Only for Internal Use:
119+
///
120+
/// This is not a public API and is for internal use only; see [API policy] for details.
121+
///
122+
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
123+
#[doc(hidden)]
118124
pub mod test;

‎docs/source/contributor-guide/api-health.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,10 @@ Examples of non-breaking changes include:
4343
- Marking a function as deprecated (`#[deprecated]`)
4444
- Adding a new function to a `trait` with a default implementation
4545

46+
<!-- Keep this anchor stable: it is linked from Rust API documentation. -->
47+
48+
<a id="internal-public-apis"></a>
49+
4650
### DataFusion-internal Public APIs
4751

4852
Some internal implementation details require `pub` visibility because they are
@@ -55,6 +59,10 @@ notice or a deprecation period. Examples include:
5559
2. Operator APIs required by the optimizer to inspect or rewrite execution plans
5660
across crate boundaries.
5761

62+
Do not expose internal APIs solely for tests or microbenchmarks. Some legacy code
63+
does so, but new tests and benchmarks should exercise observable behavior to
64+
simplify maintenance.
65+
5866
For APIs intended only for internal use, add `#[doc(hidden)]` and a doc comment
5967
section headed `# Public Only for Internal Use:`. Name the crate or component
6068
that requires access and explain why the API is not intended for downstream use.

0 commit comments

Comments
 (0)