Repository navigation
Commit 6110f74
authored
fix: avoid pruning timestamp-to-time casts (apache#26452)
## Which issue does this PR close?
- Closes apache#26451.
## Rationale for this change
Casting timestamp statistics to time-of-day values does not preserve
ordering across midnight. This can invert the minimum and maximum used
by statistics pruning and incorrectly discard files or row groups that
contain matching rows.
## What changes are included in this PR?
- Prevent timestamp-to-`Time32` and timestamp-to-`Time64` casts from
being used for statistics pruning.
- Add pruning-expression coverage for both `CAST` and `TRY_CAST` across
all supported time units.
- Add a Parquet regression test whose timestamp bounds cross midnight.
## What is the testing strategy for this PR?
The regression test verifies equality and range predicates for `Time32`
and `Time64`, including the expected row counts and pruning metrics. The
full extended workspace test suite also passes.
## Are there any user-facing changes?
Queries that cast timestamps to time-of-day values may perform less
statistics pruning, but they no longer risk incorrect results when
timestamp bounds cross a day boundary.1 parent 5db4b23 commit 6110f74
3 files changed
Lines changed: 269 additions & 2 deletions
File tree
- datafusion
- core/tests/parquet
- pruning/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
| 10 | + | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
293 | 293 | | |
294 | 294 | | |
295 | 295 | | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
296 | 341 | | |
297 | 342 | | |
298 | 343 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1349 | 1349 | | |
1350 | 1350 | | |
1351 | 1351 | | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
1352 | 1362 | | |
1353 | 1363 | | |
1354 | 1364 | | |
| |||
4971 | 4981 | | |
4972 | 4982 | | |
4973 | 4983 | | |
| 4984 | + | |
| 4985 | + | |
| 4986 | + | |
| 4987 | + | |
| 4988 | + | |
| 4989 | + | |
| 4990 | + | |
| 4991 | + | |
| 4992 | + | |
| 4993 | + | |
| 4994 | + | |
| 4995 | + | |
| 4996 | + | |
| 4997 | + | |
| 4998 | + | |
| 4999 | + | |
| 5000 | + | |
| 5001 | + | |
| 5002 | + | |
| 5003 | + | |
| 5004 | + | |
| 5005 | + | |
| 5006 | + | |
| 5007 | + | |
| 5008 | + | |
| 5009 | + | |
| 5010 | + | |
| 5011 | + | |
| 5012 | + | |
| 5013 | + | |
| 5014 | + | |
| 5015 | + | |
| 5016 | + | |
| 5017 | + | |
| 5018 | + | |
| 5019 | + | |
| 5020 | + | |
| 5021 | + | |
| 5022 | + | |
| 5023 | + | |
| 5024 | + | |
| 5025 | + | |
| 5026 | + | |
| 5027 | + | |
| 5028 | + | |
| 5029 | + | |
| 5030 | + | |
| 5031 | + | |
| 5032 | + | |
| 5033 | + | |
4974 | 5034 | | |
4975 | 5035 | | |
4976 | 5036 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1333 | 1333 | | |
1334 | 1334 | | |
1335 | 1335 | | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
| 1488 | + | |
| 1489 | + | |
| 1490 | + | |
| 1491 | + | |
| 1492 | + | |
| 1493 | + | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
| 1497 | + | |
1336 | 1498 | | |
1337 | 1499 | | |
1338 | 1500 | | |
| |||
0 commit comments