[Python] Exercise fallback case on tests for parquet.read_table in case dataset is not available

Describe the enhancement requested

While reviewing:

GH-46374: [Python][Docs] Adds type checks for source in read_table #46330

I've realized that the following code path is never tested:

Lines 1807 to 1838 in e2a5b4e

    
           except ImportError: 
        
               # fall back on ParquetFile for simple cases when pyarrow.dataset 
        
               # module is not available 
        
               if filters is not None: 
        
                   raise ValueError( 
        
                       "the 'filters' keyword is not supported when the " 
        
                       "pyarrow.dataset module is not available" 
        
                   ) 
        
               if partitioning != "hive": 
        
                   raise ValueError( 
        
                       "the 'partitioning' keyword is not supported when the " 
        
                       "pyarrow.dataset module is not available" 
        
                   ) 
        
               if schema is not None: 
        
                   raise ValueError( 
        
                       "the 'schema' argument is not supported when the " 
        
                       "pyarrow.dataset module is not available" 
        
                   ) 
        
               filesystem, path = _resolve_filesystem_and_path(source, filesystem) 
        
               if filesystem is not None: 
        
                   source = filesystem.open_input_file(path) 
        
               # TODO test that source is not a directory or a list 
        
               dataset = ParquetFile( 
        
                   source, read_dictionary=read_dictionary, 
        
                   memory_map=memory_map, buffer_size=buffer_size, 
        
                   pre_buffer=pre_buffer, 
        
                   coerce_int96_timestamp_unit=coerce_int96_timestamp_unit, 
        
                   decryption_properties=decryption_properties, 
        
                   thrift_string_size_limit=thrift_string_size_limit, 
        
                   thrift_container_size_limit=thrift_container_size_limit, 
        
                   page_checksum_verification=page_checksum_verification, 
        
               )

I've tried to enable Parquet locally on the minimal Python example build but this would require to build with snappy and others. It would be good to have a test that exercises this code path.

This was tested before due to the use of legacy_dataset flag.
See:

GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation #39112

Component(s)

Python

@AlenkaF

cc @AlenkaF @rok

apacheGH-46373: [Python] Exercise fallback case on tests for parquet.…

GH-46373: [Python] Exercise fallback case on tests for parquet.read_t…

Issue resolved by pull request 46550
#46550

Navigation Menu

Repository navigation

[Python] Exercise fallback case on tests for parquet.read_table in case dataset is not available #46373

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	except ImportError:
	# fall back on ParquetFile for simple cases when pyarrow.dataset
	# module is not available
	if filters is not None:
	raise ValueError(
	"the 'filters' keyword is not supported when the "
	"pyarrow.dataset module is not available"
	)
	if partitioning != "hive":
	raise ValueError(
	"the 'partitioning' keyword is not supported when the "
	"pyarrow.dataset module is not available"
	)
	if schema is not None:
	raise ValueError(
	"the 'schema' argument is not supported when the "
	"pyarrow.dataset module is not available"
	)
	filesystem, path = _resolve_filesystem_and_path(source, filesystem)
	if filesystem is not None:
	source = filesystem.open_input_file(path)
	# TODO test that source is not a directory or a list
	dataset = ParquetFile(
	source, read_dictionary=read_dictionary,
	memory_map=memory_map, buffer_size=buffer_size,
	pre_buffer=pre_buffer,
	coerce_int96_timestamp_unit=coerce_int96_timestamp_unit,
	decryption_properties=decryption_properties,
	thrift_string_size_limit=thrift_string_size_limit,
	thrift_container_size_limit=thrift_container_size_limit,
	page_checksum_verification=page_checksum_verification,
	)

[Python] Exercise fallback case on tests for parquet.read_table in case dataset is not available #46373

Description

Describe the enhancement requested

Component(s)

Activity

raulcd commented on May 9, 2025

raulcd commented on May 26, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions