PySpark API: Overview to track coverage #14525

binste · 2024-10-24T07:12:17Z

binste
Oct 24, 2024

I think that the Spark API is the best candidate right now for a Python DataFrame API for DuckDB. Also, would be very powerful if it could be used as a drop-in replacement for existing Spark code. See this MotherDuck blog post for more details.

There is already a very well-implemented foundation and some functions, data types, and classes are in place. It's now about building out the remaining functions and methods to get to a high coverage. As there are several PRs open, I thought it might be helpful to track the progress in one place to ease collaboration.

Looking forward to turning many of these red crosses into green ticks! 🥳

Functions

Code to update the table below

!pip install tabulate pandas requests

import re
from pathlib import Path

import pandas as pd
import requests

DUCKDB_BRANCH = "main"

spark_f_code = requests.get("https://raw.githubusercontent.com/apache/spark/refs/heads/master/python/pyspark/sql/functions/builtin.py").text

duckdb_f_code = requests.get(f"https://raw.githubusercontent.com/duckdb/duckdb/refs/heads/{DUCKDB_BRANCH}/tools/pythonpkg/duckdb/experimental/spark/sql/functions.py").text

def extract_functions(source_code: str) -> list[str]:
    functions = []
    for l in source_code.splitlines():
        m = re.search(r"^def ([^(]+)\(", l)
        if m and not (name := m.group(1)).startswith("_"):
            functions.append(m.group(1))
    return functions

spark_functions = extract_functions(spark_f_code)
duck_functions = extract_functions(duckdb_f_code)
coverage_raw = []
for f in spark_functions:
    coverage_raw.append((f, f in duck_functions))
coverage = pd.DataFrame(coverage_raw, columns=["Function", "Implemented in DuckDB"])
coverage = coverage.drop_duplicates()  # They exist due to overloaded function definitions

print("Coverage:", format(coverage["Implemented in DuckDB"].mean(), ".0%"))
coverage["Implemented in DuckDB"] = coverage["Implemented in DuckDB"].map({True: "✅", False: "❌"})
coverage["Spark URL"] = coverage["Function"].apply(lambda x: f"[{x} - Spark Docs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.{x}.html)")

coverage.sort_values("Function").to_markdown("coverage.md", index=False)

PRs which are currently open and implement some of the functions below:

...

Percentage of functions implemented in DuckDB (main or feature, depending on which one is active):

24.10: 14%
08.11: 22%
19.11: 33%
30.11: 40%
21.03.2025: 40%

Function	Implemented in DuckDB	Spark URL
abs	✅	abs - Spark Docs
acos	✅	acos - Spark Docs
acosh	❌	acosh - Spark Docs
add_months	✅	add_months - Spark Docs
aes_decrypt	❌	aes_decrypt - Spark Docs
aes_encrypt	❌	aes_encrypt - Spark Docs
aggregate	❌	aggregate - Spark Docs
any_value	✅	any_value - Spark Docs
approxCountDistinct	✅	approxCountDistinct - Spark Docs
approx_count_distinct	✅	approx_count_distinct - Spark Docs
approx_percentile	❌	approx_percentile - Spark Docs
array	✅	array - Spark Docs
array_agg	✅	array_agg - Spark Docs
array_append	✅	array_append - Spark Docs
array_compact	✅	array_compact - Spark Docs
array_contains	✅	array_contains - Spark Docs
array_distinct	✅	array_distinct - Spark Docs
array_except	❌	array_except - Spark Docs
array_insert	✅	array_insert - Spark Docs
array_intersect	✅	array_intersect - Spark Docs
array_join	✅	array_join - Spark Docs
array_max	✅	array_max - Spark Docs
array_min	✅	array_min - Spark Docs
array_position	✅	array_position - Spark Docs
array_prepend	✅	array_prepend - Spark Docs
array_remove	✅	array_remove - Spark Docs
array_repeat	✅	array_repeat - Spark Docs
array_size	✅	array_size - Spark Docs
array_sort	✅	array_sort - Spark Docs
array_union	✅	array_union - Spark Docs
arrays_overlap	✅	arrays_overlap - Spark Docs
arrays_zip	✅	arrays_zip - Spark Docs
asc	✅	asc - Spark Docs
asc_nulls_first	✅	asc_nulls_first - Spark Docs
asc_nulls_last	✅	asc_nulls_last - Spark Docs
ascii	✅	ascii - Spark Docs
asin	✅	asin - Spark Docs
asinh	❌	asinh - Spark Docs
assert_true	❌	assert_true - Spark Docs
atan	✅	atan - Spark Docs
atan2	✅	atan2 - Spark Docs
atanh	❌	atanh - Spark Docs
avg	✅	avg - Spark Docs
base64	✅	base64 - Spark Docs
bin	❌	bin - Spark Docs
bit_and	❌	bit_and - Spark Docs
bit_count	❌	bit_count - Spark Docs
bit_get	❌	bit_get - Spark Docs
bit_length	❌	bit_length - Spark Docs
bit_or	❌	bit_or - Spark Docs
bit_xor	❌	bit_xor - Spark Docs
bitmap_bit_position	❌	bitmap_bit_position - Spark Docs
bitmap_bucket_number	❌	bitmap_bucket_number - Spark Docs
bitmap_construct_agg	❌	bitmap_construct_agg - Spark Docs
bitmap_count	❌	bitmap_count - Spark Docs
bitmap_or_agg	❌	bitmap_or_agg - Spark Docs
bitwiseNOT	❌	bitwiseNOT - Spark Docs
bitwise_not	❌	bitwise_not - Spark Docs
bool_and	❌	bool_and - Spark Docs
bool_or	❌	bool_or - Spark Docs
broadcast	✅	broadcast - Spark Docs
bround	✅	bround - Spark Docs
btrim	✅	btrim - Spark Docs
bucket	❌	bucket - Spark Docs
call_function	✅	call_function - Spark Docs
call_udf	❌	call_udf - Spark Docs
cardinality	❌	cardinality - Spark Docs
cbrt	✅	cbrt - Spark Docs
ceil	✅	ceil - Spark Docs
ceiling	✅	ceiling - Spark Docs
char	✅	char - Spark Docs
char_length	❌	char_length - Spark Docs
character_length	❌	character_length - Spark Docs
coalesce	✅	coalesce - Spark Docs
col	✅	col - Spark Docs
collate	❌	collate - Spark Docs
collation	❌	collation - Spark Docs
collect_list	✅	collect_list - Spark Docs
collect_set	❌	collect_set - Spark Docs
concat	✅	concat - Spark Docs
concat_ws	✅	concat_ws - Spark Docs
contains	✅	contains - Spark Docs
conv	❌	conv - Spark Docs
convert_timezone	❌	convert_timezone - Spark Docs
corr	✅	corr - Spark Docs
cos	✅	cos - Spark Docs
cosh	❌	cosh - Spark Docs
cot	✅	cot - Spark Docs
count	✅	count - Spark Docs
countDistinct	❌	countDistinct - Spark Docs
count_distinct	❌	count_distinct - Spark Docs
count_if	❌	count_if - Spark Docs
count_min_sketch	❌	count_min_sketch - Spark Docs
covar_pop	✅	covar_pop - Spark Docs
covar_samp	✅	covar_samp - Spark Docs
crc32	❌	crc32 - Spark Docs
create_map	❌	create_map - Spark Docs
csc	❌	csc - Spark Docs
cume_dist	❌	cume_dist - Spark Docs
curdate	✅	curdate - Spark Docs
current_catalog	❌	current_catalog - Spark Docs
current_database	❌	current_database - Spark Docs
current_date	✅	current_date - Spark Docs
current_schema	❌	current_schema - Spark Docs
current_timestamp	❌	current_timestamp - Spark Docs
current_timezone	❌	current_timezone - Spark Docs
current_user	❌	current_user - Spark Docs
date_add	❌	date_add - Spark Docs
date_diff	❌	date_diff - Spark Docs
date_format	❌	date_format - Spark Docs
date_from_unix_date	❌	date_from_unix_date - Spark Docs
date_part	✅	date_part - Spark Docs
date_sub	❌	date_sub - Spark Docs
date_trunc	✅	date_trunc - Spark Docs
dateadd	❌	dateadd - Spark Docs
datediff	❌	datediff - Spark Docs
datepart	✅	datepart - Spark Docs
day	✅	day - Spark Docs
dayname	❌	dayname - Spark Docs
dayofmonth	✅	dayofmonth - Spark Docs
dayofweek	✅	dayofweek - Spark Docs
dayofyear	✅	dayofyear - Spark Docs
days	❌	days - Spark Docs
decode	❌	decode - Spark Docs
degrees	✅	degrees - Spark Docs
dense_rank	❌	dense_rank - Spark Docs
desc	✅	desc - Spark Docs
desc_nulls_first	✅	desc_nulls_first - Spark Docs
desc_nulls_last	✅	desc_nulls_last - Spark Docs
e	✅	e - Spark Docs
element_at	❌	element_at - Spark Docs
elt	❌	elt - Spark Docs
encode	✅	encode - Spark Docs
endswith	✅	endswith - Spark Docs
equal_null	❌	equal_null - Spark Docs
every	❌	every - Spark Docs
exists	❌	exists - Spark Docs
exp	✅	exp - Spark Docs
explode	❌	explode - Spark Docs
explode_outer	❌	explode_outer - Spark Docs
expm1	❌	expm1 - Spark Docs
expr	✅	expr - Spark Docs
extract	✅	extract - Spark Docs
factorial	✅	factorial - Spark Docs
filter	❌	filter - Spark Docs
find_in_set	✅	find_in_set - Spark Docs
first	✅	first - Spark Docs
first_value	❌	first_value - Spark Docs
flatten	✅	flatten - Spark Docs
floor	✅	floor - Spark Docs
forall	❌	forall - Spark Docs
format_number	❌	format_number - Spark Docs
format_string	❌	format_string - Spark Docs
from_csv	❌	from_csv - Spark Docs
from_json	❌	from_json - Spark Docs
from_unixtime	❌	from_unixtime - Spark Docs
from_utc_timestamp	❌	from_utc_timestamp - Spark Docs
from_xml	❌	from_xml - Spark Docs
get	✅	get - Spark Docs
get_json_object	❌	get_json_object - Spark Docs
getbit	❌	getbit - Spark Docs
greatest	✅	greatest - Spark Docs
grouping	❌	grouping - Spark Docs
grouping_id	❌	grouping_id - Spark Docs
hash	❌	hash - Spark Docs
hex	✅	hex - Spark Docs
histogram_numeric	❌	histogram_numeric - Spark Docs
hll_sketch_agg	❌	hll_sketch_agg - Spark Docs
hll_sketch_estimate	❌	hll_sketch_estimate - Spark Docs
hll_union	❌	hll_union - Spark Docs
hll_union_agg	❌	hll_union_agg - Spark Docs
hour	✅	hour - Spark Docs
hours	❌	hours - Spark Docs
hypot	❌	hypot - Spark Docs
ifnull	✅	ifnull - Spark Docs
ilike	✅	ilike - Spark Docs
initcap	✅	initcap - Spark Docs
inline	❌	inline - Spark Docs
inline_outer	❌	inline_outer - Spark Docs
input_file_block_length	❌	input_file_block_length - Spark Docs
input_file_block_start	❌	input_file_block_start - Spark Docs
input_file_name	❌	input_file_name - Spark Docs
instr	✅	instr - Spark Docs
is_valid_utf8	❌	is_valid_utf8 - Spark Docs
is_variant_null	❌	is_variant_null - Spark Docs
isnan	✅	isnan - Spark Docs
isnotnull	✅	isnotnull - Spark Docs
isnull	✅	isnull - Spark Docs
java_method	❌	java_method - Spark Docs
json_array_length	❌	json_array_length - Spark Docs
json_object_keys	❌	json_object_keys - Spark Docs
json_tuple	❌	json_tuple - Spark Docs
kurtosis	❌	kurtosis - Spark Docs
lag	❌	lag - Spark Docs
last	✅	last - Spark Docs
last_day	✅	last_day - Spark Docs
last_value	❌	last_value - Spark Docs
lcase	✅	lcase - Spark Docs
lead	❌	lead - Spark Docs
least	✅	least - Spark Docs
left	✅	left - Spark Docs
length	✅	length - Spark Docs
levenshtein	✅	levenshtein - Spark Docs
like	✅	like - Spark Docs
listagg	❌	listagg - Spark Docs
listagg_distinct	❌	listagg_distinct - Spark Docs
lit	✅	lit - Spark Docs
ln	✅	ln - Spark Docs
localtimestamp	❌	localtimestamp - Spark Docs
locate	❌	locate - Spark Docs
log	❌	log - Spark Docs
log10	❌	log10 - Spark Docs
log1p	❌	log1p - Spark Docs
log2	✅	log2 - Spark Docs
lower	✅	lower - Spark Docs
lpad	✅	lpad - Spark Docs
ltrim	✅	ltrim - Spark Docs
make_date	❌	make_date - Spark Docs
make_dt_interval	❌	make_dt_interval - Spark Docs
make_interval	❌	make_interval - Spark Docs
make_timestamp	❌	make_timestamp - Spark Docs
make_timestamp_ltz	❌	make_timestamp_ltz - Spark Docs
make_timestamp_ntz	❌	make_timestamp_ntz - Spark Docs
make_valid_utf8	❌	make_valid_utf8 - Spark Docs
make_ym_interval	❌	make_ym_interval - Spark Docs
map_concat	❌	map_concat - Spark Docs
map_contains_key	❌	map_contains_key - Spark Docs
map_entries	❌	map_entries - Spark Docs
map_filter	❌	map_filter - Spark Docs
map_from_arrays	❌	map_from_arrays - Spark Docs
map_from_entries	❌	map_from_entries - Spark Docs
map_keys	❌	map_keys - Spark Docs
map_values	❌	map_values - Spark Docs
map_zip_with	❌	map_zip_with - Spark Docs
mask	❌	mask - Spark Docs
max	✅	max - Spark Docs
max_by	❌	max_by - Spark Docs
md5	✅	md5 - Spark Docs
mean	✅	mean - Spark Docs
median	✅	median - Spark Docs
min	✅	min - Spark Docs
min_by	❌	min_by - Spark Docs
minute	✅	minute - Spark Docs
mode	✅	mode - Spark Docs
monotonically_increasing_id	❌	monotonically_increasing_id - Spark Docs
month	✅	month - Spark Docs
monthname	❌	monthname - Spark Docs
months	❌	months - Spark Docs
months_between	❌	months_between - Spark Docs
named_struct	❌	named_struct - Spark Docs
nanvl	❌	nanvl - Spark Docs
negative	❌	negative - Spark Docs
next_day	❌	next_day - Spark Docs
now	✅	now - Spark Docs
nth_value	❌	nth_value - Spark Docs
ntile	❌	ntile - Spark Docs
nullif	✅	nullif - Spark Docs
nullifzero	❌	nullifzero - Spark Docs
nvl	✅	nvl - Spark Docs
nvl2	✅	nvl2 - Spark Docs
octet_length	✅	octet_length - Spark Docs
overlay	❌	overlay - Spark Docs
parse_json	❌	parse_json - Spark Docs
parse_url	❌	parse_url - Spark Docs
percent_rank	❌	percent_rank - Spark Docs
percentile	❌	percentile - Spark Docs
percentile_approx	❌	percentile_approx - Spark Docs
pi	✅	pi - Spark Docs
pmod	❌	pmod - Spark Docs
posexplode	❌	posexplode - Spark Docs
posexplode_outer	❌	posexplode_outer - Spark Docs
position	❌	position - Spark Docs
positive	✅	positive - Spark Docs
pow	✅	pow - Spark Docs
printf	✅	printf - Spark Docs
product	✅	product - Spark Docs
quarter	✅	quarter - Spark Docs
quote	❌	quote - Spark Docs
radians	✅	radians - Spark Docs
raise_error	❌	raise_error - Spark Docs
rand	✅	rand - Spark Docs
randn	❌	randn - Spark Docs
randstr	❌	randstr - Spark Docs
rank	❌	rank - Spark Docs
reduce	❌	reduce - Spark Docs
reflect	❌	reflect - Spark Docs
regexp	✅	regexp - Spark Docs
regexp_count	✅	regexp_count - Spark Docs
regexp_extract	✅	regexp_extract - Spark Docs
regexp_extract_all	✅	regexp_extract_all - Spark Docs
regexp_instr	❌	regexp_instr - Spark Docs
regexp_like	✅	regexp_like - Spark Docs
regexp_replace	✅	regexp_replace - Spark Docs
regexp_substr	✅	regexp_substr - Spark Docs
regr_avgx	❌	regr_avgx - Spark Docs
regr_avgy	❌	regr_avgy - Spark Docs
regr_count	❌	regr_count - Spark Docs
regr_intercept	❌	regr_intercept - Spark Docs
regr_r2	❌	regr_r2 - Spark Docs
regr_slope	❌	regr_slope - Spark Docs
regr_sxx	❌	regr_sxx - Spark Docs
regr_sxy	❌	regr_sxy - Spark Docs
regr_syy	❌	regr_syy - Spark Docs
repeat	✅	repeat - Spark Docs
replace	❌	replace - Spark Docs
reverse	✅	reverse - Spark Docs
right	✅	right - Spark Docs
rint	❌	rint - Spark Docs
rlike	❌	rlike - Spark Docs
round	✅	round - Spark Docs
row_number	❌	row_number - Spark Docs
rpad	✅	rpad - Spark Docs
rtrim	✅	rtrim - Spark Docs
schema_of_csv	❌	schema_of_csv - Spark Docs
schema_of_json	❌	schema_of_json - Spark Docs
schema_of_variant	❌	schema_of_variant - Spark Docs
schema_of_variant_agg	❌	schema_of_variant_agg - Spark Docs
schema_of_xml	❌	schema_of_xml - Spark Docs
sec	❌	sec - Spark Docs
second	✅	second - Spark Docs
sentences	❌	sentences - Spark Docs
sequence	✅	sequence - Spark Docs
session_user	❌	session_user - Spark Docs
session_window	❌	session_window - Spark Docs
sha	❌	sha - Spark Docs
sha1	❌	sha1 - Spark Docs
sha2	✅	sha2 - Spark Docs
shiftLeft	❌	shiftLeft - Spark Docs
shiftRight	❌	shiftRight - Spark Docs
shiftRightUnsigned	❌	shiftRightUnsigned - Spark Docs
shiftleft	❌	shiftleft - Spark Docs
shiftright	❌	shiftright - Spark Docs
shiftrightunsigned	❌	shiftrightunsigned - Spark Docs
shuffle	❌	shuffle - Spark Docs
sign	✅	sign - Spark Docs
signum	✅	signum - Spark Docs
sin	✅	sin - Spark Docs
sinh	❌	sinh - Spark Docs
size	❌	size - Spark Docs
skewness	✅	skewness - Spark Docs
slice	✅	slice - Spark Docs
some	❌	some - Spark Docs
sort_array	✅	sort_array - Spark Docs
soundex	❌	soundex - Spark Docs
spark_partition_id	❌	spark_partition_id - Spark Docs
split	✅	split - Spark Docs
split_part	✅	split_part - Spark Docs
sqrt	✅	sqrt - Spark Docs
stack	❌	stack - Spark Docs
startswith	✅	startswith - Spark Docs
std	✅	std - Spark Docs
stddev	✅	stddev - Spark Docs
stddev_pop	✅	stddev_pop - Spark Docs
stddev_samp	✅	stddev_samp - Spark Docs
str_to_map	❌	str_to_map - Spark Docs
string_agg	❌	string_agg - Spark Docs
string_agg_distinct	❌	string_agg_distinct - Spark Docs
struct	✅	struct - Spark Docs
substr	✅	substr - Spark Docs
substring	✅	substring - Spark Docs
substring_index	❌	substring_index - Spark Docs
sum	✅	sum - Spark Docs
sumDistinct	❌	sumDistinct - Spark Docs
sum_distinct	❌	sum_distinct - Spark Docs
tan	✅	tan - Spark Docs
tanh	❌	tanh - Spark Docs
timestamp_add	❌	timestamp_add - Spark Docs
timestamp_diff	❌	timestamp_diff - Spark Docs
timestamp_micros	❌	timestamp_micros - Spark Docs
timestamp_millis	❌	timestamp_millis - Spark Docs
timestamp_seconds	❌	timestamp_seconds - Spark Docs
toDegrees	❌	toDegrees - Spark Docs
toRadians	❌	toRadians - Spark Docs
to_binary	❌	to_binary - Spark Docs
to_char	❌	to_char - Spark Docs
to_csv	❌	to_csv - Spark Docs
to_date	✅	to_date - Spark Docs
to_json	❌	to_json - Spark Docs
to_number	❌	to_number - Spark Docs
to_timestamp	✅	to_timestamp - Spark Docs
to_timestamp_ltz	✅	to_timestamp_ltz - Spark Docs
to_timestamp_ntz	✅	to_timestamp_ntz - Spark Docs
to_unix_timestamp	❌	to_unix_timestamp - Spark Docs
to_utc_timestamp	❌	to_utc_timestamp - Spark Docs
to_varchar	❌	to_varchar - Spark Docs
to_variant_object	❌	to_variant_object - Spark Docs
to_xml	❌	to_xml - Spark Docs
transform	✅	transform - Spark Docs
transform_keys	❌	transform_keys - Spark Docs
transform_values	❌	transform_values - Spark Docs
translate	❌	translate - Spark Docs
trim	✅	trim - Spark Docs
trunc	❌	trunc - Spark Docs
try_add	❌	try_add - Spark Docs
try_aes_decrypt	❌	try_aes_decrypt - Spark Docs
try_avg	❌	try_avg - Spark Docs
try_divide	❌	try_divide - Spark Docs
try_element_at	❌	try_element_at - Spark Docs
try_make_interval	❌	try_make_interval - Spark Docs
try_make_timestamp	❌	try_make_timestamp - Spark Docs
try_make_timestamp_ltz	❌	try_make_timestamp_ltz - Spark Docs
try_make_timestamp_ntz	❌	try_make_timestamp_ntz - Spark Docs
try_mod	❌	try_mod - Spark Docs
try_multiply	❌	try_multiply - Spark Docs
try_parse_json	❌	try_parse_json - Spark Docs
try_parse_url	❌	try_parse_url - Spark Docs
try_reflect	❌	try_reflect - Spark Docs
try_subtract	❌	try_subtract - Spark Docs
try_sum	❌	try_sum - Spark Docs
try_to_binary	❌	try_to_binary - Spark Docs
try_to_number	❌	try_to_number - Spark Docs
try_to_timestamp	❌	try_to_timestamp - Spark Docs
try_url_decode	❌	try_url_decode - Spark Docs
try_validate_utf8	❌	try_validate_utf8 - Spark Docs
try_variant_get	❌	try_variant_get - Spark Docs
typeof	❌	typeof - Spark Docs
ucase	✅	ucase - Spark Docs
udf	❌	udf - Spark Docs
udtf	❌	udtf - Spark Docs
unbase64	✅	unbase64 - Spark Docs
unhex	✅	unhex - Spark Docs
uniform	❌	uniform - Spark Docs
unix_date	✅	unix_date - Spark Docs
unix_micros	✅	unix_micros - Spark Docs
unix_millis	✅	unix_millis - Spark Docs
unix_seconds	✅	unix_seconds - Spark Docs
unix_timestamp	❌	unix_timestamp - Spark Docs
unwrap_udt	❌	unwrap_udt - Spark Docs
upper	✅	upper - Spark Docs
url_decode	❌	url_decode - Spark Docs
url_encode	❌	url_encode - Spark Docs
user	❌	user - Spark Docs
validate_utf8	❌	validate_utf8 - Spark Docs
var_pop	✅	var_pop - Spark Docs
var_samp	✅	var_samp - Spark Docs
variance	✅	variance - Spark Docs
variant_get	❌	variant_get - Spark Docs
version	❌	version - Spark Docs
weekday	✅	weekday - Spark Docs
weekofyear	✅	weekofyear - Spark Docs
when	✅	when - Spark Docs
width_bucket	❌	width_bucket - Spark Docs
window	❌	window - Spark Docs
window_time	❌	window_time - Spark Docs
xpath	❌	xpath - Spark Docs
xpath_boolean	❌	xpath_boolean - Spark Docs
xpath_double	❌	xpath_double - Spark Docs
xpath_float	❌	xpath_float - Spark Docs
xpath_int	❌	xpath_int - Spark Docs
xpath_long	❌	xpath_long - Spark Docs
xpath_number	❌	xpath_number - Spark Docs
xpath_short	❌	xpath_short - Spark Docs
xpath_string	❌	xpath_string - Spark Docs
xxhash64	❌	xxhash64 - Spark Docs
year	✅	year - Spark Docs
years	❌	years - Spark Docs
zeroifnull	✅	zeroifnull - Spark Docs
zip_with	❌	zip_with - Spark Docs

DataFrame methods and other things

...

Maxxen · 2024-10-24T07:18:04Z

Maxxen
Oct 24, 2024
Collaborator

Great initiative!

0 replies

spenrose · 2024-11-05T21:36:31Z

spenrose
Nov 5, 2024

base64/unbase64 and hex/unhex have been merged

1 reply

binste Nov 7, 2024
Author

Thanks! I've updated the overview

keen85 · 2025-04-22T17:42:40Z

keen85
Apr 22, 2025

Am I late to the party? 😂
I'd love to use DuckDB to run my PySpark unittests - and maybe even some small/medium data workloads.

Can someone help me get going?
https://discord.com/channels/909674491309850675/1363859773920907444

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PySpark API: Overview to track coverage #14525

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

PySpark API: Overview to track coverage #14525

Uh oh!

Uh oh!

binste Oct 24, 2024

Functions

DataFrame methods and other things

Replies: 3 comments · 1 reply

Uh oh!

Maxxen Oct 24, 2024 Collaborator

Uh oh!

spenrose Nov 5, 2024

Uh oh!

binste Nov 7, 2024 Author

Uh oh!

keen85 Apr 22, 2025

binste
Oct 24, 2024

Replies: 3 comments 1 reply

Maxxen
Oct 24, 2024
Collaborator

spenrose
Nov 5, 2024

binste Nov 7, 2024
Author

keen85
Apr 22, 2025