Skip to content

feat: do not use find in triplets and write a simpler implementation #268

@RayTsui

Description

@RayTsui

I conduct big data process using graphframe with the following code:

val val_std_nodes = spark.read.parquet("nodes.parquet")
val val_edges = spark.read.parquet("edges.parquet")
val graphframe = GraphFrame(val_std_nodes, val_edges)
val triplets = graphframe.triplets
triplets.write.mode("overwrite").parquet("triplets.parquet")

when execute the read operation from the above parquet
val val_triplets = spark.read.parquet("triplets.parquet")
causes "AnalysisException: unable to infer schema for parquet it must be specified manually"

Anyone came across this issue, the reason of nested dataframe schema or something else?
Thanks a lot.

Metadata

Metadata

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions