Scribd
Upload
24 views1 page

Spark Code

The document contains PySpark code for processing a CSV file. It first filters rows based on a condition and then transforms the DataFrame by exploding its columns into separate rows, generating hash values for the column values, and formatting the output. The code demonstrates basic data manipulation techniques using Spark DataFrames.

Uploaded by

Rahul Waldia
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
24 views1 page

Spark Code

The document contains PySpark code for processing a CSV file. It first filters rows based on a condition and then transforms the DataFrame by exploding its columns into separate rows, generating hash values for the column values, and formatting the output. The code demonstrates basic data manipulation techniques using Spark DataFrames.

Uploaded by

Rahul Waldia
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

from pyspark.

sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("LabExam").getOrCreate()

# Read the CSV file into a DataFrame


df = spark.read.csv("data.csv", header=True)

# Filter rows where no_of_files is greater than 100


filtered_df = df.filter(df["no_of_files"] > 100)

# Show the filtered results


filtered_df.show()

................

from pyspark.sql import SparkSession, functions as F

# Create a SparkSession
spark = SparkSession.builder.appName("LabExam").getOrCreate()

# Read the CSV file into a DataFrame


df = spark.read.csv("data.csv", header=True)

# Convert all values in columns to separate rows


exploded_df = df.select(F.explode(F.array(*df.columns)).alias("column_value"))

# Generate hash of the airlines column


hashed_df = exploded_df.withColumn("hash_value", F.hash(F.col("column_value")))

# Format the output as required


formatted_df = hashed_df.select(F.concat(F.col("hash_value"), F.lit(", "),
F.col("column_value")).alias("output"))

# Show the formatted results


formatted_df.show()

You might also like