DataFrame

Lesson 14: Data Transformation with Python

by
Lesson 14: Data Transformation with Python

1. Introduction Using <span>numpy</span> and <span>pandas</span> for data transformation. 2. Vectorized Computation Assuming we have the following housing data: Read in using <span>pandas</span>: import pandas as pd df=pd.read_csv("house_price.csv") If rows or columns are not fully displayed when viewing the DataFrame, you can add the following code to resolve it: # Display all columns pd.set_option('display.max_columns', None) … Read more

PySpark: A Powerful Python Library for Big Data Processing!

by
PySpark: A Powerful Python Library for Big Data Processing!

Hello everyone, today I want to introduce a powerful Python library – PySpark! In this era of big data, ordinary Python may struggle to handle large-scale data, but PySpark allows us to elegantly process terabytes of data. It is the Python interface for Apache Spark, inheriting Spark’s distributed computing capabilities, enabling us to handle massive … Read more

I Rewrote Spark from Scratch in Rust and Open-Sourced It

by
I Rewrote Spark from Scratch in Rust and Open-Sourced It

Author: Raja Sekar Translator: Aladdin Editor: Cai Fangfang The author, Raja Sekar, has over three years of experience using Spark. He believes that Spark’s DataFrame is excellent and can solve most analytical workload problems, but there are still some scenarios where using RDD is more convenient. Thus, he conceived the idea of re-implementing Spark using … Read more

Master Pandas Core Usage in 30 Minutes: A Beginner’s Guide

by
Master Pandas Core Usage in 30 Minutes: A Beginner's Guide

Hello everyone! Today we’re going to talk about a topic that many data analysis beginners find daunting: Pandas! Don’t be scared by the name, it’s not a real panda 🐼, but one of the most powerful data processing tools in Python. Many friends think Pandas is hard to learn? No worries! Follow my pace, and … Read more