Machine Learning on Streaming Data with
OCI Data Flow
Sujoy Chowdhury
Product Manager, OCI Data Flow
Sivanesh Selvanataraj
Software Engineer, OCI Data Flow
May 5th, 2022
Introducing OCI Data Flow Streaming
Oracle Cloud Infrastructure Data Flow, a fully managed serverless Apache
Spark service, now supports Spark Streaming. Customers can now use OCI
Data Flow to do cloud scale ETL on their continuously produced stream data.
2
Data Lakehouse Platform on OCI
Open & flexible: analyze any database, any application, from anywhere
Data Lakehouse on OCI
Managed Open Source Data Warehouse
Data sources Data target
Data Stores
Any Database Big Data Service Autonomous Data
Data Flow Warehouse
Any BI Tool
Search Insights MySQL HeatWave
Data Definition
Any Application Object Storage
Data Movement & Discovery
Relational Data
Machine Learning
& Data Science
Any Cloud
Data Integration Data Catalog
GoldenGate
OCI Streaming
Data Science Platform Any Application
Any Events/Sensors
AI Services
Lakehouse: Cloud Scale ETL & ML using OCI Data Flow
Use all data to innovate in a lakehouse.
1. Cloud Scale ETL & ML
Cloud Scale ETL & ML
• Cloud Scale ETL at ‘landing zone’ for data warehouse
• Cloud Scale ETL at Data Lake (object store to object Raw data/ Lakehouse Insights
store) to optimize ADW stream
• Data Lake to optimize ADW storage (inexpensive, easily
accessible data store for important but rarely used data,
processed data & meta-data)
• Data Catalog harvests the lake (technical metadata) Autonomous Oracle
and applies a business context. This includes ETL OCI Data Flow ETL Data Warehouse Analytics
Cloud
data/metadata created by Spark applications.
ML
• Data Integration service for ETL orchestration,
Object Storage
workflows Data Lake
2. Stream data processing OCI Data
Catalog
Object Storage OCI Data Science
OCI AI Services
Data Lake
Oracle Machine
• Streamed logs, sensor data, CDR, clickstream that Learning
needs to be processed before ADW (ETL & ML)
3. Big Data processing using Python, Java OCI Data
• Data Engineers/ Data Scientists can use their preferred Integration
programming language for processing big data
4
Analyzing Streams with OCI Data Flow Spark Streaming
OCI Streaming
OCI Streaming
OCI Data Flow
Object Storage
Data Lake Object Storage
Data Lake
• Continuously retrieve input data, not just hourly/ daily.
• Support Kafka-compatible stream/ message queue as source and sink, in addition to batch file-
formats
• Processed data made available continuously (<1 min), not just after job completion
• Outage does not require job restart, can continue from last checkpoint
• Handle late data and watermarking: can catch up backlogged data over time
• Heavy-weight stream processing: Stream-static Joins, Stream-stream Joins, Inner Joins with optional
watermarking. Outer Joins with Watermarking, streaming Deduplication
• Machine Learning on stream data using MLLib.
Spark Streaming Machine Learning Use Cases
Manufacturing Financial Services Telecom
• Predicting RUL • Real time fraud • Churn prediction
‘Remaining Useful Life’ detection
6
Manufacturing: Predicting Remaining Useful Life (RUL) from stream data
Manufacturing
Lakehouse
OCI Streaming OCI Streaming
OCI Data Flow Autonomous
Data Warehouse
Object Storage
Data Lake
Object Storage
Data Lake
Color RUL Status, Action
Red 10 – 30 days Degraded, perform maintenance.
Yellow 30 – 60 days Degrading, watch.
Green >60 days Healthy, no action needed.
7
Manufacturing: Primer on Predictive Maintenance
• Maintenance, repair and overhaul (MRO) of equipment is mission-critical for
Manufacturing industries
• 3 types of maintenance
• Preventive maintenance (PM) : Planned/Scheduled Maintenance
• Reactive maintenance (Run To Failure) : Maintenance after machine Runs To Failure
• Predictive maintenance (PdM) : Predict the trend and estimation when maintenance
is required
• Predictive maintenance
• Survival analysis:
• Spark ML Survival Regression Model: Accelerated Failure Model (AFT)
8
Spark ML Survival Regression Model
9
RUL Training dataset
• Turbofan Engine Degradation Simulation Data Set
A. Saxena and K. Goebel (2008 NASA Ames Prognostics
Data Repository (http://ti.arc.nasa.gov/project/prognostic-
data-repository), NASA Ames Research Center, Moffett
Field, CA
• Engine degradation Run-to-Failure simulation was carried
out using C-MAPSS (Commercial Modular Aero- Propulsion
System Simulation).
10
Trainer: OCI Data Flow application: RULSurvivalModelTrainer
11
Trainer: OCI Data Flow application: RULSurvivalModelTrainer
12
Simulator: OCI Data Flow Streaming application SensorDataSimulator
13
Simulating Sensor Data for Spark Streaming
Color RUL Status, Action
Red 10 – 30 days Degraded, perform maintenance.
Yellow 30 – 60 days Degrading, watch.
14 Green >60 days Healthy, no action needed.
Predictor: OCI Data Flow streaming application RealTimeRULPredictor
15
Real time RUL prediction using OCI Data Flow Spark Streaming
16
Real time RUL prediction using OCI Data Flow Spark Streaming
Color RUL Status, Action
Red 10 – 30 days Degraded, perform maintenance.
Yellow 30 – 60 days Degrading, watch.
Green 60 – 100 days Healthy, no action needed.
17
Demo: Predicting Remaining Useful Life (RUL) from stream data
Manufacturing
Lakehouse
OCI Streaming OCI Streaming
OCI Data Flow Autonomous
Data Warehouse
Object Storage
Data Lake
Object Storage
Data Lake
Color RUL Status, Action
Red 10 – 30 days Degraded, perform maintenance.
Yellow 30 – 60 days Degrading, watch.
Green >60 days Healthy, no action needed.
18
Spark Streaming with OCI Data Flow: Managed Spark UI view
19
Spark Oracle Datasource for connecting to Oracle DB
20
Spark Streaming with OCI Data Flow: OCI Logging integration
21
Fully managed Spark Streaming with OCI Data Flow
Data Flow continues to be the fully managed Spark experience with zero administration overhead:
• End-to-end exactly-once fault-tolerance guarantees through checkpointing and Write-Ahead
Logs from Spark
• Cloud native authentication: Resource principal-based authentication enables Data Flow Streaming
applications to run beyond 24 hours
• Managed automatic security patching so that customers can focus building their application, not
update/ upgrade / infra operation.
• Managed automatic run resubmission by Data Flow for additional fault tolerance on top of Spark
• Deep OCI integration with OCI Logging, OCI Metrics for simpler troubleshooting, along with other
1P OCI services
• Pay for infra only: Data Flow Streaming feature will not introduce any additional meters. Customers
will continue to pay only for the infra their Data Flow runs use.
22
Resources
Manufacturing
Lakehouse
OCI Streaming OCI Streaming
OCI Data Flow Autonomous
Data Warehouse
• RUL Survival Model
Trainer
Object Storage • Real time RUL
Data Lake Predictor
Object Storage
Data Lake
Learn more
https://docs.oracle.com/en-us/iaas/data-flow/using/spark-streaming.htm
https://docs.oracle.com/en-us/iaas/data-flow/using/spark_oracle_datasource.htm
Training Dataset: https://ti.arc.nasa.gov/tech/dash/groups/pcoe/prognostic-data-repository/#turbofan
Sample code https://github.com/oracle-samples/oracle-dataflow-samples
Contact
[email protected]
23
