18494917. EXTRACT, TRANSFORM, LOAD MONITORING PLATFORM simplified abstract (Capital One Services, LLC)
Contents
EXTRACT, TRANSFORM, LOAD MONITORING PLATFORM
Organization Name
Inventor(s)
Chanakya Kaspa of Mckinney TX (US)
Divya Mehrotra of Frisco TX (US)
Gregory Muzyn of plano TX (US)
EXTRACT, TRANSFORM, LOAD MONITORING PLATFORM - A simplified explanation of the abstract
This abstract first appeared for US patent application 18494917 titled 'EXTRACT, TRANSFORM, LOAD MONITORING PLATFORM
Simplified Explanation
The abstract describes a monitoring device that receives configuration information for an ETL pipeline, generates lineage data and predicted quality metrics, and visualizes the data flow and quality metrics.
- Monitoring device receives configuration information for ETL pipeline
- Generates lineage data for data flow from sources to sinks
- Predicts quality metrics using machine learning model
- Visualizes data flow and quality metrics in a visualization
Potential Applications
- Data quality monitoring in ETL pipelines
- Performance optimization of data pipelines
- Automated visualization of data flows
Problems Solved
- Lack of visibility into data flow in ETL pipelines
- Difficulty in monitoring and predicting data quality
- Manual generation of lineage data and quality metrics
Benefits
- Improved data quality and reliability
- Enhanced performance and efficiency of ETL pipelines
- Automated monitoring and visualization for easier analysis
Original Abstract Submitted
In some implementations, a monitoring device may receive configuration information associated with an extract, transform, load (ETL) pipeline that includes one or more data sources and one or more data sinks. The monitoring device may generate, based on the configuration information, lineage data related to a data flow from the one or more data sources to the one or more data sinks in the ETL pipeline. The monitoring device may generate one or more predicted quality metrics associated with the ETL pipeline using a machine learning model. The monitoring device may generate a visualization in which multiple nodes are arranged to indicate the data flow from the one or more data sources to the one or more data sinks and further in which the one or more predicted quality metrics are encoded within the visualization.
