18456675. LOW LATENCY INGESTION INTO A DATA SYSTEM simplified abstract (Snowflake Inc.)

From WikiPatents
Jump to navigation Jump to search

LOW LATENCY INGESTION INTO A DATA SYSTEM

Organization Name

Snowflake Inc.

Inventor(s)

Dhiraj Gupta of San Mateo CA (US)

Subramanian Muralidhar of Mercer Island WA (US)

LOW LATENCY INGESTION INTO A DATA SYSTEM - A simplified explanation of the abstract

This abstract first appeared for US patent application 18456675 titled 'LOW LATENCY INGESTION INTO A DATA SYSTEM

Simplified Explanation

Described in this patent application are techniques for enhancing the transfer of metadata from a metadata database to a data system's database, such as a data warehouse. The metadata is stored in the metadata database with a version stamp and a partition identifier. Multiple readers can scan the metadata database based on these values to export the metadata to a cloud storage location. The exported data can then be automatically ingested into the database, which includes a journal and snapshot table.

  • Techniques for improving transfer of metadata from a metadata database to a data system's database
  • Metadata is stored in the metadata database with a version stamp and a partition identifier
  • Multiple readers can scan the metadata database based on version stamp and partition identifier values
  • Exported metadata is stored in a cloud storage location
  • Exported data is automatically ingested into the database, which includes a journal and snapshot table

Potential Applications

  • Data warehousing
  • Database management systems
  • Cloud storage and data transfer

Problems Solved

  • Efficient transfer of metadata from a metadata database to a data system's database
  • Simplified export and ingestion process
  • Improved organization and management of metadata

Benefits

  • Streamlined transfer process reduces manual effort and potential errors
  • Faster and more efficient data transfer and ingestion
  • Improved organization and accessibility of metadata


Original Abstract Submitted

Described herein are techniques for improving transfer of metadata from a metadata database to a database stored in a data system, such as a data warehouse. The metadata may be written into the metadata database with a version stamp, which is monotonic increasing register value, and a partition identifier, which can be generated using attribute values of the metadata. A plurality of readers can scan the metadata database based on version stamp and partition identifier values to export the metadata to a cloud storage location. From the cloud storage location, the exported data can be auto ingested into the database, which includes a journal and snapshot table.

Retrieved from "http://wikipatents.org/index.php?title=18456675._LOW_LATENCY_INGESTION_INTO_A_DATA_SYSTEM_simplified_abstract_(Snowflake_Inc.)&oldid=18440"