Re-engineering efforts at Fidelity, CNN, and other companies have enabled faster access to real-time data. Experts share their strategies for better management. Organizations need a secure data pipeline to extract real-time analytics from workloads and deliver trusted data. However, data pipelines are becoming increasingly complex to manage. This article explores how leading companies are transforming their data pipelines using Snowflake\u2019s Apache Iceberg and new solutions like Iceberg Tables to enhance efficiency and business outcomes.<\/i><\/p>\nThe Complexity of Modern Data Pipelines<\/b><\/h2>\n
Modern\u00a0data pipelines\u00a0are essential for organizations to manage and analyze vast amounts of data in real-time. However, they are also increasingly complex, involving numerous processes and technologies to ensure data is processed, stored, and analyzed efficiently. This complexity necessitates robust solutions that can simplify pipeline management while maintaining flexibility and scalability.<\/p>\n
Data pipelines are the backbone of any data-driven organization. They facilitate the flow of data from various sources to a destination where it can be analyzed and used to generate insights. Effective data pipelines are crucial for real-time analytics, enabling businesses to make timely decisions based on current information. However, managing these pipelines can be challenging due to the sheer volume of data and the need for real-time processing.<\/p>\n
Companies such as Booking.com, Capital One, Fidelity, and CNN are re-engineering their data pipelines using\u00a0Snowflake\u2019s Apache Iceberg\u00a0and a new solution, Iceberg Tables. These technologies include data lakehouses, data lakes, and data meshes, allowing IT leaders to simplify pipeline development and work with open data flexibly.<\/p>\n
Snowflake\u2019s Apache Iceberg and Iceberg Tables offer several benefits for data pipeline management. They enable organizations to handle large datasets efficiently, support various data formats, and ensure data consistency. These solutions also provide the flexibility to scale according to business needs, making them ideal for organizations looking to enhance their data management capabilities.<\/p>\n
\u201cWith Iceberg, we can broaden our use cases for Snowflake as our open data lakehouse for machine learning, AI, business intelligence, and geospatial analysis \u2014 even for data stored externally,\u201d said Thomas Davey, chief data officer for Booking.com.<\/p>\n
Iceberg Tables, announced June 4 at the Snowflake Summit in San Francisco, comes on the heels of the recently announced Polaris Catalog, a vendor-neutral and fully open catalog implementation for Apache Iceberg. Polaris Catalog enables cross-engine interoperability, giving organizations more choice, flexibility, and control over their data.<\/p>\n
Organizations can get started running Polaris Catalog hosted in Snowflake\u2019s AI Data Cloud or using containers within their own infrastructure. This flexibility allows businesses to choose the deployment model that best fits their needs, ensuring they can leverage Polaris Catalog\u2019s capabilities to enhance their data management strategies.<\/p>\n
Fidelity has reimagined its data pipelines using Snowflake Marketplace, saving the company time and resources in data engineering. Its supported business units, including fixed income and data science, can now analyze data faster, spending \u201cmore time on research and less on pipeline management,\u201d said Balaram Keshri, vice president of architecture at Fidelity.<\/p>\n
With Snowflake managing its data, Fidelity has significantly improved performance, enabling faster data loading, querying, and analysis. The Snowflake Performance Index reports that it has \u201creduced organizations\u2019 query duration by 27% since it started tracking this metric, and by 12% over the past 12 months,\u201d according to a press release.<\/p>\n
Capital One, reportedly the first U.S. bank to migrate its entire on-premises data center to the cloud, has also found success with its new data pipelines, thanks to Snowflake\u2019s data sharing capabilities. This feature allows multiple analysts to access related data without affecting one another\u2019s performance. Users can also categorize data according to workload type.<\/p>\n
\u201cSnowflake is so flexible and efficient that you can quickly go from \u2018data starved\u2019 to \u2018data drunk.\u2019 To avoid that data avalanche and associated costs, we worked to put some controls in place,\u201d wrote Salim Syed, head of engineering for Capital One Software, in a blog post.<\/p>\n
CNN\u2019s dramatic pipeline transformation has provided accelerated access to analytics. Over the past year, the multinational news channel and website, owned by Warner Bros. Discovery, has shifted to using real-time data pipelines for workloads that support critical parts of its content delivery strategy. The goal is to move the horizon of actionable data down \u201cfrom hours to seconds\u201d by replacing existing batch pipelines.<\/p>\n
\u201cWe will move around 100 terabytes of data a day across about 600,000 queries from our various partners,\u201d said Zach Lancaster, engineering manager of Warner Bros. Discovery. Now, with its scalable and newly managed pipeline, CNN can scrape the data for core use cases and prioritize workloads that drive the most business value.<\/p>\n
As user-friendly as the Snowflake platform is, IT leaders still need a clear strategy in mind as they improve their data pipelines. Here are three steps to help transform your data pipeline effectively.<\/p>\n
For starters, \u201cthink about how you can bring your stakeholders on board. You want them to become the ultimate stewards of the process,\u201d Lancaster said. Engaging stakeholders ensures that the pipeline transformation aligns with business goals and receives the necessary support for successful implementation.<\/p>\n
Second, revisit your use cases. \u201cPlatforms develop over the years, as does your business, so try to re-evaluate your use cases and dial back your system,\u201d Torrance advised. This approach can help with cost optimization and ensure that the pipeline meets current business needs.<\/p>\n
Third, \u201cmake sure you understand the ask of each request and how you expect to use it over time in your data pipeline,\u201d Lancaster said. Clear understanding of requests ensures that the pipeline is designed to handle future demands and remains flexible enough to accommodate changes.<\/p>\n
If a company is redesigning its data pipeline, it needs to be cross-functional and serve the most central parts of the business. Consider \u201cmachine-to-machine use cases,\u201d as these are important for interoperability within your entire tech stack.<\/p>\n
Finally, remember that more intricate systems aren\u2019t always better. \u201cThink carefully. Just because I have a request, do I need to accomplish it? And does the added complexity add value to the business, or does it do a disservice to the stakeholder?\u201d Lancaster said. This consideration ensures that the pipeline remains efficient and aligned with business objectives.<\/p>\n
Transforming data pipelines is essential for organizations looking to leverage real-time analytics and improve data management. By adopting solutions like Snowflake\u2019s Apache Iceberg and Iceberg Tables, companies can simplify pipeline development, enhance performance, and ensure scalability. Engaging stakeholders, revisiting use cases, and understanding requests are crucial steps in this transformation process.\u00a0As organizations continue to navigate the complexities of data management, adopting flexible and scalable solutions like those offered by Snowflake will be key to maintaining competitive advantage. By focusing on efficient pipeline management and leveraging advanced technologies, businesses can unlock the full potential of their data and drive innovation.<\/p>\n","protected":false},"excerpt":{"rendered":"
Re-engineering efforts at Fidelity, CNN, and other companies have enabled faster access to real-time data. Experts share their strategies for better management. Organizations need a secure data pipeline to extract real-time analytics from workloads and deliver trusted data. However, data… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-71","post","type-post","status-publish","format-standard","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/posts\/71","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=71"}],"version-history":[{"count":1,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/posts\/71\/revisions"}],"predecessor-version":[{"id":72,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=\/wp\/v2\/posts\/71\/revisions\/72"}],"wp:attachment":[{"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=71"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=71"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/r229.rookiessportsbarny.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=71"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}