Extract, Transform and Load (ETL) refers to a process of connecting to data sources, integrating data from various data sources, improving data quality, aggregating it and then storing it in staging data source or data marts or data warehouses for consumption of various business applications including BI, Analytics and Reporting. It offers high quality data, which otherwise resides in poorly structured heterogeneous, complicated data sources.
Think of your data warehouse as an active repository that is ever changing as new data sources keep on getting added and existing data sources keep on getting updated. In order to manage the environment, an organization must dedicate resources to monitor and track ETL process, its data flow, data integration and data updates.
It is no wonder that the average enterprise is cautious when any suggestion is made to change the process they have in place. So, the idea of data access by business users may cause concern, and the IT staff may wonder whether this access signals the end of the true ETL process along with the comprehensive maintenance and data governance policies.
But, before the organization discounts the democratization of data and the concept of Self-Serve Data Preparation, it is important to understand the need for (and benefits of) comprehensive ETL policies and maintenance AND the flexibility, agility and accessibility provided by business user access to self-serve data preparation tools.
When an enterprise provides a solid ETL foundation with appropriate monitoring, management, integration processes and skills and adds the flexibility of self-serve data preparation for business users, the organization can acquire good data quickly and offer insight to business users and ultimately to the entire business organization. This approach ensures that the users and the organization can make better business decisions, and make them more quickly. By combining traditional ETL, data warehouse management and technical skills with self-serve data preparation and business user access to ETL and cube management, your organization can balance and optimize quality vs agility to create an agile analytical environment.
Let’s look at the benefits of, and the need for, traditional ETL as well as self-serve data preparation performed by business users.
Give the Power to Business Users
When an organization invests in a business intelligence solution that provides flexible options for data access and management, users can choose the option that is best for a use case scenario. They can use self-serve data preparation tools to connect to data sources like databases, OLAP cubes and spreadsheets using simple wizard based connection interface.
Today’s business users and managers face the daunting task of compiling and analyzing data simply and easily and using that data to make confident decisions. Users need a scalable, high-performance solution that can integrate data from data warehouses, data marts, ERP, best-of-breed applications, CRM and any number of operational system databases, spreadsheets, cloud based data sources and legacy systems and leverage Deep Dive Analytics and self-serve data preparation to compile and analyze data quickly and easily.
The key focus in this scenario is agility. When business users need information they don’t always need exacting, detailed data extraction and analysis. Many times, they are testing a hypothesis or creating a prototype to analyze options or to see how things will play out in a particular set of circumstances, and in those cases, 100% accuracy of data is not necessary. By providing access to data and simple data preparation tools, the organization can keep things moving and allow for empowerment and creativity.
Preserving Traditional ETL
Data Extraction, Data Transformation and Data Management solution provides the foundation for an enterprise to extract, load and transform (ETL) data. While it is important to provide business users with access to sophisticated, easy-to-use tools, the foundational work of ETL must be preserved.
If the integrity of enterprise data and data integration is to be preserved, the data warehouse environment must integrate, and consolidate data from a disparate group of systems and allow for data extraction and compilation across the enterprise. Highly skilled, trained professionals manage these systems with a thorough understanding of the source of the data and how to manipulate and report on that data to create complex reports and provide support for business users and managers.
When combined with in-depth IT and analytical knowledge, this complex data environment and resource pool provides crucial support to refine and cleanse data and to ensure that only the necessary data is extracted and analyzed so that critical resources are not wasted and data is not misinterpreted. There are times when an organization must look to these foundational processes and team members to clarify data and to ensure that hardware, networks, and data sources come together to optimize extraction and reporting and assure appropriate interpretation.
Traditional ETL performed by experts produces great data quality at the right level of aggregation to ensure stable performance and integrity of data governance. Because these tasks are performed by IT certified professionals, the organization can be sure of the reliability and quality of the data. But these resources should be optimized to optimize the special skill set and knowledge of these team members and to prevent competing priorities, and delays in crucial data analysis and timeliness.
By combining the traditional ETL processes and skills with business user access to self-serve data preparation tools, your organization can optimize skilled resources and move quickly to addresses opportunities, challenges and trends.
When the organization needs very precise, comprehensive ETL, your trained team can provide those services. When a business user requires information, but does NOT require exacting precision, self-serve data preparation tools can speed decisions and empower users to analyze data for day-to-day tasks and discussions.
The rationale for any organizational or process change is always the bottom line. When you balance self-serve data preparation with traditional ETL processes and skills, you get the best of both worlds and improve resource optimization and the bottom line. By balancing speed and agility with quality and data governance, the organization can save its skilled resources for the most important projects and business users can prepare and analyze data for those projects and decisions that do not require high-level, dedicated attention.
Original Post: Self-Serve Data Preparation Doesn’t Mean Traditional ETL is Dead!
