How to systematically define data quality requirements for your organization

DQ Requirements Engineering is the systematic process of eliciting, specifying, documenting, and validating data quality requirements. It bridges the gap between business requirements and the concrete measures of data quality management (DQM).

**Why is it important?**

* **Targeted DQM:** Without clear requirements, DQ measures are often ineffective or miss the actual need. Requirements engineering ensures that DQM focuses on the truly critical aspects.

* **Measurability:** It defines *what* should be measured (metrics) and *what* level of quality is expected (thresholds), enabling the success of DQ initiatives to be evaluated.

* **Foundation for DQ rules:** Precise requirements are the basis for formulating unambiguous and technically implementable DQ rules.

* **Communication & understanding:** It creates a shared understanding between business units and IT regarding expectations for data quality.

* **Efficiency:** By focusing on relevant requirements, unnecessary DQ checks and cleansing activities are avoided.

While general requirements engineering focuses on the requirements for software, systems, or processes, DQ Requirements Engineering focuses specifically on the **quality requirements for the data itself** that is used or produced by these systems and processes.

**Specific aspects of DQ Requirements Engineering:**

* **Focus on data objects:** Specific data objects (e.g., customer, product, contract) and their attributes are examined.

* **DQ dimensions:** Requirements are structured along specific data quality dimensions (accuracy, completeness, consistency, timeliness, etc.).

* **Measurability as a priority:** A strong focus is placed on defining measurable metrics and thresholds for data quality.

* **Context dependency:** DQ requirements are often highly context-dependent (e.g., the required timeliness of customer data may vary depending on the process).

* **Reference to business processes:** Requirements are often derived directly from the needs of specific business processes (What data quality does this process need to function correctly?).

* **Lifecycle:** Considers the entire data lifecycle from capture to archiving.

Although it uses similar techniques (interviews, workshops, analysis), it applies them with a specific focus on the properties and quality characteristics of data.

Eliciting data quality requirements is a communicative process that combines various methods to understand stakeholder needs:

* **Interviews:** Targeted conversations with subject matter experts, data stewards, process owners, and data users to understand their expectations, issues, and needs regarding data quality.

* **Workshops:** Facilitated group sessions with various stakeholders for the joint identification, discussion, and prioritization of DQ requirements.

* **Document analysis:** Review of existing process descriptions, functional concepts, reports, legal requirements, or guidelines for implicit or explicit DQ requirements.

* **Data profiling:** Technical analysis of data sets to uncover patterns, anomalies, and potential quality issues. This often provides indications of necessary requirements (e.g., if many null values occur -> requirement for completeness).

* **Observation:** Accompanying employees in their work with data to understand how data is used and where quality issues arise.

* **Questionnaires:** Standardized collection of requirements from a larger number of stakeholders.

* **Analysis of error logs/tickets:** Examination of reported issues and errors to identify recurring DQ problems and their causes.

The combination of these methods enables a comprehensive picture of the required data quality from various perspectives.

Since it is often not possible or economically sensible to implement *all* potential DQ requirements immediately, prioritization is essential. Common criteria for prioritization are:

1. **Business criticality (business impact):**

* What are the consequences of not fulfilling the requirement for important business processes, decisions, costs, or regulatory compliance?

* Requirements whose violation causes high risks or costs receive a higher priority.

2. **Frequency of the problem:**

* How often does the data quality problem addressed by the requirement occur?

* Frequently occurring problems are often prioritized higher.

3. **Dependencies:**

* Are there dependencies between different requirements? Must one requirement be fulfilled before another can be meaningfully implemented?

4. **Implementation effort:**

* How complex and resource-intensive is the technical and organizational implementation of the requirement (e.g., implementation of a DQ rule, adjustment of a process)?

* Sometimes \"quick wins\" (high benefit with low effort) are preferred.

5. **Strategic importance:**

* Does fulfilling the requirement support the company's strategic goals (e.g., improving customer experience, introducing new digital services)?

6. **Regulatory necessity:**

* Is fulfilling the requirement legally or regulatorily mandated?

Prioritization is often carried out in coordination with data owners and management, for example through an evaluation using a matrix (e.g., business impact vs. effort).

Clear, precise, and consistent documentation of data quality requirements is essential for their successful implementation. The following aspects should be considered:

**Formal structure:**

* **Unique ID/identifier:** For easy referencing and tracking.

* **Classification:** Categorization by DQ dimension (e.g., completeness, accuracy) and affected data objects/attributes.

* **Description:** Clear and precise formulation of what exactly is required.

* **Metrics:** How is fulfillment of the requirement measured?

* **Thresholds:** When is the requirement considered fulfilled/not fulfilled?

* **Priority/criticality:** How important is this requirement compared to others?

* **Responsibilities:** Who is responsible for implementation and monitoring?

* **Source:** Where does the requirement originate (e.g., business process, regulatory requirement)?

**Documentation levels:**

* **Business requirements:** Formulated in the language of the business units, without technical details.

* **Functional specifications:** Translation into measurable, technically implementable requirements.

* **Technical specifications:** Concrete technical implementation (e.g., SQL queries, rule definitions for DQ tools).

**Best practices:**

* **Template-based:** Use of standardized templates for consistent documentation.

* **Traceability:** Clear connection between business requirements and technical implementation.

* **Appropriate level of detail:** Detailed enough for unambiguous implementation, but not excessively complex.

* **Central management:** Use of a central repository (e.g., wiki, data quality tool, requirements tool) for all DQ requirements.

* **Versioning:** Tracking changes to requirements over time.

DQ Requirements Engineering and data governance are closely linked and mutually reinforcing. Their relationship can be described as follows: **Data governance as a framework:** * Data governance forms the organizational and procedural framework for data quality management. * It defines roles, responsibilities, and decision-making structures for data quality-related topics. * It specifies who may define, approve, and review data quality requirements. **DQ requirements as operational input:** * Precisely defined DQ requirements are in turn the basis for effective data governance. * They specify what must concretely be monitored and managed. * They enable the operationalization of the quality objectives defined in governance. **Concrete points of contact:** * **Data quality policy:** The overarching DQ policy (part of data governance) sets the framework conditions for the definition of specific DQ requirements. * **Data ownership:** Data governance defines data owners who are significantly involved in defining DQ requirements for "their" data. * **Data stewardship:** Data stewards (defined in the governance model) implement DQ requirements and monitor compliance. * **Approval processes:** Governance defines who must approve DQ requirements.

Deriving data quality requirements from business processes is a methodical approach to ensure that DQ measures actually address the areas where they create the greatest business value. The process involves the following steps: 1. **Process prioritization:** * Identification of critical business processes with a high dependency on data quality. * Consideration of factors such as business criticality, known DQ issues, and degree of automation. 2. **Process analysis:** * Detailed analysis of the selected process (e.g., using BPMN). * Identification of all process steps that use or generate data. * Capture of data flows within the process. 3. **Identification of critical data points:** * Which data/attributes are particularly important for the process? * At which points in the process do data quality issues have particularly severe consequences? 4. **Impact analysis:** * Examination of how inadequate data quality would affect the process. * Creation of "failure scenarios" (What happens if certain data is incorrect, incomplete, or outdated?). * Assessment of potential impacts (e.g., financial, operational, regulatory, customer relationship). 5.

Various categories of tools and technologies can support the DQ Requirements Engineering process: 1. **Requirements management tools:** * **General RM tools:** JIRA, IBM Rational DOORS, Polarion Requirements * **Functions:** Capture, versioning, tracking of requirements, traceability matrices * **Advantages:** Structured capture, change history, linkage with other project artifacts 2. **Data quality management platforms:** * **Examples:** Informatica Data Quality, Talend Data Quality, IBM InfoSphere Information Server * **Functions:** Integrated environments for definition, implementation, and monitoring of DQ rules * **Advantages:** End-to-end solution, specialized for DQ requirements 3. **Data profiling tools:** * **Examples:** SAS Data Management, profilers in ETL tools, open-source solutions such as Apache Griffin * **Functions:** Automated analysis of data sets to identify potential quality issues * **Advantages:** Supports evidence-based definition of DQ requirements 4. **Data catalog & metadata management:** * **Examples:** Collibra, Alation, Informatica Enterprise Data Catalog * **Functions:** Central management of metadata, including DQ requirements and rules * **Advantages:** Context for DQ requirements, connection to business glossaries and data lineage 5.

Business glossaries are essential foundations for effective DQ Requirements Engineering and play several important roles:

1. **Creating a common language:**

* Business glossaries define terms and concepts uniformly across the entire organization.

* They prevent misunderstandings when defining DQ requirements due to differing interpretations of technical terms.

* Example: If a "customer" is defined differently in sales than in customer service, this leads to contradictory DQ requirements.

2. **Providing context for data objects:**

* They provide the business context for data objects and attributes.

* This context is essential for identifying relevant DQ dimensions and requirements.

* Example: The definition of an "active customer" determines what timeliness requirements must be placed on customer data.

3. **Connection to the data model:**

* Modern business glossaries are often linked to the technical data model.

* This linkage enables the tracing of business terms to the concrete data fields in various systems.

* This ensures that DQ requirements are applied to the correct technical entities.

4. **Foundation for DQ rules:**

* Definitions in the glossary can be directly translated into DQ rules.

* Example: The definition of a "valid postal code format" in the glossary becomes a technical validation rule.

5. **Support for prioritization:**

* Business-critical concepts in the glossary often indicate critical data points for which DQ requirements should be defined as a priority.

A well-maintained business glossary is thus the basis for consistent, business-relevant DQ requirements that are understood equally by all stakeholders.

Data protection and regulatory requirements are an integral part of comprehensive DQ Requirements Engineering. Their consideration encompasses the following aspects: 1. **Identification of relevant regulations:** * Determination of all legal requirements applicable to the organization and the specific data (e.g., GDPR, BDSG, industry-specific regulations such as Basel III, Solvency II, HIPAA). * Inclusion of internal policies and standards. * Regular review for updates to the regulatory landscape. 2. **Translation into DQ requirements:** * Analysis of which data quality aspects are required to comply with the regulations. * Examples: * GDPR principle of "accuracy": Requirements for the accuracy and timeliness of personal data. * Reporting obligations: Completeness and consistency of regulatory reports. * Retention obligations: Requirements for historization and traceability. 3. **Prioritization of regulatory DQ requirements:** * Regulatory requirements often have a high priority due to potential legal consequences for non-compliance. * Risk assessment based on the severity of possible violations and probability. 4. **Integration into the DQ framework:** * Embedding regulatory requirements into the existing DQ rule set.

Validating the effectiveness of DQ requirements is essential to ensure that the defined requirements actually contribute to improving data quality and supporting business processes. This process encompasses several aspects: 1. **Review of appropriateness:** * **Subject matter validation:** Confirmation by subject matter experts that the requirements cover the actual business needs. * **Completeness check:** Ensuring that all relevant DQ dimensions for critical data have been considered. * **Consistency check:** Review for contradictions between different requirements. 2. **Technical feasibility:** * Assessment of whether the requirements are measurable and verifiable with existing systems and tools. * Identification of technical constraints that could hinder implementation. * Review of the implementability of DQ rules. 3. **Piloting:** * Trial application of the requirements to a subset of the data. * Assessment of initial results and adjustment as needed. * Identification of unexpected challenges or side effects. 4. **Effectiveness measurement after implementation:** * **Quantitative metrics:** Measuring data quality improvement based on defined KPIs. * **Qualitative assessment:** Feedback from data users on perceived quality improvement.

DQ requirements vary considerably depending on the data type, as different data has different characteristics, usage patterns, and significance for the organization: **Master data:** * **Characteristic:** Fundamental, long-lived business entities (e.g., customers, products, employees, suppliers). * **Typical DQ dimensions in focus:** * **Uniqueness:** No duplicates in customer data, unique product codes. * **Accuracy:** Correct attribute values such as addresses, product specifications. * **Completeness:** All required attributes are present. * **Standardization:** Uniform formats and classifications. * **Specific requirements:** * Strict governance for changes. * Clear responsibilities (data ownership). * Rules for golden records when multiple source systems exist. **Transactional data:** * **Characteristic:** Business events and transactions (e.g., orders, payments, logistics processes). * **Typical DQ dimensions in focus:** * **Timeliness:** Fast processing and availability. * **Integrity:** Correct linkage with master data. * **Completeness:** All transaction-relevant information captured. * **Traceability:** Audit trails for critical transactions. * **Specific requirements:** * Performant validation during capture. * Checking for business rule violations. * Timely error detection and correction.

A comprehensive DQ requirements strategy goes beyond the definition of individual requirements and embeds them in a systematic, organization-wide framework. The following elements are essential: 1. **Strategic alignment:** * **Alignment with corporate objectives:** Linking the DQ requirements strategy with overarching business goals. * **Consideration of the data strategy:** Integration into the company's overall data strategy. * **Business focus:** Prioritization of requirements that offer the greatest business value. 2. **Organizational aspects:** * **Responsibilities:** Clear definition of who is responsible for creating, approving, and monitoring DQ requirements. * **Stakeholder involvement:** Systematic participation of all relevant business units and IT. * **Training and awareness:** Measures to raise awareness of the importance of data quality and the defined requirements. 3. **Methodological elements:** * **Standardized processes:** Establishment of uniform processes for eliciting, documenting, and validating DQ requirements. * **Classification system:** Framework for categorizing requirements (e.g., by data domain, DQ dimension, criticality). * **Lifecycle management:** Processes for regular review and updating of requirements. 4. **Technological support:** * **Tool strategy:** Selection of suitable tools for managing, implementing, and monitoring DQ requirements.

Measurable DQ requirements and metrics are essential for objectively evaluating the success of data quality measures. Here are the key principles and steps for formulation: **Principles for measurable DQ requirements:** * **Specific:** Formulated clearly and precisely, without room for interpretation. * **Measurable:** Quantifiable through a defined metric. * **Achievable:** Realistically implementable in the given context. * **Relevant:** Actual added value for business processes or compliance. * **Time-bound:** With a clear timeframe for fulfillment. **Steps for formulation:** 1. **Identification of the relevant DQ dimension:** * Determine which dimension (e.g., completeness, accuracy, consistency) is to be addressed. * Example: Completeness of customer contact data. 2. **Definition of the concrete DQ requirement:** * Formulate precisely what is required. * Example: "All active customer accounts must contain an email address and a telephone number." 3. **Development of the appropriate metric:** * Determine how fulfillment is measured. * Mathematical definition: e.g., "Percentage of active customer accounts with a completed email.

Integrating DQ Requirements Engineering into agile development processes presents a particular challenge, as it traditionally requires comprehensive and forward-looking planning, while agile methods work incrementally and adaptively. Here are approaches for successful integration: **Core principles:** 1. **Early integration:** * Consider DQ requirements as part of the product backlog from the outset. * Data quality by design: DQ as an integral part of solution development, not an afterthought. 2. **Incremental approach:** * Break DQ requirements down into small, actionable user stories. * Gradual refinement from general to specific requirements. 3. **Continuous validation:** * Regular review of compliance with DQ requirements in each sprint. * Early feedback on the effectiveness of implemented DQ measures. **Practical implementation:** 1. **User stories for DQ requirements:** * Formulate DQ requirements as standalone user stories. * Example: "As a marketing manager, I want to ensure that all customer records have a valid email address so that I can run targeted email campaigns." * Extend the Definition of Done (DoD) to include DQ criteria. 2.

DQ Requirements Engineering is associated with various challenges that can make successful implementation more difficult. Here are the most common issues and proven approaches to address them: 1. **Lack of business prioritization:** * **Problem:** Data quality is often viewed as a purely technical topic and does not receive the necessary management attention. * **Solution:** * Develop a business case for DQ initiatives with concrete cost savings and benefit potential. * Illustrate DQ problems with concrete business impacts (e.g., lost revenue, customer churn). * Secure executive sponsorship for DQ initiatives. 2. **Siloed thinking and organizational barriers:** * **Problem:** DQ requirements are defined in isolation per department, without consideration of end-to-end processes. * **Solution:** * Conduct cross-departmental DQ workshops. * Involve process owners who are responsible for end-to-end processes. * Create shared incentives for cross-functional DQ improvements. 3. **Requirements that are too abstract or too detailed:** * **Problem:** DQ requirements are either too vague ("The data must be correct") or excessively detailed. * **Solution:** * Pursue a multi-level approach with business requirements and technical specifications.

Artificial intelligence (AI) and machine learning (ML) are increasingly transforming DQ Requirements Engineering by automating manual processes, assisting in the detection of complex patterns, and enabling proactive approaches. Their role encompasses several dimensions: 1. **Automated identification of DQ requirements:** * **Pattern recognition:** ML algorithms can automatically detect patterns, anomalies, and outliers in large data sets that indicate potential DQ issues. * **Recommendation systems:** AI can automatically suggest suitable DQ rules based on data structures, usage patterns, and historical DQ issues. * **Prioritization:** Intelligent algorithms assist in assessing the criticality of DQ issues and prioritizing requirements based on business impact. 2. **Enhanced data profiling:** * **Semantic analysis:** AI can understand the content of data fields and recognize semantic relationships between data that go beyond simple syntactic rules. * **Contextual validation:** Detection of values that are syntactically correct but implausible in the given context (e.g., a date of birth in the future). * **Unstructured data:** Analysis of text, images, and other unstructured data for which traditional profiling methods are insufficient. 3.

Defining data quality requirements for big data environments requires an adapted approach that accounts for the particular characteristics of these data landscapes. The key differences and adaptations are: 1. **Volume-related adaptations:** * **Sample-based approaches:** With enormous data volumes, a complete check is often not practicable. DQ requirements must incorporate statistical sampling and confidence intervals. * **Performance-optimized rules:** DQ requirements must be formulated so that they can be applied efficiently to large data volumes without significantly slowing down processing. * **Flexible metrics:** Key figures must remain meaningful and calculable even as data volumes grow. 2. **Variety-related adaptations:** * **Structured vs. unstructured data:** For unstructured data (text, images, videos), different DQ dimensions and metrics are relevant than for structured data. * **Schema-less databases:** In NoSQL environments, fixed schemas are often absent, making it more difficult to define completeness and consistency requirements. * **Polyglot persistence:** In multi-database environments, DQ requirements must be defined and measured across technologies. 3.

Effective DQ Requirements Engineering can significantly influence the total cost of ownership (TCO) of a data project, both through cost savings and strategic investments. The effects can be observed in various phases and areas: **Cost-reducing effects:** 1. **Avoidance of rework and corrections:** * Early definition of clear DQ requirements reduces costly revisions in later project phases. * Studies show that fixing errors in the production phase can cost up to 100 times more than in the requirements phase. 2. **Reduction of data cleansing efforts:** * Preventive DQ measures based on clear requirements minimize ongoing cleansing efforts. * Automated DQ controls reduce the manual effort required for data quality assurance. 3. **Avoidance of poor decisions:** * Higher data quality through precise requirements reduces the risk of costly poor decisions. * The financial impact of business decisions based on poor data can far exceed the implementation effort for DQ. 4. **More efficient resource utilization:** * Prioritized DQ requirements focus resources on the most business-relevant aspects. * Avoidance of over-engineering through clear delineation of necessary quality levels. 5.

The migration of data infrastructures to the cloud changes DQ Requirements Engineering in several dimensions. Cloud-specific aspects must be considered when defining, implementing, and monitoring DQ requirements: 1. **Architectural adaptations:** * **Multi-service environments:** Cloud environments frequently combine various specialized services (e.g., storage, databases, analytics). DQ requirements must be defined consistently across services. * **Serverless architecture:** In event-driven, serverless architectures, DQ controls must be integrated into individual functions rather than being located at central data access points. * **Microservices:** Distributed data responsibilities require clear DQ requirements at service interfaces and API contracts. 2. **Shared responsibility model:** * **Delineation of responsibilities:** Clear definition of which DQ requirements are fulfilled by the cloud provider and which remain the responsibility of the organization itself. * **Provider SLAs:** Incorporation of the service levels guaranteed by the provider (availability, latency, data loss risk) into DQ requirements. * **Compliance requirements:** Consideration of the cloud provider's capabilities and limitations in fulfilling regulatory DQ requirements. 3.