Core banking data quality assessment: Automated validation frameworks for ML-ready datasets

 Corresponding author: Sandeep Ravichandra Gourneni. Copyright © 2025 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution License 4.0. Core banking data quality assessment: Automated validation frameworks for MLready datasets Sandeep Ravichandra Gourneni * Acharya Nagarjuna University, India. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 Publication history: Received on 18 March 2025; revised on 26 April 2025; accepted on 28 April 2025 Article DOI: https://doi.org/10.30574/gjeta.2025.23.1.0141 Abstract This article presents a comprehensive framework for automated data quality assessment in core banking systems, focusing on preparing high-quality datasets for machine learning applications. We examine the unique challenges of banking data validation, including regulatory compliance, security requirements, and the complex relationships between financial data entities. The proposed framework integrates traditional banking data governance principles with modern machine learning validation techniques to create a robust system for ensuring data readiness. Through case studies, empirical analysis, and practical implementation guidelines, we demonstrate how financial institutions can leverage automated validation to improve decision-making processes, risk assessment, and customer experience while maintaining data integrity and compliance. Keywords: Core Banking; Data Quality; Machine Learning; Validation Frameworks; Financial Data Governance; Regulatory Compliance 1. Introduction The banking sector generates vast amounts of data through daily transactions, customer interactions, risk assessments, and compliance activities. The quality of this data directly impacts operational efficiency, regulatory compliance, and the effectiveness of increasingly important machine learning (ML) applications. Financial institutions face unique challenges in maintaining data quality due to the sensitive nature of financial information, complex regulatory requirements, and the interconnected nature of banking systems. This paper introduces a novel framework for automated validation of core banking data, specifically designed to ensure datasets are "ML-ready." We define ML-ready datasets as those that not only meet traditional data quality requirements but also satisfy the specific needs of machine learning algorithms in terms of completeness, consistency, accuracy, and representativeness. This work is significant because it bridges the gap between traditional banking data governance approaches and modern machine learning validation requirements. By integrating these perspectives, we provide financial institutions with a practical roadmap for implementing automated validation frameworks for operational and analytical purposes. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 474 2. The banking data landscape 2.1. Core Banking Data Sources Modern banking systems encompass numerous data sources, forming the core banking data landscape. Table 1 provides an overview of these primary data sources and their characteristics. Table 1 Core Banking Data Sources and Characteristics Data Source Description Data Types Update Frequency Quality Challenges Transaction Processing Customer deposits, withdrawals, transfers Structured numerical Real-time Volume, velocity, accuracy Customer Information Demographics, KYC, contact details Structured, semistructured Periodic Completeness, currency, duplication Account Management Account status, balances, products Structured Daily Consistency, integrity Loan Management Loan applications, approvals, repayments Structured, document-based Daily/Real-time Completeness, consistency Risk Management Credit scores, risk models, exposure data Structured, analytical Daily/Weekly Timeliness, accuracy Compliance Data Regulatory reporting, audit trails Structured, document-based Periodic Completeness, auditability External Data Market data, economic indicators Semi-structured, unstructured Varied Integration, standardization Digital Banking Online/mobile banking interactions Structured, event data Real-time Volume, variety, veracity 2.2. Core Banking System Architecture Core banking systems typically follow a multi-tier architecture with data flowing through various layers. Figure 1 Core Banking System Architecture with Data Flow and Validation Checkpoints Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 475 2.3. Data Integration Challenges 2.3.1. Banking data integration presents unique challenges due to the following factors: • Legacy Systems Integration: Many financial institutions operate with modern and legacy systems, creating data format and synchronization issues. • Multi-channel Data Collection: Data collected through various channels (branch, ATM, online, mobile) may have different structures and quality levels. • Real-time vs. Batch Processing: Banking systems must reconcile real-time transaction data with batchprocessed analytical data. • Third-party Data Integration: External data sources must be integrated seamlessly while maintaining data quality standards. • Cross-border Operations: Global banks must harmonize data across jurisdictions with varying regulatory requirements and data standards. 3. Data Quality Dimensions in Banking 3.1. Critical Data Quality Dimensions Banking data quality can be assessed across multiple dimensions, particularly relevant to financial operations. Figure 2 presents these dimensions and their relative importance in banking contexts. Figure 2 Radar Chart of Banking Data Quality Dimensions 3.2. Data Quality Issues Specific to Banking 3.2.1. Banking data faces unique quality challenges • Reference Data Management: Maintaining accurate and consistent reference data across systems (currency, country, and product codes). • Customer Identity Resolution: Ensuring consistent customer identification across multiple accounts and services. • Transaction Integrity: Maintaining the atomicity and consistency of financial transactions across distributed systems. • Time-sensitive Data: Managing time-dependent data such as interest, exchange, and product pricing. • Regulatory Reporting Accuracy: Ensuring that the data aggregated for regulatory reporting reflects accurate underlying transactions. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 476 3.3. Impact of Poor Data Quality in Banking Poor data quality in banking environments can have severe consequences, as shown in Table 2. Table 2 Banking Data Quality Issues and Their Impacts Data Quality Issue Operational Impact Financial Impact Regulatory Impact ML Model Impact Inaccurate customer data Failed communications, poor service Customer attrition KYC violations Biased customer segmentation Duplicate transactions Reconciliation efforts Financial losses Reporting errors Skewed pattern detection Missing account information Service delays Revenue leakage Compliance gaps Incomplete feature sets Inconsistent product data Incorrect product offerings Pricing errors Mis-selling issues Poor recommendation accuracy Outdated risk data Incorrect risk assessments Capital misallocation Capital adequacy errors Inaccurate risk predictions 4. Regulatory Considerations for Banking Data 4.1. Regulatory Framework Impact on Data Validation Banking data validation must comply with numerous regulations that vary by jurisdiction. Key regulatory frameworks include: • Basel Committee on Banking Supervision (BCBS) 239: Principles for effective risk data aggregation and risk reporting. • General Data Protection Regulation (GDPR): Requirements for personal data protection, including data accuracy and customer rights. • Payment Services Directive 2 (PSD2): Standards for payment data, including strong customer authentication. • Financial Action Task Force (FATF): Requirements for anti-money laundering (AML) and countering financing of terrorism (CFT) data. • Sarbanes-Oxley Act (SOX): Controls over financial reporting data. • Local Banking Regulations: Country-specific requirements for banking data management. 4.2. Regulatory Reporting and Data Quality Regulatory reporting demands particularly high data quality standards, as illustrated in Figure 3. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 477 Figure 3 Regulatory Reporting Data Quality Requirements 4.3. Compliance Documentation for Data Validation A comprehensive data validation framework must produce documentation to demonstrate regulatory compliance. Key documentation includes: • Data quality policies and standards • Data validation methodologies • Exception handling procedures • Remediation processes • Audit trails and change logs • Validation testing results • Regulatory submission approvals 5. Automated validation framework architecture 5.1. Architectural Components Our proposed automated validation framework consists of six core components designed to address the unique needs of banking data. Figure 4 illustrates this architecture. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 478 Figure 4 Automated Banking Data Validation Framework Architecture 5.2. Data Ingestion and Profiling The ingestion layer employs specialized connectors for banking systems, including: • Core Banking System Connectors: Custom adapters for major core banking platforms. • Real-time Transaction Monitoring: Streaming data validation for transaction processing. • Document Processing: Validation for semi-structured documents like loan applications. • External Data Integration: Validation for market data, credit bureau information, and other external sources. Automated profiling generates metadata including: • Data type verification • Value range analysis • Distribution statistics • Completeness metrics • Pattern detection • Cross-field relationships 5.3. Rules-based Validation Engine The rules engine implements domain-specific validation rules for banking data: Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 479 Table 3 Banking-Specific Validation Rule Categories Rule Category Description Example Rules Structural Validation Data format and type checking Account numbers conform to the institutionspecific format Referential Integrity Cross-system data relationships Customer IDs exist in the customer master database Business Logic Industry-specific rules Loan amount within product-specific limits Regulatory Compliance Rules enforcing regulatory requirements Transaction reporting thresholds for AML Temporal Validation Time-based data rules Interest rate effective dates must not overlap Computational Validation Calculation verification Account balance matches transaction history Cross-system Consistency Validation across multiple systems Customer details match across banking channels 5.4. Machine Learning for Validation Enhancement The ML validation layer enhances traditional rule-based approaches: • Anomaly Detection: Using unsupervised learning to identify unusual patterns in transaction data. • Pattern Recognition: Using supervised learning to identify complex relationships between banking data elements. • Predictive Data Quality: Anticipating data quality issues based on historical patterns. • Auto-correction Suggestions: Generating potential corrections for common data errors. 5.5. Quality Monitoring and Alerting Continuous monitoring provides: • Real-time Data Quality Dashboards: Visual indicators of data quality across banking systems. • Threshold-based Alerting: Notifications when quality metrics fall below defined thresholds. • Trend Analysis: Tracking data quality metrics over time to identify degradation patterns. • Impact Assessment: Evaluating the business impact of identified data quality issues. 6. Machine Learning Requirements for Financial Data 6.1. ML-Specific Data Quality Requirements Machine learning models in banking have specific data quality requirements beyond traditional validation: • Class Balance: Ensuring proper representation of different classes (e.g., fraudulent vs. legitimate transactions). • Feature Distribution Stability: Monitoring for distribution shifts in key features over time. • Missing Value Patterns: Understanding patterns in missing data that may contain predictive information. • Outlier Characterization: Distinguishing between anomalies representing data quality issues and legitimate but unusual patterns. • Feature Independence: Assessing multicollinearity among features that can impact model performance. Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 480 6.2. Data Readiness Assessment Framework Figure 5 ML Data Readiness Assessment Framework for Banking 6.3. Banking ML Use Case Requirements Matrix Different banking ML applications have varying data quality requirements, as shown in Table 4. Table 4 Data Quality Requirements by Banking ML Use Case ML Use Case Volume Requirements Freshness Requirements Completeness Requirements Special Considerations Credit Scoring Moderate historical data Monthly updates sufficient High completeness needed Regulatory fairness requirements Fraud Detection Large transaction volume Real-time data critical Can handle some incompleteness Class imbalance management Customer Segmentation Comprehensive customer data Quarterly updates sufficient Moderate completeness Feature richness important Churn Prediction 1-2 years of history Weekly updates High completeness Temporal consistency critical Product Recommendation Rich interaction history Daily updates Moderate completeness Cold start problem for new customers AML Monitoring Large transaction history Near-real-time High completeness Complex pattern detection 7. Feature Engineering for Banking ML Models 7.1. Banking-Specific Feature Types Effective banking ML models require specialized feature engineering approaches: Global Journal of Engineering and Technology Advances, 2025, 23(01), 473-486 481 • Temporal Features: Transaction frequencies, sequence patterns, seasonal behaviors. • Network Features: Relationship metrics between accounts, entities, and transactions. • Behavioral Features: Customer interaction patterns, channel preferences, response rates. • Financial Ratio Features: Derived metrics such as debt-to-income and utilization ratios. • Risk Indicator Features: Early warning signals derived from account activities. 7.2. Automated Feature Validation Our framework includes automated validation specifically for engineered features: Table 5 Feature Validation Metrics and Thresholds Validation Type Metric Acceptable Threshold Critical Threshold Missing Rate Percentage of null values <5% >20% Cardinality Unique value count ratio >0.1% for categorical <0.01% Distribution Drift KL divergence from baseline <0.2 >0.5 Correlation Feature-target correlation >0.05 <0.01 Multicollinearity Variance Inflation Factor <5 >10 Information Value IV score >0.1 <0.02 Predictive Power AUC contribution >0.02 <0.005 7.3. Feature Store Integration Figure 6 Banking Feature Store with Integrated Validation