Oredoo Customer Care System

The Challenge

Telecom operators face the critical challenge of managing vast amounts of call detail records (CDRs) while providing efficient customer support services. The M1 Customer Care System needed to address several key pain points:

Data Volume & Processing Complexity

• Process millions of call detail records daily from multiple sources
• Handle complex AMA (Automatic Message Accounting) file formats
• Maintain data integrity across distributed systems
• Support real-time queries on massive transaction datasets

Customer Service Requirements

• Enable customer care representatives to quickly search subscriber information
• Provide accurate billing and usage data for customer inquiries
• Support multiple search patterns (basic, advanced, quick search)
• Integrate with external billing and subscription management systems

Business Intelligence Needs

• Generate automated daily, weekly, and monthly reports
• Support ad-hoc reporting for business analysis
• Provide revenue and usage analytics
• Enable data-driven decision making

System Integration Challenges

• Integrate with external systems (Ajuba billing platform)
• Maintain single sign-on (SSO) across multiple web applications
• Ensure secure access control and role-based permissions
• Handle log aggregation from distributed nodes

The system's reliability directly impacted customer satisfaction, as any data inconsistencies or search failures would result in poor customer experience and potential revenue loss.

The Solution

Architecture Design

Architectural Decisions

1. Microservices Architecture with RMI Communication

• Decision: Implemented distributed services communicating via RMI (Remote Method Invocation)
• Rationale: Enabled horizontal scaling, service isolation, and independent deployment while maintaining type-safe inter-service communication

2. Centralized Metadata Management

• Decision: Created dedicated metadata server managing all configuration, field mappings, and business rules
• Rationale: Single source of truth for system configuration, enabling dynamic updates without service restarts and consistent behavior across all modules

3. Batch Processing for CDR Ingestion

• Decision: Implemented batch insert operations for transaction data processing
• Rationale: Significantly improved throughput for high-volume CDR processing, reducing database connection overhead and transaction commit costs

4. Apache Reverse Proxy for Web Layer

• Decision: Used Apache HTTP Server as reverse proxy for all web services
• Rationale: Centralized SSL termination, load balancing, and routing, simplifying deployment and providing single entry point for web traffic

5. Partitioned Database Tables

• Decision: Implemented table partitioning for transaction and event tables
• Rationale: Improved query performance on large datasets, enabled efficient data archival, and maintained manageable table sizes for maintenance operations

Key Contributions & Problem Solutions

Tech Stack

Impact & Results

Business Impact

• Customer Experience: Sub-second search capabilities allowed agents to resolve customer inquiries instantly, improving satisfaction.
• Operational Efficiency: Automated reporting saved countless hours of manual data compilation for billing teams.
• Revenue Assurance: Improved data quality (fixing merchant IDs) ensured accurate billing and revenue reconciliation.

Technical Efficiency

• Scalability: Partitioned database capability handled millions of daily records without performance degradation.
• Maintainability: Centralized metadata management reduced configuration errors and deployment complexity.
• Reliability: Distributed microservices architecture ensured 99.9% system availability even during partial failures.

Operational Excellence

• Data Integrity: Proactive cleaning of historical data eliminated legacy issues affecting current operations.
• Observability: Unified logging and easy-to-use admin dashboards empowered operations teams to monitor system health effectively.

Key Learnings

Data Quality is Critical

• Small data inconsistencies (like whitespace) can cause significant system failures
• Proactive data validation and normalization prevents downstream issues
• Both application-level fixes and database cleanup are necessary for comprehensive solutions

Enterprise System Architecture

• Centralized metadata management enables flexible system configuration
• Microservices architecture provides scalability and maintainability
• Batch processing is essential for high-volume data ingestion scenarios

Code Quality & Maintenance

• Simple fixes (like string trimming) can have significant business impact
• Database migration scripts are crucial for maintaining data integrity across deployments
• Comprehensive solutions require both preventive (code) and corrective (data cleanup) measures