Shortened URL Resolver System

The Challenge

The telecommunications industry requires the ability to resolve shortened URLs in real-time for SMS and messaging services. However, traditional HTTP proxy methods face a fundamental limitation: when using the HTTP CONNECT method for HTTPS traffic, the proxy cannot see the full URL being requested. The HTTP client first establishes an SSL tunnel to the target host, and then sends the actual request through that encrypted tunnel, making URL extraction impossible.

This created several critical business problems:

• SSL Interception Challenge: Standard proxy methods cannot extract full URLs from HTTPS requests, blocking resolution of encrypted shortened URLs
• Performance Requirements: The system needed to handle high-volume traffic with minimal latency for real-time messaging services
• Scalability Concerns: Without proper caching and optimization, the system would struggle under production load
• Observability Gaps: Lack of comprehensive monitoring and reporting made it difficult to track system health, performance metrics, and business insights
• Reliability Issues: Network failures, timeouts, and retry logic needed robust handling to ensure service availability

The inability to resolve shortened URLs directly impacts user experience in messaging platforms, where users expect immediate link resolution and delivery.

The Solution

Architecture Design

Architectural Decisions

1. Squid Proxy with SSL Bump: Chose Squid Proxy with SSL Bump functionality to intercept and decrypt HTTPS traffic, enabling full URL extraction. This acts as a man-in-the-middle proxy specifically for URL resolution purposes, allowing the system to see the complete URL path before the SSL tunnel is established.

2. Reactive Programming Model: Implemented the system using Spring WebFlux and Project Reactor to handle high-concurrency scenarios efficiently. The non-blocking reactive model ensures optimal resource utilization and scalability for high-throughput URL resolution requests.

3. Multi-Layer Caching Strategy: Designed a two-tier caching approach with Redis for resolved URL caching and in-memory caching for whitelist domains. This reduces database load and improves response times significantly for frequently accessed URLs.

4. Intelligent Retry and Fallback Mechanisms: Implemented configurable retry policies with exponential backoff for network failures, connection timeouts, and transient errors. The system gracefully falls back to direct resolution when cache operations fail, ensuring high availability.

5. Comprehensive Observability Architecture: Built a complete observability stack with structured logging, SNMP notifications, event publishing, automated data loading, and reporting rollups. This enables real-time monitoring, historical analysis, and proactive issue detection.

Key Contributions & Problem Solutions

Tech Stack

Impact & Results

Performance Improvements

The system achieved significant performance improvements through multiple optimization initiatives:

• Throughput Enhancement: Increased concurrent connection handling capacity by 200% through proxy configuration optimization
• Response Time Optimization: Achieved sub-second response times through intelligent caching and reactive programming model
• Cache Efficiency: Implemented multi-layer caching strategy reducing database load and improving response times for frequently accessed URLs
• Resource Management: Optimized connection pooling and session management preventing resource leaks and improving stability

Reliability & Availability

Comprehensive reliability improvements ensure high system availability:

• Graceful Degradation: Implemented fallback mechanisms allowing system to continue operating when cache or database services experience issues
• Error Recovery: Built robust retry mechanisms with exponential backoff for transient failures
• Status Tracking: Implemented comprehensive status tracking system enabling idempotent operations and partial recovery from failures
• Monitoring Integration: Integrated SNMP notifications and comprehensive logging for proactive issue detection

Observability & Analytics

Complete observability infrastructure provides real-time insights:

• Real-time Dashboards: Deployed Grafana dashboards providing real-time visibility into system performance, business metrics, and health indicators
• Automated Reporting: Implemented automated hourly and daily rollup scripts with comprehensive status tracking
• Structured Logging: Comprehensive logging with correlation IDs enabling efficient troubleshooting and analysis
• Historical Analysis: Summary tables and rollup infrastructure enable trend analysis and historical reporting

Business Impact

The system delivers significant business value:

• Service Availability: High availability ensures uninterrupted URL resolution for messaging services
• Operational Efficiency: Automated reporting and monitoring reduce operational overhead and enable proactive management
• Scalability: Reactive architecture and optimized configurations support high-volume traffic requirements
• Maintainability: Comprehensive testing, documentation, and code quality improvements reduce long-term maintenance costs

Key Learnings

1. SSL Interception Complexity: Implementing SSL interception for URL extraction requires careful consideration of security implications and proper certificate management. The Squid SSL Bump approach provides a viable solution but requires thorough understanding of SSL/TLS protocols.

2. Reactive Programming Benefits: The reactive programming model with Spring WebFlux and Project Reactor proved highly effective for high-concurrency scenarios, enabling efficient resource utilization and scalability.

3. Multi-Layer Caching Strategy: Combining Redis caching with in-memory caching for different data types (resolved URLs vs. whitelist domains) provides optimal performance while maintaining data consistency.

4. Observability as First-Class Citizen: Building comprehensive observability infrastructure from the beginning pays dividends in production operations, enabling proactive issue detection and efficient troubleshooting.

5. Idempotent Operations: Implementing idempotent operations with status tracking for batch processing (rollup scripts) ensures reliability and enables safe retry mechanisms.

6. Test Infrastructure Investment: Investing in proper test infrastructure (Testcontainers, H2) significantly improves development velocity and code quality.