Enterprise Data Synchronization Architecture: OCI + API Gateway + Spring Boot
Executive Summary
As a Senior Java Software Engineer with 20+ years of experience, I recommend a microservices-based architecture using OCI native services, API Gateway, and Spring Boot to replace OIC. This approach provides better control, scalability, and cost-effectiveness for high-volume data synchronization scenarios.
🏗️ Comprehensive Architecture
High-Level Architecture Diagram
architecture-beta group oci(cloud)[OCI Cloud Infrastructure] service lb(internet)[OCI Load Balancer] in oci service apigw(internet)[API Gateway] in oci service waf(shield)[Web Application Firewall] in oci group compute(server)[Compute Layer] in oci service sync1(server)[Sync Service Pod 1] in compute service sync2(server)[Sync Service Pod 2] in compute service sync3(server)[Sync Service Pod 3] in compute group data(database)[Data Layer] in oci service adb(database)[Autonomous DB] in data service objstore(disk)[Object Storage] in data service redis(disk)[Redis Cache] in data service streaming(disk)[OCI Streaming] in data group integration(server)[Integration Layer] in oci service adapter1(server)[ERP Adapter] in integration service adapter2(server)[CRM Adapter] in integration service adapter3(server)[Legacy Adapter] in integration group monitoring(server)[Observability] in oci service apm(server)[APM] in monitoring service logging(disk)[Logging Analytics] in monitoring service external1(internet)[External System 1] service external2(internet)[External System 2] waf:R --> L:lb lb:R --> L:apigw apigw:B --> T:sync1 apigw:B --> T:sync2 apigw:B --> T:sync3 sync1:R --> L:adb sync2:R --> L:objstore sync3:R --> L:redis sync1:B --> T:streaming sync2:B --> T:streaming sync1:R --> L:adapter1 sync2:R --> L:adapter2 sync3:R --> L:adapter3 adapter1:R --> L:external1 adapter2:R --> L:external2 sync1:T --> B:apm sync2:T --> B:logging
🎯 Detailed Component Architecture
Data Synchronization Flow
sequenceDiagram
participant Client
participant APIGateway as API Gateway
participant SyncService as Sync Service
participant ObjectStorage as Object Storage
participant Streaming as OCI Streaming
participant Adapter as System Adapter
participant ExternalSystem as External System
participant ADB as Autonomous DB
Client->>APIGateway: POST /sync/data (>10MB payload)
APIGateway->>APIGateway: Validate JWT & Rate Limit
APIGateway->>SyncService: Forward Request
alt Large Payload (>10MB)
SyncService->>ObjectStorage: Store payload with UUID
SyncService->>Streaming: Publish sync event
SyncService-->>Client: 202 Accepted (Job ID)
Note over Streaming,Adapter: Async Processing
Streaming->>Adapter: Consume event
Adapter->>ObjectStorage: Retrieve payload
Adapter->>Adapter: Transform data
Adapter->>ExternalSystem: Sync data (chunked)
ExternalSystem-->>Adapter: Acknowledgment
Adapter->>ADB: Update sync status
Adapter->>Streaming: Publish completion event
else Small Payload (<10MB)
SyncService->>Adapter: Direct sync call
Adapter->>ExternalSystem: Sync data
ExternalSystem-->>Adapter: Response
Adapter-->>SyncService: Result
SyncService-->>Client: 200 OK
end
Client->>APIGateway: GET /sync/status/{jobId}
APIGateway->>SyncService: Get status
SyncService->>ADB: Query status
ADB-->>SyncService: Status details
SyncService-->>Client: Status response
📋 Key Recommendations
1. OCI Services Stack
| Component | OCI Service | Purpose |
|---|---|---|
| API Management | OCI API Gateway | Rate limiting, authentication, routing |
| Compute | OCI Container Engine (OKE) | Run Spring Boot microservices |
| Database | Autonomous Database (ATP/ADW) | Transactional data & sync metadata |
| Large File Storage | Object Storage | Store payloads >10MB |
| Message Queue | OCI Streaming / Queue | Async processing & event-driven architecture |
| Cache | OCI Cache with Redis | Performance optimization |
| Load Balancing | OCI Load Balancer | High availability & traffic distribution |
| Monitoring | APM + Logging Analytics | Observability & troubleshooting |
| Security | WAF + Vault | DDoS protection & secrets management |
2. Spring Boot Microservices Architecture
Service Structure
├── sync-gateway-service # API Gateway aggregation layer
├── sync-orchestrator-service # Orchestration & workflow management
├── adapter-erp-service # ERP system adapter
├── adapter-crm-service # CRM system adapter
├── adapter-legacy-service # Legacy system adapter
├── data-transformer-service # Data transformation & mapping
├── sync-status-service # Status tracking & monitoring
└── common-lib # Shared utilities & models
Key Spring Boot Dependencies
<!-- Core Spring Boot --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> <!-- For reactive/async --> </dependency> <!-- OCI SDK --> <dependency> <groupId>com.oracle.oci.sdk</groupId> <artifactId>oci-java-sdk-objectstorage</artifactId> </dependency> <dependency> <groupId>com.oracle.oci.sdk</groupId> <artifactId>oci-java-sdk-streaming</artifactId> </dependency> <!-- Resilience --> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-spring-boot2</artifactId> </dependency> <!-- Monitoring --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> <!-- Caching --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency>
3. Handling Large Data Volumes (>10MB)
Strategy: Hybrid Sync Pattern
@Service public class DataSyncService { private static final long LARGE_PAYLOAD_THRESHOLD = 10 * 1024 * 1024; // 10MB @Autowired private ObjectStorageService objectStorageService; @Autowired private StreamingService streamingService; @Autowired private DirectSyncService directSyncService; public SyncResponse syncData(SyncRequest request) { long payloadSize = calculateSize(request.getData()); if (payloadSize > LARGE_PAYLOAD_THRESHOLD) { return handleLargePayload(request); } else { return handleSmallPayload(request); } } private SyncResponse handleLargePayload(SyncRequest request) { // 1. Store in Object Storage String objectId = objectStorageService.upload( request.getData(), "sync-bucket" ); // 2. Publish event to OCI Streaming SyncEvent event = SyncEvent.builder() .jobId(UUID.randomUUID().toString()) .objectStorageId(objectId) .targetSystem(request.getTargetSystem()) .timestamp(Instant.now()) .build(); streamingService.publish("sync-stream", event); // 3. Return async response return SyncResponse.builder() .jobId(event.getJobId()) .status(SyncStatus.PROCESSING) .message("Large payload processing initiated") .build(); } private SyncResponse handleSmallPayload(SyncRequest request) { // Direct synchronous processing return directSyncService.sync(request); } }
java
Copy
public class DataSyncService { private static final long LARGE_PAYLOAD_THRESHOLD = 10 * 1024 * 1024; // 10MB private ObjectStorageService objectStorageService; private StreamingService streamingService; private DirectSyncService directSyncService; public SyncResponse syncData(SyncRequest request) { long payloadSize = calculateSize(request.getData()); if (payloadSize > LARGE_PAYLOAD_THRESHOLD) { return handleLargePayload(request); } else { return handleSmallPayload(request); } } private SyncResponse handleLargePayload(SyncRequest request) { // 1. Store in Object Storage String objectId = objectStorageService.upload( request.getData(), “sync-bucket” ); // 2. Publish event to OCI Streaming SyncEvent event = SyncEvent.builder() .jobId(UUID.randomUUID().toString()) .objectStorageId(objectId) .targetSystem(request.getTargetSystem()) .timestamp(Instant.now()) .build(); streamingService.publish(“sync-stream”, event); // 3. Return async response return SyncResponse.builder() .jobId(event.getJobId()) .status(SyncStatus.PROCESSING) .message(“Large payload processing initiated”) .build(); } private SyncResponse handleSmallPayload(SyncRequest request) { // Direct synchronous processing return directSyncService.sync(request); } }
Object Storage Configuration
java
Copy
public class ObjectStorageConfig { public ObjectStorage objectStorageClient() { return ObjectStorageClient.builder() .region(Region.US_ASHBURN_1) .build(AuthenticationDetailsProvider.builder().build()); } public MultipartUploadConfig multipartConfig() { return MultipartUploadConfig.builder() .partSize(5 * 1024 * 1024) // 5MB chunks .parallelUploads(5) .build(); } }
4. Custom Adapter Pattern (Replacing OIC Adapters)
java
Copy
// Base Adapter Interface public interface SystemAdapter<T, R> { R sync(T data); boolean healthCheck(); AdapterMetrics getMetrics(); } // ERP Adapter Implementation public class ERPAdapter implements SystemAdapter<ERPSyncRequest, ERPSyncResponse> { private RestTemplate restTemplate; private CircuitBreaker circuitBreaker; private RateLimiter rateLimiter; public ERPSyncResponse sync(ERPSyncRequest data) { log.info(“Syncing data to ERP system: {}”, data.getEntityId()); // Transform data to ERP format ERPPayload payload = transformToERPFormat(data); // Call ERP API with chunking for large data if (payload.getSize() > CHUNK_SIZE) { return syncInChunks(payload); } HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON); headers.setBearerAuth(getERPToken()); HttpEntity<ERPPayload> request = new HttpEntity<>(payload, headers); ResponseEntity<ERPSyncResponse> response = restTemplate.exchange( erpConfig.getEndpoint() + “/api/v1/sync”, HttpMethod.POST, request, ERPSyncResponse.class ); return response.getBody(); } private ERPSyncResponse syncInChunks(ERPPayload payload) { List<ERPPayload> chunks = chunkPayload(payload, CHUNK_SIZE); List<CompletableFuture<ERPSyncResponse>> futures = chunks.stream() .map(chunk -> CompletableFuture.supplyAsync(() -> syncChunk(chunk))) .collect(Collectors.toList()); return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])) .thenApply(v -> aggregateResponses(futures)) .join(); } public boolean healthCheck() { try { ResponseEntity<String> response = restTemplate.getForEntity( erpConfig.getEndpoint() + “/health”, String.class ); return response.getStatusCode().is2xxSuccessful(); } catch (Exception e) { log.error(“ERP health check failed”, e); return false; } } }
5. High Availability & Performance Strategies
A. Kubernetes Deployment (OKE)
yaml
Copy
apiVersion: apps/v1 kind: Deployment metadata: name: sync-service spec: replicas: 3 # Minimum 3 for HA strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 template: spec: containers: – name: sync-service image: iad.ocir.io/namespace/sync-service:latest resources: requests: memory: “2Gi” cpu: “1000m” limits: memory: “4Gi” cpu: “2000m” livenessProbe: httpGet: path: /actuator/health/liveness port: 8080 initialDelaySeconds: 60 periodSeconds: 10 readinessProbe: httpGet: path: /actuator/health/readiness port: 8080 initialDelaySeconds: 30 periodSeconds: 5 apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: sync-service-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: sync-service minReplicas: 3 maxReplicas: 10 metrics: – type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 – type: Resource resource: name: memory target: type: Utilization averageUtilization: 80
B. Resilience Configuration
yaml
Copy
# application.yml resilience4j: circuitbreaker: instances: erpAdapter: registerHealthIndicator: true slidingWindowSize: 10 minimumNumberOfCalls: 5 permittedNumberOfCallsInHalfOpenState: 3 automaticTransitionFromOpenToHalfOpenEnabled: true waitDurationInOpenState: 30s failureRateThreshold: 50 retry: instances: erpAdapter: maxAttempts: 3 waitDuration: 2s exponentialBackoffMultiplier: 2 ratelimiter: instances: erpAdapter: limitForPeriod: 100 limitRefreshPeriod: 1s timeoutDuration: 5s bulkhead: instances: erpAdapter: maxConcurrentCalls: 50 maxWaitDuration: 10s
C. Caching Strategy
java
Copy
public class CacheConfig { public RedisCacheManager cacheManager(RedisConnectionFactory connectionFactory) { RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig() .entryTtl(Duration.ofMinutes(10)) .serializeKeysWith(RedisSerializationContext.SerializationPair .fromSerializer(new StringRedisSerializer())) .serializeValuesWith(RedisSerializationContext.SerializationPair .fromSerializer(new GenericJackson2JsonRedisSerializer())); return RedisCacheManager.builder(connectionFactory) .cacheDefaults(config) .withCacheConfiguration(“syncStatus”, config.entryTtl(Duration.ofMinutes(5))) .withCacheConfiguration(“adapterHealth”, config.entryTtl(Duration.ofMinutes(1))) .build(); } } public class SyncStatusService { public SyncStatus getStatus(String jobId) { return syncRepository.findByJobId(jobId); } public void updateStatus(String jobId, SyncStatus status) { syncRepository.updateStatus(jobId, status); } }
6. API Gateway Configuration
yaml
Copy
# OCI API Gateway Deployment Specification specification: requestPolicies: authentication: type: JWT tokenHeader: Authorization publicKeys: type: REMOTE_JWKS uri: https://identity.oraclecloud.com/oauth2/v1/keys rateLimiting: rateInRequestsPerSecond: 100 rateKey: CLIENT_IP cors: allowedOrigins: – “*” allowedMethods: – GET – POST – PUT – DELETE allowedHeaders: – “*” routes: – path: /sync/data methods: – POST backend: type: HTTP_BACKEND url: http://sync-service.default.svc.cluster.local:8080/api/v1/sync requestPolicies: bodyValidation: required: true content: application/json: validationMode: ENFORCING – path: /sync/status/{jobId} methods: – GET backend: type: HTTP_BACKEND url: http://sync-service.default.svc.cluster.local:8080/api/v1/status/{jobId}
7. Monitoring & Observability
java
Copy
public class MetricsConfig { public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() { return registry -> registry.config() .commonTags(“application”, “sync-service”) .commonTags(“environment”, “production”); } } public class SyncMetricsService { private final Counter syncSuccessCounter; private final Counter syncFailureCounter; private final Timer syncDurationTimer; private final Gauge activeSyncsGauge; public SyncMetricsService(MeterRegistry registry) { this.syncSuccessCounter = Counter.builder(“sync.success”) .description(“Number of successful syncs”) .tag(“type”, “data”) .register(registry); this.syncFailureCounter = Counter.builder(“sync.failure”) .description(“Number of failed syncs”) .tag(“type”, “data”) .register(registry); this.syncDurationTimer = Timer.builder(“sync.duration”) .description(“Time taken for sync operations”) .register(registry); this.activeSyncsGauge = Gauge.builder(“sync.active”, this, SyncMetricsService::getActiveSyncs) .description(“Number of active sync operations”) .register(registry); } }
📊 OIC vs Spring Boot Comparison
| Aspect | OIC | Spring Boot + OCI |
|---|---|---|
| Cost | High (subscription-based) | Lower (pay-per-use) |
| Flexibility | Limited to pre-built adapters | Full customization |
| Performance | Good for standard scenarios | Optimized for specific needs |
| Large Payloads | Limited (typically <10MB) | Unlimited (Object Storage) |
| Scalability | Auto-scaling (limited control) | Full control with OKE |
| Monitoring | Built-in dashboards | Custom APM + Prometheus |
| Development Speed | Faster for simple integrations | Requires more development |
| Vendor Lock-in | High | Lower (portable code) |
| Complex Logic | Limited | Full programming capability |
🚀 Implementation Roadmap
Phase 1: Foundation (Weeks 1-2)
- Set up OCI infrastructure (OKE, ADB, Object Storage)
- Configure API Gateway
- Implement base Spring Boot services
- Set up CI/CD pipeline
Phase 2: Core Services (Weeks 3-4)
- Develop sync orchestrator service
- Implement Object Storage integration
- Set up OCI Streaming
- Build adapter framework
Phase 3: Adapters (Weeks 5-6)
- Develop system-specific adapters
- Implement data transformation logic
- Add resilience patterns
- Performance testing
Phase 4: Observability (Week 7)
- Configure APM and logging
- Set up monitoring dashboards
- Implement alerting
- Load testing
Phase 5: Production (Week 8)
- Security hardening
- Production deployment
- Documentation
- Knowledge transfer
💡 Best Practices
- Use Reactive Programming for high-throughput scenarios (Spring WebFlux)
- Implement Idempotency for all sync operations
- Use Distributed Tracing (Jaeger/Zipkin) for debugging
- Implement Dead Letter Queues for failed messages
- Use Connection Pooling for database and HTTP clients
- Implement Graceful Shutdown for zero-downtime deployments
- Use Secrets Management (OCI Vault) for credentials
- Implement Health Checks at multiple levels
- Use Asynchronous Processing for large payloads
- Monitor Business Metrics not just technical metrics
🎓 Conclusion
This architecture provides a production-ready, scalable, and cost-effective alternative to OIC for high-volume data synchronization. The Spring Boot microservices approach gives you:
✅ Full control over performance optimization
✅ Better handling of large payloads (>10MB)
✅ Lower operational costs with OCI native services
✅ High availability through Kubernetes orchestration
✅ Custom adapters tailored to your specific needs
✅ Enterprise-grade observability and monitoring
The initial development effort is higher than OIC, but the long-term benefits in performance, cost, and flexibility make it the superior choice for your requirements.