API World + CloudX + Dev Innovation Summit 2024

Mohammed Amine Garmes, Amadeus, Principal Data Engineer
Liam Mulhall, Amadeus, Innovation Leader

The Amadeus Hexadata Platform (AHP) is an advanced suite of capabilities for constructing and managing multiple End-to-End (E2E) big data pipelines.
- Processes approximately 2 billion transactions daily.
- Fully migrated from on-premise to Azure Cloud.
- Data size 10PB.
Data Integration:
- Integrates data from around 6,000 servers across various Amadeus divisions and external clients.
- 90% of data handled via Kafka, hosted on Azure or within a private data center.
- Utilizes Mirror Maker to replicate data across four Kafka clusters.
- Utilizes KafkaConnectors to stream files to Kafka
Technologies Used:
KStreams, Databricks, Kafka Connectors, Airflow, Internal Framework
Additional Capabilities:
Operational on OpenShift. (4000 Pods)
ElasticSearch
MLOPS pipeline : CICDCD for ML models on the platform in Azure. - on-going -
Key Benefits:
Provides scalable, reliable, and efficient big data processing and analytics solutions.