Siemens Energy: Event-Driven, AI-Powered Data Pipeline on AWS
Challenge
Siemens Energy’s data division manages enormous volumes of product measurement data, such as documents, images, audio recordings, and video files, generated across global manufacturing operations, turbines or other facilities and equipment.
The team needed a system that could:
- Ingest files of arbitrary size from kilobytes to multi-gigabyte measurement datasets and high-resolution imagery without hitting payload limits or degrading performance.
- Automatically extract rich metadata from every uploaded file, regardless of modality (text, image, audio, video), using AI. However, when certain metadata were known upfront, they wanted the ability to mix them into the generated metadata.
- Enforce partition-level access control so that different teams and users can only see and modify data within their designated scope, integrated with Siemens Energy’s existing Microsoft Entra ID (Azure AD) SSO identity provider.
- Scale to hundreds of thousands of files without the lifecycle management limitations of tools like OneDrive or SharePoint (~300,000 file ceiling).
- Provide a web-based interface for non-technical users to upload, browse, download, and preview files after authenticating with corporation-wide SSO.
The system had to be production-grade, stress-tested, and deployable across multiple environments (sandbox, development, UAT, production) with full Infrastructure as Code (IaC).
Siemens Energy’s data division manages enormous volumes of product measurement data, such as documents, images, audio recordings, and video files, generated across global manufacturing operations, turbines or other facilities and equipment.
The team needed a system that could:
- Ingest files of arbitrary size from kilobytes to multi-gigabyte measurement datasets and high-resolution imagery without hitting payload limits or degrading performance.
- Automatically extract rich metadata from every uploaded file, regardless of modality (text, image, audio, video), using AI. However, when certain metadata were known upfront, they wanted the ability to mix them into the generated metadata.
- Enforce partition-level access control so that different teams and users can only see and modify data within their designated scope, integrated with Siemens Energy’s existing Microsoft Entra ID (Azure AD) SSO identity provider.
- Scale to hundreds of thousands of files without the lifecycle management limitations of tools like OneDrive or SharePoint (~300,000 file ceiling).
- Provide a web-based interface for non-technical users to upload, browse, download, and preview files after authenticating with corporation-wide SSO.
The system had to be production-grade, stress-tested, and deployable across multiple environments (sandbox, development, UAT, production) with full Infrastructure as Code (IaC).
Solution
We designed and delivered the Product Measurement Data Pipeline (PMDP) — a suite of interconnected microservices and interfaces, all running serverless on AWS.
Architecture Overview
The platform consists of:
- File Manager API: A RESTful service handling file CRUD, multi-chunk uploads, presigned URL generation, metadata operations, and partition-scoped access control.
- File Metadata Enrichment Microservice: An event-driven, multi-modal AI pipeline that automatically extracts structured metadata from uploaded files using Amazon Bedrock Data Automation.
- Web Application: An AWS Amplify-hosted React application providing a Storage Browser UI with Siemens Energy SSO integration.
- Stress Test Suite: A CLI tool for validating the system under load with concurrent uploads and downloads of files up to multiple gigabytes.
All infrastructure is defined in Python CDK, validated and automatically deployed through CI/CD pipelines (self-hosted GitLab).
We designed and delivered the Product Measurement Data Pipeline (PMDP) — a suite of interconnected microservices and interfaces, all running serverless on AWS.
Architecture Overview
The platform consists of:
- File Manager API: A RESTful service handling file CRUD, multi-chunk uploads, presigned URL generation, metadata operations, and partition-scoped access control.
- File Metadata Enrichment Microservice: An event-driven, multi-modal AI pipeline that automatically extracts structured metadata from uploaded files using Amazon Bedrock Data Automation.
- Web Application: An AWS Amplify-hosted React application providing a Storage Browser UI with Siemens Energy SSO integration.
- Stress Test Suite: A CLI tool for validating the system under load with concurrent uploads and downloads of files up to multiple gigabytes.
All infrastructure is defined in Python CDK, validated and automatically deployed through CI/CD pipelines (self-hosted GitLab).
Results
File size support: Unlimited (tested up to 5GB+)
Upload concurrency: 20+ simultaneous uploads
Metadata extraction: Automatic for documents, images, audio, and video
Access control: Per-user and per-group partition isolation via S3 Access Grants
Environments: 4 (sandbox, development, UAT, production) from a single CDK codebase
Identity integration: Microsoft Entra ID SSO with OIDC federation
Infrastructure: 100% Infrastructure as Code (Python CDK)
File size support: Unlimited (tested up to 5GB+)
Upload concurrency: 20+ simultaneous uploads
Metadata extraction: Automatic for documents, images, audio, and video
Access control: Per-user and per-group partition isolation via S3 Access Grants
Environments: 4 (sandbox, development, UAT, production) from a single CDK codebase
Identity integration: Microsoft Entra ID SSO with OIDC federation
Infrastructure: 100% Infrastructure as Code (Python CDK)