Public Thesis Defense of Donatien SCHMITZ - ICTEAM
sst |
Resource Management in Distributed Stream Processing: From Fine-Grain Elasticity to Proactive Planning
Thursday, February 19, 2026 - 3:00pm - Auditorium BARB94 -Place Sainte-Barbe, 1348 Louvain-la-Neuve
Distributed Stream Processing (DSP) systems have become essential for processing large-scale, real-time data streams in various applications, ranging from social media analytics to IoT data processing. DSP applications are represented as directed graphs of operators that continuously process incoming data tuples. These operators have diverse and dynamic resource requirements, making resource management a challenging task. Efficient resource management in DSP is crucial to ensure optimal performance, cost-effectiveness, and adaptability to dynamic workloads. Current approaches treat resource management either at a coarse granularity, leading to suboptimal resource utilization, or rely on reactive mechanisms that may not adequately capture the individual requirements of operators.
This thesis addresses the challenges of resource management in DSP by proposing novel approaches that span from fine-grain elasticity to proactive planning. First, we conduct an in-depth analysis of real-world DSP workloads to understand their characteristics and resource requirements. Based on these insights, we develop a fine-grain elasticity mechanism that allows DSP systems to dynamically adjust CPU and memory resources at the operator level, enabling more efficient resource utilization and improved performance under varying workloads. Next, we introduce a multi-query auto scaling framework that leverages fine-grain elasticity combined with optimized placement strategies to manage resources across multiple concurrent DSP applications. This framework aims to minimize resource costs while ensuring that performance objectives are met for all applications. Finally, we propose a proactive resource planning approach that models the performance of individual DSP applications to predict their resource requirements for unseen workloads.
Jury members
Prof. Etienne Rivière (UCLouvain)(Supervisor)
Prof. Peter Van Roy (UCLouvain) (Chairperson)
Prof. Ramin Sadre (UCLouvain) (Secretary)
Dr. Anne-Cécile Orgerie (CNRS, Rennes, France)
Prof. Romain Rouvoy (Inria, Univ. Lille, France)
Mr. Sabri Skhiri (Eura Nova, Belgique)