Retour aux activités
Discussion DS4DM autour d'un café

Hindsight Learning for MDPs with Exogenous Inputs


13 fév. 2024   11h00 — 12h00

Sean Sinclair MIT, États-Unis

Sean Sinclair

Présentation sur YouTube.

Many resource management problems require sequential decision-making under uncertainty, where the only uncertainty affecting the decision outcomes are exogenous variables outside the control of the decision-maker. We model these problems as Exo-MDPs (Markov Decision Processes with Exogenous Inputs) and design a class of data-efficient algorithms for them termed Hindsight Learning (HL). Our HL algorithms achieve data efficiency by leveraging a key insight: having samples of the exogenous variables, past decisions can be revisited in hindsight to infer counterfactual consequences that can accelerate policy improvements. We compare HL against classic baselines in the multi-secretary and airline revenue management problems. We also scale our algorithms to a business-critical cloud resource management problem: allocating Virtual Machines (VMs) to physical machines, and simulate their performance with real datasets from a large public cloud provider. We find that HL algorithms outperform domain-specific heuristics, as well as state-of-the-art reinforcement learning methods.

Paper Link:

Bio: Sean Sinclair is a Postdoctoral Associate with the Laboratory for Information and Decision Sciences at Massachusetts Institute of Technology. He received his Ph.D. in Operations Research and Information Engineering at Cornell University, co advised by Sid Banerjee and Christina Yu. Prior to that, he finished his undergraduate degree in Mathematics and Computer Science from McGill University and afterwards served as a math teacher in Ghana with the Peace Corps. His research focuses on developing algorithms for data-driven sequential decision making in societal systems.

Federico Bobbio responsable
Defeng Liu responsable


Activité hybride au GERAD
Zoom et salle 4488
Pavillon André-Aisenstadt
Campus de l'Université de Montréal
2920, chemin de la Tour

Montréal Québec H3T 1J4

Axe de recherche

Application de recherche