Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to Rest-Based Architecture

Slack modernized its data platform by replacing SSH based execution in Amazon EMR pipelines with a REST driven orchestration layer called Quarry. The migration covered 700 plus Airflow operators, improving security, reliability, and observability while eliminating direct SSH access across production clusters and enabling a server side job lifecycle model.

By Leela Kumili