Data pipelines and queues

Event bus that routes events between AWS services and apps.

Used byiFood

AWS's data warehouse for analytics with SQL.

Used byCreditas

Managed message queue on AWS.

Used byiFood,Olist

Apache Airflow

Open source

Schedules and monitors data pipelines written in Python.

Used byAirbnb,QuintoAndar

Apache Spark

Open source

Engine for processing large amounts of data across many machines.

Used byNotion,Nubank

Apache Superset

Open source

Open source dashboards and charts over your data, born at Airbnb.

Used byAirbnb

BigQuery

Open source

Google's serverless data warehouse for SQL analytics at scale.

Used byDiscord,Nubank

Orchestrates data pipelines with a focus on the data assets they produce.

Used byDiscord,PostHog

Databricks

Open source

Data and AI platform built on Spark for analytics, ETL and machine learning.

Used byiFood,Nubank,PicPay,QuintoAndar+1

Transforms data inside the warehouse with tested, versioned SQL.

Used byDiscord

Streams every change from your database as events (change data capture).

Used byNotion

Google Cloud Pub/Sub

Open source

Managed messaging between services on Google Cloud.

Used byDiscord,Linear

Durable background jobs and workflows for TypeScript, Python and Go.

Used byDocumenso

Kafka

Open source

Distributed event streaming: services publish and consume streams of events.

Used byAirbnb,iFood,Notion,Nubank+6

Background jobs for Ruby, backed by Redis.

Used byGitLab,RD Station

Snowflake

Open source

Cloud data warehouse for analytics on large datasets.

Used byNotion,Vercel

Temporal

Open source

Durable workflows: long-running jobs that survive crashes and retries.

Used byFigma,Linear,PostHog

Open source background jobs and long-running tasks for TypeScript.

Used byCal.com