DATA_INFRA.md

Reading

• Machine Learning: The High Interest Credit Card of Technical Debt
• The Care & Feeding of Data Scientists
• Designing Data-Intensive Applications
• Storytelling with Data: A Data Visualization Guide for Business Professionals

Distributed Frameworks

• spark
• https://github.com/ray-project/ray
• https://dask.org/

Warehouse

• redshift
• snowflake
• vertica

Data Lake

• databricks delta
• https://www.dremio.com/

Change data capture / streaming

• https://www.attunity.com/
• https://streamsets.com/
• https://github.com/debezium/debezium

Streaming

• kafka
• flink
• beam

Notebooks / exploration

• jupyter
• https://zeppelin.apache.org/
• https://github.com/ironmussa/Optimus

Labeling

• https://www.snorkel.org/
• https://www.figure-eight.com/
• https://scale.ai
• https://www.labelbox.com

Orchestration

• airflow
• luigi
• oozie
• https://github.com/dagster-io/dagster
• https://github.com/PrefectHQ/prefect
• https://www.getdbt.com/product/

Dataflow/transformation managers

• streamset
• nifi

Notebook management

• https://github.com/nteract/papermill

Solutions

• snowplow
• https://www.prefect.io/
• https://www.ascend.io/
• https://www.datmo.com/
• TFX-OSS

A/B testing

• https://www.optimizely.com/
• https://github.com/YahooArchive/mendel

Behavioral analytics

• https://www.fullstory.com/
• https://www.hotjar.com/

Feature flagging

• https://launchdarkly.com/
• https://www.split.io/
• https://github.com/checkr/flagr

Catalogue

• https://github.com/airbnb/knowledge-repo
• https://github.com/lyft/amundsen

BI

• superset
• looker
• periscope

piping data around for salesOps/marketingOps/CRM etc

• https://tray.io/
• https://www.snaplogic.com/
• https://www.polytomic.com/
• https://www.getcensus.com/

Sales analytics

• https://www.chorus.ai/
• https://www.gong.io/

Neat utils

• query CSVs w/ SQL: cq

Generating fake data

• https://github.com/lk-geimfari/mimesis