Tools to increase data infrastructure robustness and resilience

data-concepts-and-tools.md

[approaches] Increase data infrastructure robustness and resiliency

With increasing scalability comes the need to use strategies and approaches to ensure the availability of our services. Here I list some very important tools and concepts to achieve these goals.

Most of these tools are multifunctional.

Data Catalog

• Amundsen, the leading open source data catalog

Data Observability

what is data observability

three pillars of observability data

• Metrics refer to a numeric representation of data measured over time.
• Logs, a record of an event that took place at a given timestamp, also provide valuable context regarding when a specific event occurred.
• Traces represent causally related events in a distributed environment.

• Prometheus, from metrics to insight
• Grafana
• Elastic, centralizador de logs
• DataHub

Data collectors

• Elastic Beats
• Fluentd

Data Reliability

Introducing Soda Core: The New Way for Data Reliability

• Soda Core
• great_expectations
• Datafold

Data Virtualization

• Trino, novo Presto

Data Discovery and Visualization

• Metabase
• Apache Superset

Data Modeling

• Delta lake archtecture ➡️ bronze(raw) ⇒ silver(treated) ⇒ gold(final)
• Data Mesh concept

Data orchestration

• Apache Airflow
• Prefect
• Dagster

Data processing

apache spark vs dask vs pandas

• Apache Spark
• Dask
• dbt

Data enrichment

Data scraping vs Data crawling "Data Crawling means dealing with large data sets where you develop your crawlers (or bots) which crawl to the deepest of the web pages. Data scraping, on the other hand, refers to retrieving information from any source (not necessarily the web)." By Trifacta

• Scrapy

More

https://www.snowflake.com/

https://www.rudderstack.com/

https://github.com/joelparkerhenderson/architecture-decision-record

https://nifi.apache.org/