Everything for Data Engineers
Data engineers move between SQL reviews, pipeline reliability discussions, and stakeholder reporting on data quality. This hub gathers the database and pipeline vocabulary, relevant grammar drills, and interview and blog resources tagged data, SQL, or pipelines.
Vocabulary sets
Grammar & writing
Interview prep
Related deep-dive hub
Blog articles (0)
No tagged blog posts yet for this role — check back soon, or browse the full blog.
Other role hubs
Explore more
Browse every exercise category, or search the full site.
Frequently Asked Questions
What is a Slowly Changing Dimension (SCD) Type 2 and when would I use it?
An SCD Type 2 update involves replacing an existing record with new values while preserving historical data. This is crucial for dimensions where attributes change over time, like customer address or product category, ensuring accurate reporting across different periods without creating phantom records.
Explain the difference between a materialized view and an index in a SQL database.
A materialized view pre-computes and stores query results, improving performance for frequently executed queries. An index is a data structure that speeds up searches by creating pointers to rows based on specific column values; it doesn't store the entire result set.
What are Delta Lake transactions, and why are they important for Data Lakes?
Delta Lake provides ACID (Atomicity, Consistency, Isolation, Durability) properties to data lakes by implementing optimistic concurrency control and versioning of changes. This ensures reliable updates and prevents data corruption when multiple users or processes modify the same files simultaneously.
What's a Kafka Connect cluster, and how does it relate to ETL?
Kafka Connect is a framework for streaming data between Apache Kafka and other systems – databases, search indexes, etc. It simplifies the process of building and managing ETL pipelines by handling connectivity, serialization/deserialization, and fault tolerance.
Can you describe the concept of 'schema evolution' in a data warehouse?
Schema evolution refers to the ability to modify the structure of a database schema (adding columns, changing datatypes) without disrupting existing queries or applications. Techniques like forward and backward compatibility are key to managing changes gracefully during data warehousing development.
What is the purpose of a 'data lineage' tool and why is it valuable?
Data lineage tracks the origin, transformations, and movement of data throughout its lifecycle. It provides visibility into how data has been processed, aiding in debugging issues, ensuring compliance with regulations (like GDPR), and understanding impact analysis when changes are made.
What is a 'data quality rule' and how do I implement it?
A data quality rule defines criteria for assessing the accuracy, completeness, consistency, and validity of data. Implementation involves defining these rules within a data quality tool or ETL pipeline, triggering checks during data ingestion or transformation to identify and rectify anomalies.
Explain 'window functions' in SQL – what are they used for?
Window functions perform calculations across a set of rows that are related to the current row, without grouping. They're valuable for tasks like calculating running totals, moving averages, or ranking within partitions of data – providing more sophisticated analytical capabilities than standard aggregate functions.
What is a 'data mesh' architecture and how does it differ from a centralized data warehouse?
A data mesh is a decentralized approach to data management, where domain teams own their data products and are responsible for making them accessible to others. Unlike a centralized data warehouse, it promotes self-service data access and reduces dependencies on a central IT team.
What's the difference between 'star schema' and 'snowflake schema' in dimensional modeling?
'Star schema' is a simple design with a central fact table surrounded by dimension tables, optimizing query performance for common reporting scenarios. 'Snowflake schema' extends this by normalizing dimension tables further, potentially increasing complexity but also reducing redundancy – often chosen when dealing with highly normalized source systems.