2021 was an year where I got to fulfill some of my goals. I got a greater track of writing articles (most of them reading updates) and I got to talk at a podcast and at Coalesce. Of course this meant getting new knowledge in areas like dbt, data testing. I also got started on a second brain which is helping me to learn more and better (writing really helps). And these have been some of my main goals.

I got introduced into Scala and technologies like Spark but for 2022 I’m hoping to:

• Read Scala book
• Read flink book
• Work with apache iceberg either in a batch and a streaming job
• Talk in two conferences/meetups
• Write more on the second brain (summarize of new things learned on each article)

Data Engineering

• Don’t Let the Internet Dupe You, Event Sourcing is Hard - Event sourcing is great but, at least for now, can be quite harder than batch mode
• Metadata Indexing in Iceberg - The creator of iceberg gives some insights into how it’s metadata indexing works
• Iceberg Spec - not an article but I think it’s a must for those trying to use apache iceberg
• How to ETL at Petabyte-Scale with Trino - some ideas on how to have etl using trino
• Announcing OpenMetadata - Standardization is great and openmetadata is a step in the correct direction
• How Uber Achieves Operational Excellence in the Data Quality Experience - Another intake on how uber has standardized it’s data ops and how it ensures quality
• Launching at LinkedIn: The Story of Apache Pinot - A history on the how apache Pinot started
• Revisiting Java in 2021 - Java 17 has gotten some great features since Java8 and this article tries to summarize them
• Using an ETL framework vs writing yet another ETL script - Airbyte is an open source version of fivetran and tries to convene how it’s the best approach vs closed source ones
• Series on Data protection at Airbnb
  • Automating Data Protection at Scale, Part 1
  • Automating Data Protection at Scale, Part 2
  • Automating Data Protection at Scale, Part 3