Oct 7, 2023

Meetup potential for data stream | Readings

Web Dev

Progress: 5%

Tipping a bit again into typescript I’m foreseeing a lot of code for the next 2 weeks. And specially with testing I’ve gotten a lot of fun yesterday between checking components and integration tests. Sincerely if I get to deploy the changes with the same quality of a PR I reviewed this week I’ll feel a lot more confident. At least to move the progression bar to 20%.

Code challenges

Progress: 4%

Again no challenges done. Luckly this is one of those where I can stick with it for a couple of weeks and get the lost challenges done in a single swoop. But my escuse for having done none this week was due to have focus a bit more on the Data Stream project.

Data Stream

Progress: 10%

With the first loader prototype done I’ve gotten a proposal to go present in a meetup. For this to work I’ve decided to focus a bit on the capabilities of duckdb. So my plan is to structure the presentation in the following way:

Explain the problem with porto public transportation
Show the API
Show the loaders
- Sqlite
- Parquet
- Iceberg? (if done in time). Alternative is to show a glue job to migrate data into Iceberg
Show duckdb reading the data from the storages. Possibily as live demo
Show some examples of running queries on all data. How fast it is. Show integrations with python notebooks
Present some of the insights in the form of the dashboard

In terms of progress I truly would like to have the loaders set using either spark or flink but I’ll leave the finished data pipeline to after the presentation (middle of the November). From all the rest I’m thinking that I should have 80% of “Data Stream” done.

Communication

Progress: 0%

I’m presenting next week internally on some of the capabilities of Duckdb to my company. Some of this insights will be usefull for the presentation on November. The focus however will be on the meetup talk.

Wellness

Progress: 0%

Gotten back on track with the soccer game but not much luck with the other two. I’m getting aa kind of blocker to return so I’m thinking of doing 2 things.

In the morning I’ll just go for a quick walk before starting to work. During lunch this week I’ll pick on my bike and go visit the swimming pool to check how busy it is. I currently don’t have the car available so my only solution is to go do the workout during lunch with my bike.

Readings of the week

I’ve gotten more articles in the list than actually read. Luckly I also think I need to do a bit of trimming again as I added them with almost no criteria.