Zach Wilson's Recent LinkedIn Posts

Senior data engineers at Netflix can make $500k per year. The interview process is intense though: - SQL Make sure you know: - window functions - table scans and optimization techniques - indices and data modeling - Data pipeline Make sure you know: - the failure modes of Spark - when to use Kafka vs Spark Stream…

AI and data engineering are so similar it's painful! - Agent workflows = data pipelines - Chunking text = data modeling - Embeddings = dimensionality reduction - Retrieval = indexing and partitioning - Vector search = join optimization - Prompt engineering = query tuning - Guardrails = data quality checks If you d…

Don’t be afraid to reinvent yourself! Don’t stop learning the lessons the universe is throwing at you. Sleep well to allow your brain strong plasticity so you don’t have to relearn painful lessons as many times in a row. Remember you can fail for 7 years straight and still be young and able to continue taking r…

I’m glad I relied on StackOverflow for regexes long enough so that AI can do them for me now and I can avoid learning them entirely. What skill do you feel smart avoiding learning because AI can now do it?

Databricks is making Spark brain dead simple and it's both happy and sad. The only knob that is left is spark.sql.shuffle.partitions. This determines the number of chunks that Spark processes your data. You can scale it up and down with repartition and coalesce. Although I only use coalesce near the end of the j…

I hate to say it but product manager is the safest role from AI. Data engineer will feel pressure. Analytics engineer will feel less pressure. Data scientist will feel even less pressure. PM will be elevated. CEO will be put on the stratosphere. Why is PM so safe? Strategic roles that involve tons of in person int…