Apache Spark: Deep Dive into internal architecture

Present Sir :)

Thanks Gaurav. I often work with AWS EMR (EC2) along with spark-3. 4.1 in my job. We have large batch processing clusters running on a daily scheduled Airflow DAGs. Acc to me if we use MapReduce with Spark it leverages computation with processing speed at a large scale.

Hi Gaurav, can you please do apache airflow

Bro if you have time please make a single video with everything covered up in your channel as a full video to system design

It will be more helpful 🙂

Nice one. Can you also do a deep dive on Google Dataflow and compare with Spark Streaming?

Hi Gaurav can you please do one on Druid

what makes you say that backprop will benefit by 1000x with spark? can you share the source?

Thanks

One more point I would like to add, RDD is made of Trie data structure that makes RDD to get the lineage and evaluate. if you can find out the RDD and Data Frame architecture docs and make a video it would be great help.

wait, what happens when results are too big for memory?