Distributed System comparison

less than 1 minute read

Again, I summarized the comparison between each distributed systems here. Couple of interesting points to DASK

Hadoop

runs at seconds in disk, SLOW

Spark

10~100 ms
Barrel Mode
Gang scheduling
DL SGD is not good fit into MapReduce

DASK

micro-seconds
peaked in 2021
declined (data doesn NOT show, dependency for other systems)

BDFL Benevolent Dictator For Life

Ray

Rely on upstream SQL and ML preprocessing support
Less support for deploymet

pyArrow

data storage memery format

Twitter Facebook LinkedIn

You May Also Enjoy

TensorRL-LLM

October 30 2024

To understand NIM, you can not avoid deep undertanding of TRT-LLM, Triton and even vLLM. Those will be focus for the near future.

Lookahead

October 24 2024

Last blog about Medusa was going longer than I expected. So I will write separete blogs about lookahead and EAGLE1/2

Concurrency Execution

October 21 2024

It’s all over the internet about how KAN will revolutionize ML by replacing MLP. Here are my frist read about KAN

Boltzmann Machine

October 20 2024

I started learning ML with Andrew Ng’s course, and at the same time, I also took Neuron Network from Hinton. The second one is actually very hard for me and ...