[Haifux] Lecture Suggestion - Disco Project, an open source Map-Reduce framework based on Erlang and Python (mostly Python :-) )
Orna Agmon Ben-Yehuda
ladypine at gmail.com
Thu Jan 15 00:13:22 MSK 2009
In short, this is the method google use for parallel processing. Every
operation which needs to be parallel is divided into a mapping stage (where
each worker does something on their own data, and produces a result) and a
reduction stage, where the results of the map are collected into a
meaningful result. This parallelization scheme is highly scalable.
This (upcoming) Sunday's slides on Map-Reduce from the concurrent and
Distributed Programming course:
And a canonical paper, linked from the same place:
On Wed, Jan 14, 2009 at 10:27 PM, guy keren <choo at actcom.co.il> wrote:
> can you explain, briefly, what map-reduce is, so those not in the know
> (like me) will be able to decide if this is interesting? ;)
> Eran Sandler wrote:
> > Hi all,
> > It's been a while since I've posted to Haifux (or Linux-IL for that
> > matter) but I am watching the mailing list from time to time and due to
> > personal reasons found myself as a Haifa citizen for the past year (and
> > probably for a couple more years :-) ).
> > Somewhere in 2004 I even did a lecture on Mono, the open source .NET
> > implementation, if some of you recall.
> > Recently I've been involved with a cool open source project called Disco.
> > Disco is an open source Map-Reduce framework written in Erlang and
> > Python. It was written at Nokia's Palo Alto research center as a
> > lightweight framework for rapid scripting of distributed data processing
> > tasks but grew to become even more than that and is now even used for
> > probabilistic modeling, data mining, full text indexing, etc.
> > You can read more about Disco at http://discoproject.org
> > Would a lecture on Map-Reduce in general and specifically Disco would
> > interest people?
> > If so, I'm more than willing to give the lecture and show some examples.
> > Eran
> > ------------------------------------------------------------------------
> > _______________________________________________
> > Haifux mailing list
> > Haifux at haifux.org
> > http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
> Haifux mailing list
> Haifux at haifux.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Haifux