<div dir="ltr">Thanks for the explanation Orna :-)<br><br>It would also be interesting to have a lecture on Hadoop just to show another Map-Reduce implementation out on the wild and also to compare Disco to Hadoop.<br>I guess an additional follow up lecture on Hadoop can work but I'm less familiar with it (and all of its components - and it has quite a few) than I am with Disco.<br>
<br>Eran<br><br><div class="gmail_quote">On Wed, Jan 14, 2009 at 11:13 PM, Orna Agmon Ben-Yehuda <span dir="ltr"><<a href="mailto:ladypine@gmail.com">ladypine@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr"><br>
In short, this is the method google use for parallel processing.
Every operation which needs to be parallel is divided into a mapping stage (where each worker does
something on their own data, and produces a result) and a reduction
stage, where the results of the map are collected into a meaningful
result. This parallelization scheme is highly scalable.<br><br>This (upcoming) Sunday's slides on Map-Reduce from the concurrent and Distributed Programming course:<br> <a href="http://webcourse.cs.technion.ac.il/236370/Winter2008-2009/ho/WCFiles/map-reduce-lecture.pdf" target="_blank">http://webcourse.cs.technion.ac.il/236370/Winter2008-2009/ho/WCFiles/map-reduce-lecture.pdf</a><br>
And a canonical paper, linked from the same place:<br><a href="http://labs.google.com/papers/mapreduce-osdi04.pdf" target="_blank">http://labs.google.com/papers/mapreduce-osdi04.pdf</a><br><font color="#888888"><br>Orna.</font><div>
<div></div><div class="Wj3C7c"><br><br><div class="gmail_quote">On Wed, Jan 14, 2009 at 10:27 PM, guy keren <span dir="ltr"><<a href="mailto:choo@actcom.co.il" target="_blank">choo@actcom.co.il</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
can you explain, briefly, what map-reduce is, so those not in the know<br>
(like me) will be able to decide if this is interesting? ;)<br>
<br>
thanks,<br>
<font color="#888888">--guy<br>
</font><div><div></div><div><br>
Eran Sandler wrote:<br>
> Hi all,<br>
><br>
> It's been a while since I've posted to Haifux (or Linux-IL for that<br>
> matter) but I am watching the mailing list from time to time and due to<br>
> personal reasons found myself as a Haifa citizen for the past year (and<br>
> probably for a couple more years :-) ).<br>
><br>
> Somewhere in 2004 I even did a lecture on Mono, the open source .NET<br>
> implementation, if some of you recall.<br>
><br>
> Recently I've been involved with a cool open source project called Disco.<br>
><br>
> Disco is an open source Map-Reduce framework written in Erlang and<br>
> Python. It was written at Nokia's Palo Alto research center as a<br>
> lightweight framework for rapid scripting of distributed data processing<br>
> tasks but grew to become even more than that and is now even used for<br>
> probabilistic modeling, data mining, full text indexing, etc.<br>
><br>
> You can read more about Disco at <a href="http://discoproject.org" target="_blank">http://discoproject.org</a><br>
><br>
> Would a lecture on Map-Reduce in general and specifically Disco would<br>
> interest people?<br>
><br>
> If so, I'm more than willing to give the lecture and show some examples.<br>
><br>
> Eran<br>
><br>
><br>
</div></div>> ------------------------------------------------------------------------<br>
<div><div></div><div>><br>
> _______________________________________________<br>
> Haifux mailing list<br>
> <a href="mailto:Haifux@haifux.org" target="_blank">Haifux@haifux.org</a><br>
> <a href="http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux" target="_blank">http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux</a><br>
<br>
_______________________________________________<br>
Haifux mailing list<br>
<a href="mailto:Haifux@haifux.org" target="_blank">Haifux@haifux.org</a><br>
<a href="http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux" target="_blank">http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux</a><br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>