“初探Hadoop MapReduce NextGen|MRv2|YARN(工作上下文紧急切换, 该篇待续)”
fujohnwang
2011-12-16
人家年初提出来的想法,我年末才刚开始看, 惭愧…
Architecture
- further separation of concerns on JobTracker
- resource management concern
- job scheduling/monitoring concern
- not limited to MapReduce pattern, multiple different parallel programming patterns can be used too, like master-workers, MPI, iterative monads, etc. This is a more generic distributed parallel programming framework.
Components
- ResourceManager
- Scheduler
- ApplicationManager
- NodeManager
- ApplicationMaster
Interesting projects built on hadoop
- http://www.lilyproject.org/lily/index.html - LILY - SMART DATA, AT SCALE, MADE EASY
Hadoop in multiple com
##Hadoop @ 百度 1. 透明压缩 - linux idle class调度 - 异步block级别压缩
2. 资源隔离
- cgroup
References
- http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
- http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/
- http://www.mesosproject.org/
- http://developer.yahoo.com/blogs/hadoop/posts/2011/02/capacity-scheduler/
- http://wiki.apache.org/hadoop/SupportingProjects
「为AI疯狂」星球上,扶墙老师正在和朋友们讨论有趣的AI话题,你要不要⼀起来呀?^-^
这里
- 不但有及时新鲜的AI资讯和深度探讨
- 还分享AI工具、产品方法和商业机会
- 更有体系化精品付费内容等着你,加入星球(https://t.zsxq.com/0dI3ZA0sL) 即可免费领取。(加入之后一定记得看置顶消息呀!)

存量的时代,省钱就是赚钱。
在增量的时代,省钱其实是亏钱。
避坑儿是省钱的一种形式,更是真正聪明人的选择!
弯路虽然也是路,但还是能少走就少走,背后都是高昂的试错成本。
订阅「福报」,少踩坑,少走弯路,多走一步,就是不一样的胜率!
