~ Home ~
This course aims to provide students an understanding in the operating principles and hands-on experience with mainstream Big Data Computing systems. Open-source platforms for Big Data processing and analytics would be discussed. Topics to be covered include:
Programming models and design patterns for mainstream Big Data computational frameworks ;
System Architecture and Resource Management for Data-center-scale Computing ;
System Architecture and Programming Interface of Distributed Big Data stores ;
High-level Big Data Query languages and their processing systems ;
Operational and Programming tools for different stages of the Big Data processing pipeline including data collection/ ingestion, serialization and migration, workflow coordination.
This course contains substantial hands-on components which require solid background in programming and hands-on operating systems experience. ENGG4030 is an official pre-requisite.
Lecture time and venue:
- LSB LT3 ; TUE 9:30am - 11:15am
- ERB 407 ; FRI 9:30am - 11:15am
Lecture time and venue(ESTR4316):
- ERB 1009 ; FRI 4:30pm - 5:30pm
- Prof. Wing Cheong Lau.
wclau [at] ie [dot] cuhk [dot] edu [dot] hk
- Office hours: Tue 4:00pm - 5:00pm or by appointment
- Venue LSB_G35; Tue 11:30am - 12:15pm
- Ronghai Yang
yr013 [at] ie [dot] cuhk [dot] edu [dot] hk
- Office hour: TBD (SHB 802)
- Liu Yang
ly016 [at] ie [dot] cuhk [dot] edu [dot] hk
- Office hour: TBD (SHB 802)
User: ierg4330 Password: spring4330ierg
[HadoopAppArch] Hadoop Application Architectures 1st Edition, by Mark Grover, Ted Malaska, Jonathan Seidman and Gwen Shapira, Publisher: O’Reilly Media, July 2015.
[Hadoop] Hadoop: The Definitive Guide 4th Edition, by Tom White, published by Oreilly, April 2015.
[JLin] Data-Intensive Text Processing with MapReduce by Jimmy Lin and Chris Dyer, Morgan and Claypool Publishers, 2010, can be freely downloaded from http://lintool.github.io/MapReduceAlgorithms/
[DataIntensive] Designing Data-Intensive Applications: The Big Ideas behind Reliable, Scalable and Maintainable Systems, Preview Edition, by Martin Kleppmann, Publisher: O'Reilly Media, 1st Edition to be published in 2016.
[StormApplied] Storm Applied, by Sean T. Allen, Matthew Jankowski and Peter Pathirana, Publisher: Manning, 2015
[BigData] Big Data: Principles and Best Practices of Scalable Realtime Data Systems, by Nathan Marz and James Warren, Publisher: Manning, 2015
[NoSQL] NoSQL Overview, Appendix A of the book titled "Graph Databases", by Ian Robinson, Jim Webber and Emil Eifrem (Can request a free copy from http://graphdatabases.com)
[MMDS] Mining of Massive Datasets (Download version 1.3) by Anand Rajaraman, Jeff Ullman and Jure Leskovec, Cambridge University Press. Latest version can be downloaded from http://i.stanford.edu/~ullman/mmds.html#latest
[LearnSpark] Learning Spark: Lightning-Fast Big Data Analysis, 1st Edition, by Karau, Konwinski, Wendell and Zaharia, published by Oreilly, 2015
[Spark 1] Apache® Spark™ Analytics Made Simple, http://go.databricks.com/apache-spark-analytics-made-simple-databricks
[Spark 2] Mastering Advanced Analytics with Apache Spark®,http://go.databricks.com/mastering-advanced-analytics-apache-spark
[Spark 3] Lessons for Large-Scale Machine Learning Deployments on Apache Spark,http://go.databricks.com/large-scale-machine-learning-deployments-spark-databricks
[Spark 4] Mastering Apache Spark 2.0, http://go.databricks.com/mastering-apache-spark-2.0
Recommended List of Research Papers for Reading
[MapReduce] Jeffrey Dean and Sanjay Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” OSDI 2004.
[GoogleFileSystem] Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung, “The Google File System,” ACM SOSP 2003.
[MapReduceFamilySurvey2013] Sakr S, Liu A, Fayoumi A G. The family of mapreduce and large-scale data processing systems[J]. ACM Computing Surveys (CSUR), 2013.
[YARN] V.K. Vavilapalli, A.C.Murthy, “Apache Hadoop YARN: Yet Another Resource Negotiator,” ACM Symposium on Cloud Computing (SoCC) 2013.
[Mesos] B. Hindman et al, “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center”, NSDI 2011.
[DRF] A. Ghodsi et al, “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types,” NSDI 2011.
[Borg] A. Verma, L. Pedrosa, “Large-scale cluster management at Google with Borg”, Eurosys 2015
[Omega] M. Schwarzkopf, A. Konwinski, M.Abd-El-Malek, J. Wilkes, “Omega: flexible, scalable schedulers for large compute clusters,” Eurosys 2013
[Sparrow] K. Ousterhout et al, “Sparrow: Distributed, Low Latency Scheduling”, ACM SOSP 2013
[Apollo] E. Boutin et al, “Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing”, OSDI 2014
[Mercury] K. Karanasos et al, “Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters”, Usenix ATC 2015
[GraphLab1] Yucheng Low, Joseph Gonzalez et al, “GraphLab: A New Framework for Parallel Machine Learning,” UAI 2010.
[PowerGraph] Joseph Gonzalez et al, “PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs,” OSDI 2012.
[GraphChi] Aapo Kyrola, Guy Blelloch, Carlos Guestrin, “GraphChi: Large-Scale Graph Computation on Just a PC,” OSDI 2012.
[Storm@Twitter] Ankit Toshniwal et al, “Storm@Twitter,” ACM SIGMOD 2014.
[PigLatin] Christopher Olston et al, “Pig Latin: A Not-So-Foreign Language for Data Processing,” ACM SIGMOD 2008.
[Hive1] Ashish Thusoo et al, “Hive: a warehousing solution over a map-reduce framework,” VLDB 2009.
[Hive2] Ashish Thusoo et al, "Data warehousing and analytics infrastructure at facebook,” ACM SIGMOD 2010
[Hive3] Ashish Thusoo et al, “Hive - A Petabyte Scale Data Warehouse Using Hadoop,” IEEE ICDE 2010.
[HiveAdvances] Yin Huai et al, “Major Technical Advancements in Apache Hive,” ACM SIGMOD 2014.
[Dryad] Michael Isard et al, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks,” Eurosys 2007.
[DryadLINQ] Yuan Yu et al, “DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language, “ OSDI 2008.
[DryadLINQ2] Michael Isard, Yuan Yu, "Distributed Data-Parallel Computing Using a High-Level Programming Language,” ACM SIGMOD 2009
[Tez] Bikas Saha et al, "Apache Tez: A Unifying Framework for Modeling and Building Data Processing Applications,” ACM SIGMOD 2015.
[Dynamo] Giuseppe DeCandia et al, "Dynamo: Amazon's Highly Available Key-value Store," ACM SOSP 2007.
[BigTable] Fay Chang et al, “Bigtable: A Distributed Storage System for Structured Data,” OSDI 2006.
[Cassandra] Avinash Lakshman, Prashant Malik, “Cassandra - A Decentralized Structured Storage System,” ACM SIGOPS Operating Systems Review, Apr 2010.
[RealtimeHadoopFacebook] Dhruba Borthakur et al, “Apache Hadoop goes realtime at Facebook,” ACM SIGMOD 2011.
[SparkRDD] Matei Zaharia et al, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,” NSDI 2012.
[Spark] Matei Zaharia et al, “Fast and Interactive Analytics over Hadoop Data with Spark,” Usenix ;login Aug 2012.
[Spark Streaming] Matei Zaharia et al, "Discretized streams: Fault-tolerant streaming computation at scale,” ACM SOSP 2013.
[SharkSQL] Reynold S. Xin et al, "Shark: SQL and rich analytics at scale,” ACM SIGMOD 2013
[SparkSQL] Michael Armbrust et al, “Spark SQL: Relational Data Processing in Spark,” ACM SIGMOD 2015.
[GraphX] Joseph E. Gonzalez et al, "GraphX: Graph Processing in a Distributed Dataflow Framework,” OSDI 2014.
[SparkScaling] Michael Armburst et al, “Scaling Spark in the real world: performance and usability,” VLDB 2015.
[MapReduceVsSpark] Juwei Shi et al, "Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics," VLDB 2015.
[SparkMLbase] T. Kraska, A. Talwalkar, J.Duchi, R. Griffith, M. Franklin, M.I. Jordan, "MLbase: A Distributed Machine Learning System," In Conference on Innovative Data Systems Research (CIDR), 2013.
[SparkMLI] E. Sparks, A. Talwalkar, V. Smith, J. Kottalam, X. Pan, J. Gonzalez, J. Gonzalez, M. Franklin, M. I. Jordan, T. Kraska. MLI: An API for Distributed Machine Learning. In International Conference on Data Mining, 2013.
[SparkMLlib] Xiangrui Meng et al, "MLlib: Machine learning in Apache Spark,” arXiv:1505.06807, May 2015.
[SparkNet] Philipp Moritz, Robert Nishihara, Ion Stoica, Michael Jordan, "SparkNet: Training Deep Networks on Spark,” ICLR 2016.
[Naiad] Derek G. Murray et al, "Naiad: A Timely Dataflow System,” ACM SOSP 2013.
[ZooKeeper1] P Hunt, M Konar, FP Junqueira, B Reed , “ZooKeeper: Wait-free Coordination for Internet-scale Systems,” Usenix ATC 2010.
[ZAB1] Benjamin Reed, Flavio P. Junqueira, “A simple totally ordered broadcast protocol,” 2nd Workshop on Large-scale Distributed Systems and Middleware (LADIS), 2008.
[ZAB2] F.P. Junqueira, B.C. Reed, M. Serafini, "High-performance broadcast for primary-backup systems,” IEEE/IFIP DSN, 2011.
Recommended Programming References
[SparkAnalytics] Advanced Analytics with Spark, by Sandy Ryza, Uri Laserson, Sean Owen and Josh Wills, Publisher: O’Reilly Media, April 2015
[DataAlgorithms] Data Algorithms: Recipes for Scaling Up with Hadoop and Spark, by Mahmoud Parsian, Publisher: O'Reilly Media, Aug 2015
[LearnSpark] Learning Spark, by Holden Karau, Andy Konwinski, Patrick Wendell and Matei Zaharia, Publisher: O’Reilly Media, Feb 2015
[HBase] HBase: The Definitive Guide, by Lars George, published by O’Reilly Media,.
[CassandraBook] Cassandra: The Definitive Guide, by Eben Hewitt, published by O’Reilly Media,.
[ZooKeeper] ZooKeeper: Distributed Process Coordination, by Flavio Junqueira and Benjamin Reed, published by O’Reilly Media, 2013
[Pig] Programming Pig, by Alan Gates, published by O’Reilly Media
[Hive] Programming Hive, by Edward Capriolo, Dean Wampler, Jason Rutherglen, published by O’Reilly Media,
[OpenStackOp] OpenStack Operations Guide, published by O’Reilly Media, (current-version available online at: http://docs.openstack.org/openstack-ops/content )
[OozieBook] Apache Oozie : The Workflow Scheduler for Hadoop, by Mohammad Kamrul Islam, Aravind Srinivasan published by O'Reilly Media, 2015.
|Lecture Date||Topic||Period||Recommended Readings||Additional References|
|Jan 10, 13, 17, 20||Course Admin; Resource Management for Data-center-scale Computing: Hadoop YARN, Mesos and beyond ;||9:30am - 11:15am||[YARN], [Mesos], [Hadoop]Ch.2-3, [CloudData] ;||[Borg], [Omega], [Sparrow], [Apollo], [Mercury], [MapReduceFamilySurvey2013]|
|Jan 24, Feb 7, 10||Programming Models (beyond MapReduce) for Big Data Computing I: Graph-based Computing frameworks: Pregel/Giraph and GraphLab||9:30am - 11:15am||[GraphLab1], [PowerGraph]||[GraphChi]|
|**Jan 27 - Feb 2 Chinese New Year Holidays**|
|Feb 10, 14, 17||Programming Models (beyond MapReduce) for Big Data Computing II: Stream-based Processing: Storm; ZooKeeper||9:30am - 11:15am||[Storm@Twitter], [StormApplied], [ZooKeeper1]||[BigData], [ZooKeeper], [ZAB1], [ZAB2]|
|Feb 21, 24||High-level Data Query Languages for Big Data Analytics: Pig and Hive||9:30am - 11:15am||[PigLatin], [Hive1], [Hadoop]Ch.16-17||[Hive2], [Hive3], [HiveAdvances], [Pig], [Hive]|
|**No Class on Feb 28**|
|**Mar 3 (Fri) 2-hr Mid-term Examination**|
|Mar 3, 7||Programming Models (beyond MapReduce) for Big Data Computing III: DAG-based Computational Frameworks: Dryad, DryadLINQ, TeZ||9:30am - 11:15am||[Dryad], [DryLINQ], [Tez]||-||Mar 7, 10, 14, 17||Distributed Big Data Stores: BigTable/ HBase, Dynamo, Cassandra||9:30am - 11:15am||[Dynamo], [BigTable], [Cassandra], [RealtimeHadoopFacebook], [NoSQL], [Hadoop]Ch.20||[HBase], [CassandraBook]|
|Mar 21, 24, 28, 31||Programming Models (beyond MapReduce) for Big Data Computing IV: Spark: Spark and BDAS, Quick Tour of Scala, Spark RDDs||9:30am - 11:15am||[SparkRDD], [Spark], [SparkAnalytics]Ch.2||[SparkScaling], [MapReduceVsSpark], [LearnSpark]Ch.1, Ch.10 ; [SparkAnalytics] Appendix A|
|**April 4 Qing Ming Festival**|
|Apr 7, 11,||Programming Models (beyond MapReduce) for Big Data Computing IV: Spark (cont'd): SparkSQL, Spark Streaming, GraphX, MLlib||9:30am - 11:15am||[GraphX], [SparkStreaming], [SparkSQL], [SparkMLlib], [LearnSpark]Ch.11 ;||[SharkSQL], [SparkMBase], [SparkMLI]|
|**April 14 - April 17 Easter Day**|
|Apr 18, 21||The Overall Big Data Processing Pipeline: Tools for Data Ingestion, Data Serialization and Workflow Coordination||9:30am - 11:15am||[HadoopAppArch]||[SparkAnalytics], [OozieSWEET12], [OozieHUG13], [OozieAWS16], Appendix B|
Your grade will be based on the following components:
- Homework & Programming assignments (4 sets in total): 50%
- Mid-term Exam: 10% (1-hour mid-term examination)
- Final Exam: 40% (2-hour final examination)
Student/Faculty Expectations on Teaching and Learning
You are expected to do your own work and acknowledge the use of anyone else's words or ideas. You MUST put down in your submitted work the names of people with whom you have had discussions.
Refer to http://www.cuhk.edu.hk/policy/academichonesty for details
When scholastic dishonesty is suspected, the matter will be turned over to the University authority for action.
You MUST include the following signed statement in all of your submitted homework, project assignments and examinations. Submission without a signed statement will not be graded.
I declare that the assignment here submitted is original except for source material explicitly acknowledged, and that the same or related material has not been previously submitted for another course. I also acknowledge that I am aware of University policy and regulations on honesty in academic work, and of the disciplinary guidelines and procedures applicable to breaches of such policy and regulations, as contained in the website http://www.cuhk.edu.hk/policy/academichonesty/.
Thanks to Amazon Web Services, Google and Microsoft Azure for providing free computing resource support of this course