~ Announcements ~
Major announcements will be made on both this page and your registered email on Blackboard. Please make sure you can receive emails from Blackboard.
IE DIC Cluster [posted on Sept 23]
We have set up the IE DIC (Data-Intensive Cluster) account for you. For students who cannot setup the single node Hadoop cluster in HW#0, please contact the TAs. You can either choose to set up a single node hadoop cluster with TAs’ help or you can use the IE DIC Cluster account to run MapReduce programs. Hadoop is well installed in the IE DIC Cluster and you can login the cluster to submit jobs via the following command:
ssh s[your student_id]@dicvmc4.ie.cuhk.edu.hk
where student_id is your student ID number. You can find the password on the grading board of the elearning system. Note that this machine can only be accessed within IE network, you can follow the instruction document, which is placed in the elearning system under the course contents directory, to setup IE VPN using your IE account. For those who are from other departments and would like to use the DIC cluster, please contact TAs to get a temporary IE account.
The overview of the DIC cluster is provided below.
Cluster Overview:
- 10 nodes,
- Memory: 100 GB * 10 = 1 TB
- Virtual CPU Cores: 24 * 10 = 240 cores
- Disk: 1 TB * 10 = 10 TB
- Resource management platform: YARN
- Installed applications: MapReduce
Cluster login:
- Login the cluster via:
ssh s[your student_id]@dicvmc4.ie.cuhk.edu.hk - The cluster can only be accessed within the IE network. You can follow the instruction document to set up your IE VPN.
Find the logs of applications:
- Users can find the logs of all applications in the cluster via the web UI: http://dicvmc2.ie.cuhk.edu.hk:19888/
- Users can find the details of a particular application via the web UI: http://dicvmc2.ie.cuhk.edu.hk:19888/jobhistory/job/job_1473396442288_0004 where job_1473396442288_0004 is the ID of the job you created.
- The log information of an application includes
- How many containers are allocated
- The scheduling time and the completion time of each container
- The stderr file which can help you to find bugs of your code.
Need to Schedule the Tutorial Time [posted on Sept 6]
TAs are now scheduling the time of course tutorials. Please login to the Blackboard of E-learning system, check the announcement, and use the dedicated google form link to submit your preferences. The deadline for you to submit is Sept 10 (Friday), 11:59am (Midnoon).
Homework 0 Released [posted on Sept 6]
Homework 0 has already released. It will be due on Sept 19 (Sunday) at 11:59pm. It will take you MUCH MUCH MUCH time, so please take some time to read it and start as soon as possible. Please note that the deadline will NOT BE EXTENDED.