Feed

Major announcements will be made on both this page and your registered email on Blackboard. Please make sure you can receive emails from Blackboard.

IE DIC Cluster [posted on Sept 25]

We have set up the IE DIC (Data-Intensive Cluster) account for you. For students who cannot setup the single node Hadoop cluster in HW#0, please contact the TAs. You can either choose to set up a single node hadoop cluster with TAs’ help or you can use the IE DIC Cluster account to run MapReduce program. Hadoop has already been installed in IE DIC cluster. Please note that the Hadoop in IE DIC cluster is based on Hortonworks, and it is a little different from the standard Hadoop system. For more information, please refer to https://hortonworks.com/. Hadoop is well installed in the IE DIC Cluster and you can login the cluster to submit jobs via the following command:

ssh student_id@dic14.ie.cuhk.edu.hk

where student_id is your student ID number. You can find the password on the grading board of the elearning system. Note that this machine can only be accessed within IE network, you can follow the link http://mobitec.ie.cuhk.edu.hk/ierg4300Fall2019/homework/vpn_setup.html to setup IE VPN using your IE account. For those who are from other departments and would like to use the DIC cluster, please contact the TAs to get a temporary IE account.

The overview of the DIC cluster is provided below.

Cluster Overview:

• 28 nodes,
• Memory: 47 GB 2 + 30 GB*4 + 63 GB*3 + 16 GB19 = 707 GB
• Virtual CPU Cores: 24*2 + 16*7 + 8*19 = 312 cores
• Disk: 44.2TB
• Resource management platform: YARN
• Installed applications: MapReduce, Hive, Pig, Spark

Cluster login:

• Login the​ clu​ster via​: ssh user_id@dic14.ie.cuhk.edu.hk
• The cluster can only be accessed within the IE network, you can follow the link below to setup IE VPN using your (temporary) IE account: http://mobitec.ie.cuhk.edu.hk/ierg4300Fall2020/homework/vpn_setup.html

Find the logs of applications:

• Users can find the logs of all applications in the cluster via the web UI: http://dicvm1.ie.cuhk.edu.hk:19888/
• Users can find the details of a particular application via the web UI: http://dicvm1.ie.cuhk.edu.hk:19888/jobhistory/job/job_1473396442288_0004​ ​ where ​job_​1473396442288_0004 is the ID of the job you created.
• The log information of an application includes
  • How many containers are allocated
  • The scheduling time and the completion time of each container
  • The stderr file which can help you to find bugs of your code.

IERG4300/ESTR4300 tutorial time and ESTR4300 lecture time determined [posted on Sept 13]

Dear all,

The tutorial (2 sessions per week) is finally scheduled for :

• Wednesday 11:30 - 12:15
• Thursday 19:30 - 20:15

We will start the first tutorial on Sept 16 (Wed 11:30-12:15) using the same Zoom link for our IERG/ESTR4300 lectures. Please be noted and schedule your time accordingly.

For those who want to attend the weekly tutorials but cannot attend any tutorial sessions due to time-clash with other registered courses, please come to TAs' office hours:

• Da Sun Handason Tam: Thursday 21:30 - 22:15
• Siyue Xie: Tuesday 10:00 - 10:45

• Liu Yang: Thursday 15:00 - 15:45 For the Extra-1-hr-per-week lecture for ESTR4300, it will be scheduled for:

• Wednesday 17:30 - 18:15

We will start this extra lecture on Sept 16 (Wed) using the same Zoom link for our IERG/ESTR4300 lectures.

Best regards,

TAs

Important: doodle poll for ESTR4300 meetings [posted on Sept 8]

We are scheduling the meetings for ESTR4300. Please follow the doodle link below and indicate your availability in different time slots and enter your SID as your Name. The deadline of the anonymous doodle poll is 23:00, 11th, Sep (Friday). https://doodle.com/poll/8hnb5p8kwmrzt9pe#calendar

Important: doodle poll for our Tutorial arrangement [posted on Sept 8]

We are scheduling our tutorials. Please follow the doodle link below and indicate your availability in different time slots and enter your SID as your Name. The deadline of the anonymous doodle poll is 23:00, 11th, Sep (Friday). https://doodle.com/poll/sqbqrvrc39i72uvy#calendar