Instructions on IE DIC Cluster

We have set up the IE DIC (Data-Intensive Cluster) account for you. For students who cannot setup the single node Hadoop cluster in HW#0, please contact the TAs. You can either choose to set up a single node hadoop cluster with TAs’ help or you can use the IE DIC Cluster account to run MapReduce programs.

Hadoop is well installed in the IE DIC Cluster and you can login the cluster to submit jobs via the following command:

ssh s[your student_id]@dicvmc4.ie.cuhk.edu.hk

where student_id is your student ID number. You can find the password on the My Grades page of the elearning system.

Note that this machine can only be accessed within IE network. You can follow the instruction document, which is placed in the elearning system under the Course Contents directory, to setup IE VPN using your IE account.

For those who are from other departments and would like to use the DIC cluster, please contact TAs to get a temporary IE account.

Please note that IE DIC can only be used for your homework. Any user that is found to use it for other purposes will be removed from the system immediately and be punished. To better allocate resources, a program/ job will be terminated without notification if it consumes more than 25GB RAM.

The overview of the DIC cluster is provided below.

Cluster Overview:

  • 10 nodes,
  • Memory: 100 GB * 10 = 1 TB
  • Virtual CPU Cores: 24 * 10 = 240 cores
  • Disk: 1 TB * 10 = 10 TB
  • Resource management platform: YARN
  • Installed applications: MapReduce

Cluster login:

  • Login the​ clu​ster via: ssh s[your student_id]@dicvmc4.ie.cuhk.edu.hk
  • The cluster can only be accessed within the IE network. You can follow the instruction document to set up your IE VPN.

Find the logs of applications: