Tentative Schedule

This is a tentative schedule. It will be updated according to the actual progress.

Week/Date

Description

Course Material

Remark
Week 1
08/01/2025 (Wed)
09/01/2025 (Thu)

Overview of Foundation Models in AI; Pre-Transformer Era of Deep Learning models
[slides]

Readings: [LLM] [ORFM] [SparksAGI] [LLMintro] [Word2Vec] [Word2Vec-Note] [Word2Vec-Explained] [seq2seqNMT] [BERT] [illustrated-BERT] [NeurProb] [NeurNetwork] [ScalingNLM] [ScalingInterpre]
Week 2/ Week 3
15/01/2025 (Wed)
16/01/2025 (Thu)
22/01/2025 (Wed)
23/01/2025 (Thu)

Transformer: Basic Architecture
[slides]

Readings: [VisualGPT] [VisualAtt] [VisualMLP] [VisualTrans] [buildGPT] [Attention] [illustrated-Transformer] ch10 of [SLP] [WhatTransformer] [RotaryPosiEmbed] [LLaMa] [LLaMa2]
29/01/2025 (Wed)
30/01/2025 (Thu)

Chinese New Year Holiday
Week 4
05/02/2025 (Wed)
06/02/2025 (Thu)

Efficient Transformer Architectures
[slides]

Readings: [Sys4FM] [MoD] [SwitchTransformers] [Mistral7B] [MixtralExperts] [DemystifyingMoE] [DeepSeekMoE] [LongContextExtension] [FlashAttention]
Week 5
12/02/2025 (Wed)
13/02/2025 (Thu)

Beyond Transformers: Alternative Architectures for LLMs
[slides]

Readings: [SSM@SimonInstitute] [SSM@icml24] [Post-TransformerArch] [Mamba] [introRWKV&WorldToken] [Eagle7B&RWKVv5] [RWKVv7]
12/02/2025 23:59PM
Wednesday

Homework 1 is due!

[Homework 1]

Due
15/02/2025 23:59PM
Saturday

Reading Assignment 1 is due!

[Reading Assignment 1]

Due
Week 6
19/02/2025 (Wed)
20/02/2025 (Thu)

Distributed/ Parallel Training of Foundation Models
[slides]

Readings: [Megatron] [DeepSpeedZeRO] [ZeRO] Sec2.5 of [OPT]
22/02/2025 23:59PM
Saturday

Reading Assignment 2 is due!

[Reading Assignment 2]

Due
26/02/2025 (Wed)
27/02/2025 (Thu)

The instructor will be on conference leave - No Lectures. The make-up lecture will be on 23 Apr.

No lectures
01/03/2025 23:59PM
Saturday

Reading Assignment 3 is due!

[Reading Assignment 3]

Due
Week 7
05/03/2025 (Wed)
06/03/2025 (Thu)

Pre-training of Foundation Models and Data Preparation
[slides]

Readings: [Data] [DataCentric] [C4] [PretrainCS224n]
08/03/2025 23:59PM
Saturday

Reading Assignment 4 is due!

[Reading Assignment 4]

Due
Week 8
12/03/2025 (Wed)
13/03/2025 (Thu)

Post-training/ Adaptation of Foundation Models;
[slides]

Readings: [UnderstandInContext] [FewShot] [ZeroShot] [UnsuperMulti] [HumanFeedback] [PromptEngg] [EmergentAbi] [IntrepreTrans] [LoRA] [MEND]
15/03/2025 23:59PM
Saturday

Homework 2 is due!

[Homework 2]

Due
Week 9
19/03/2025 (Wed)
20/03/2025 (Thu)

Augmentation for Foundation models

Readings: [RAG] [GraphRAG] [GraphFoundaModel]
Week 10
26/03/2025 (Wed)
27/03/2025 (Thu)

Teach LLMs to Reason

Readings: [CoT] [CoTw/oPrompt] [LLMSelfCorrect] [PremiseOrder] [CoTSerialProb] [SuperAlign] [LearnReason] [MisguidAtt] [MemoReason]
Week 11
02/04/2025 (Wed)
03/04/2025 (Thu)

LLM Agents and Agentic Frameworks

Readings: [ReAct] [CompoundAI] [FutureCompoundAI] [GoogleAgent] [MicrosoftAutoGen] [Smolagents]
Week 12
09/04/2025 (Wed)
10/04/2025 (Thu)

Applications of Foundation Models

Readings: [ProgSynthesis] [AlphaCode] [Codex] [ImageTrans] [TransferableVisual] [Text2Image] [TransferLimit] [Flamingo] [DecisionTrans] [GR00T] [Voyager] [Eureka] [DrEureka]
Week 13
16/04/2025 (Wed)
17/04/2025 (Thu)

Paradigm Shift in the Development and Adoption of AI Applications/Systems due to Foundation Models

Readings: [DSPy] [LangChain] [OpenAPIPrompt]
Week 14
23/04/2025 (Wed)

AI Safety and Security

Readings: [MLSys79] [ConstitutionalAI] [AgentSafety]

Make-up lecture