Optimisation for Deep Learning

This course will be offered in Semester 2 2021 via the AMSI ACE network. 
Please scroll down for historical information about past offerings.

Lecturers

Textbooks

  • Optimisation part: Convex Optimization by Stephen Boyd and Lieven Vandenberghe
  • Deep learning part: Deep learning by Yoshua Bengio, Ian Goodfellow, Aaron Courville

Motivation

A number of problems in machine learning and, in particular, deep learning, can be formulated as optimisation problems and solved using a suitable optimisation method. Most modern software packages on deep learning use a default optimisation method (“black box”). For many applications, these methods are suitable and the users only need to change the number of layers, nodes, change the activation function, etc. In other cases, however, users need to open the “black box”, in order to improve the performance. The main purpose of this course is to provide a good theoretical foundation to optimisation theory, apply it to formulate optimisation problems appearing in deep learning models and provide a range of optimisation methods that can be used to solve these problems. This unit can be seen as a mathematical unit focused on deep learning as
its important potential application.

Preliminary structure

  • Weeks 1: Introduction to optimisation, machine learning and deep learning: general terminology and convention. Overview of optimisation problems appearing in deep learning.
  • Weeks 2-4: Linear optimisation and elements of linear integer optimisation.
  • Weeks 5-7: Convex optimisation.
  • Weeks 8-10: Non-convex optimisation.
  • Weeks 11-12: Guided study and project submission.

Quick Downloads

Detailed description

PythonInstallationWeek1-1Download

Current Offering

Semester 2 2021

Lecture (Zoom): The zoom link will be emailed to you by ACE.

Week-by-week material (password protected)

Past Offerings

Semester 2 2020

Lecture (Zoom): Wed., 8.30AM-10.30PM, starting from the 5th of August. The zoom link has been emailed to you by ACE.

Week-by-week material (password protected)