Nach positiver Absolvierung der Lehrveranstaltung sind Studierende in der Lage unter Verwendung des PyTorch-Frameworks eigene neuronale Netzwerkmodelle für die Verarbeitung natürlicher Sprache zu entwerfen, zu implementieren und zu verstehen.

Welcome to this course on deep learning for natural language processing (NLP)! This course is designed to unravel the complexities of NLP through the lens of deep learning techniques. From fundamental concepts to advanced neural network architectures, we will explore the intricacies of how deep learning models can comprehend, generate, and manipulate human language. Get ready to embark on a transformative learning journey, where theory meets hands-on applications.

The topics covered in the course include the following: 

• word vectors, word window classification, language models
• backpropagation and neural networks, dependency parsing
• PyTorch
• recurrent neural networks and language models
• seq2seq, machine translation, subword models
• self-attention and transformers
• pretraining, natural language generation
• Hugging Face transformers
• prompting, reinforcement learning from human feedback
• question answering 
• convolutional neural networks, tree recursive neural networks and constituency parsing
• insights between NLP and linguistics
• code generation
• training large language models
• multimodal deep learning
• co-reference resolution 
• interpretability and explainability

Lecture part with frontal lectures from April to June (12 weeks, 2 teaching hours per week).

Practicals with solving exercises and working in a team project are from April to June (12 weeks, 2 teaching hours per week):

• In week 1 to 4, you will solve exercise problems.
• In week 5 to 6, you will review TensorFlow and PyTorch tutorials during the practicals with the help of the demonstrators.
• In weeks 7 to 8, you are expected to organize in groups, and your group is expected to select a paper from the list of papers in the assessment document (which will be available on the course resources section), and prepare a work plan with a timeline, to be submitted by 6pm Friday of week 10 by email to Thomas Lukasiewicz. Each group should submit an initial plan stating your assigned group number and team members, the person in charge of submitting the presentation, the title of the paper selected, and a brief plan of action (equivalent to roughly half a side of A4).
• In weeks 9 to 11, each group is expected to reproduce the results in the selected paper, extending on the methodology and experiments.
• In week 12, each group presents their work in a presentation (with slides), and the members of each group are orally examined.

Exercises have to be solved by each student individually.

ECTS Breakdown:

18h Lectures in the lecture hall
18h Exercises in the lecture hall
18h Preparation at home
18h Solving exercise problems
57h project
20h Preparation for final oral exam
  1h Presentation and final oral exam
-------------------------------------------------------------------------------
150h = 6 ECTS Total workload

Solving exercise problems, working in a team project, and presentation and oral exam at the end of the course.   

Your final grade results from your solutions to the given exercise problems, from your submitted work plan, the type of extensions made on the selected paper, the submitted slides of the presentation, the presentation itself, and the final oral exam.

Die endgültige Anmeldung erfolgt während der ersten Vorlesung: Die Plätze werden unter den Anwesenden in der Reihenfolge der Erstanmeldung in TISS vergeben.

Basic Python knowledge will be an advantage.