Seminar Deep Learning (Master) [SS23]
Titel: Natural Language Processing
Seminar Facilitator: Hermann Höhne, Marco Pawlowski, Hendrik Annuth
Description: After computer vision, natural language processing (NLP) represents the next revolution in deep learning. The topic can be seen from different perspectives. This includes the technical aspects and the functionality of those models, the abilities and limits of those models, and application cases. In this seminar, we will shed light on those different areas.
Language: The scientific literature and the scientific field of data science is vastly represented in the English language. Therefore, the written version of the presentation and the presentation material must be written in English language. However, the presentation itself can be done in German language, even so, English is preferred. The English skill set of participants will not influence the evaluation of the lecture. Participants are expected to prepare their presentations, thus, if the presentation content is hard to understand, due to ill-prepared presenters, this will influence the evaluation of the lecture for obvious reasons, however.
Expectations: Every participant is expected to do a 45min presentation followed by a discussion on their topic and create a 10-page written version of their lecture. All presentations in the seminar are held under compulsory attendance (Anwesenheitspflicht). Participants are expected to communicate with one another to avoid overlapping topics and discuss who will present which topic. The submission date for the written submissions one week before your presentation. The document needs to adhere with the PDF file standard. The written submission is a preparation for the master’s thesis and is thus expected to aim for scientific standards. Texts should be easy to understand and properly illustrated. Participants are expected to name important sources and authors, and quote publications. Participants are expected to use source material that meets scientific standards, such as scientific literature and other forms of scientific publications. The written submission cannot exceed 10-page length. Any type and form of page is counted, including pages such as flyleafs or tables of content . The evaluation of participants' lectures is not only based on their expressed knowledgeability in their given field, but also on their didactical preparation of their presentation and written material. Thus, participants are expected to have their explanations of content, representations, and illustrations ready and practiced. The focus of the evaluation is on the presentation. It is important to remember that the written version and the oral version may differ in terms of content. This is particularly useful if you deepen topics in the presentation to make them easier to understand and when use additional examples.
Schedule: The seminar has 5 ECTS points and thus an expected workload of 150 working hours. To honor each individual work, there will be a maximum of two presentations a day—possibly three, given special circumstances—leading to 6 expected appointments. Thus, the appointments for the presentations might already start on the first day of the semester! Participants have to be aware of this, especially when choosing one of the first topics.
Feedback: The seminar is a preparation for the master’s thesis. Thus, we would like to give you feedback on both your presentation and your written work. We believe that the learning effect is maximized, when everyone participates in this process. If this, for whatever reason, is inacceptable to you, please inform us before your presentation.
Available Topics:
1. Introduction of NLP inside of Deep Learning
Vision is a very prominent subject in the field of deep
learning. This presentation should present the general approach to
NLP and the differences to other prominent Deep Learning subjects.
2. Development of Language Translation
Translation is one major aspect of processing language. The
available approaches, difficulties and current/future development
of language translation should be analyzed and presented.
3. Development of Transfer Learning in NLP
With around 175 billion parameters GPT-3 is one of the biggest
language model ever created. It is not feasible to learn a model
from scratch for every new problem. Thus, transfer learning is
a major topic for NLP.
4. NLP Systems in Customer Service Applications
Chatbots, recommendation systems and other customer services are
fields where AI and NLP got a lot of attention.
This leads to special use cases and additional barriers for NLP systems.
5. Plagiarism Detection using NLP
Detecting plagiarism includes besides similarity of text also the
similarity of the meaning. How can new developments in NLP help in
this subject?
6. Biases in NLP Systems
Biases in Data is a general problem in training of neural
network. Wrong labeled data and imbalances can lead to wrong
predictions.
7. Adversarial Attacks on Natural Language Processing Systems
If the activation criteria of a neural network are known, data can be
manipulated in such a way that it comes to wrong
classifications. Impressive examples in the field of vision are the one-pixel
attack.
8. Data Extraction from Natural Language Processing Systems
Data is used to train a neural network. Depending on the application, this
data may contain sensitive information. Additionally, the question
arises how much of the information is contained in the weights of
the networks and can it be extracted.
9. Scaling Language Models
The number of neural network weights has
increased enormously in recent years. Can neural networks scale
arbitrarily? Where are the limits. What is the argument for using
ever larger networks, and what is the argument that this does not
work.
10. Pathways Language Model (PaLM)
"Too often, machine learning systems
overspecialize at individual tasks, when they could excel at
many. That’s why we’re building Pathways—a new AI architecture
that will handle many tasks at once, learn new tasks quickly and
reflect a better understanding of the world."
(https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/)
11. Meta Learning for NLP
An attempt to solve the limitations of
neural networks are the so-called meta learning. This involves
deriving rules or knowledge from data in order to be able to use
these rules to solve new problems. Or more strikingly expressed:
"Learning to learn".
12. NLP vs NLU
Processing language is not (always) language understanding.
Distribution of Topics:
On Monday 9.01.2023 at 12:30.
Virtual Meeting Room:
https://fh-wedel.zoom.us/j/81935321320?pwd=OHJIOE1nZEoxMjJZelZWREdJUFFRQT09
Seminar Facilitator: Hermann Höhne, Marco Pawlowski, Hendrik Annuth
Description: After computer vision, natural language processing (NLP) represents the next revolution in deep learning. The topic can be seen from different perspectives. This includes the technical aspects and the functionality of those models, the abilities and limits of those models, and application cases. In this seminar, we will shed light on those different areas.
Language: The scientific literature and the scientific field of data science is vastly represented in the English language. Therefore, the written version of the presentation and the presentation material must be written in English language. However, the presentation itself can be done in German language, even so, English is preferred. The English skill set of participants will not influence the evaluation of the lecture. Participants are expected to prepare their presentations, thus, if the presentation content is hard to understand, due to ill-prepared presenters, this will influence the evaluation of the lecture for obvious reasons, however.
Expectations: Every participant is expected to do a 45min presentation followed by a discussion on their topic and create a 10-page written version of their lecture. All presentations in the seminar are held under compulsory attendance (Anwesenheitspflicht). Participants are expected to communicate with one another to avoid overlapping topics and discuss who will present which topic. The submission date for the written submissions one week before your presentation. The document needs to adhere with the PDF file standard. The written submission is a preparation for the master’s thesis and is thus expected to aim for scientific standards. Texts should be easy to understand and properly illustrated. Participants are expected to name important sources and authors, and quote publications. Participants are expected to use source material that meets scientific standards, such as scientific literature and other forms of scientific publications. The written submission cannot exceed 10-page length. Any type and form of page is counted, including pages such as flyleafs or tables of content . The evaluation of participants' lectures is not only based on their expressed knowledgeability in their given field, but also on their didactical preparation of their presentation and written material. Thus, participants are expected to have their explanations of content, representations, and illustrations ready and practiced. The focus of the evaluation is on the presentation. It is important to remember that the written version and the oral version may differ in terms of content. This is particularly useful if you deepen topics in the presentation to make them easier to understand and when use additional examples.
Schedule: The seminar has 5 ECTS points and thus an expected workload of 150 working hours. To honor each individual work, there will be a maximum of two presentations a day—possibly three, given special circumstances—leading to 6 expected appointments. Thus, the appointments for the presentations might already start on the first day of the semester! Participants have to be aware of this, especially when choosing one of the first topics.
Feedback: The seminar is a preparation for the master’s thesis. Thus, we would like to give you feedback on both your presentation and your written work. We believe that the learning effect is maximized, when everyone participates in this process. If this, for whatever reason, is inacceptable to you, please inform us before your presentation.
Available Topics:
1. Introduction of NLP inside of Deep Learning
Vision is a very prominent subject in the field of deep
learning. This presentation should present the general approach to
NLP and the differences to other prominent Deep Learning subjects.
2. Development of Language Translation
Translation is one major aspect of processing language. The
available approaches, difficulties and current/future development
of language translation should be analyzed and presented.
3. Development of Transfer Learning in NLP
With around 175 billion parameters GPT-3 is one of the biggest
language model ever created. It is not feasible to learn a model
from scratch for every new problem. Thus, transfer learning is
a major topic for NLP.
4. NLP Systems in Customer Service Applications
Chatbots, recommendation systems and other customer services are
fields where AI and NLP got a lot of attention.
This leads to special use cases and additional barriers for NLP systems.
5. Plagiarism Detection using NLP
Detecting plagiarism includes besides similarity of text also the
similarity of the meaning. How can new developments in NLP help in
this subject?
Biases in Data is a general problem in training of neural
network. Wrong labeled data and imbalances can lead to wrong
predictions.
If the activation criteria of a neural network are known, data can be
manipulated in such a way that it comes to wrong
classifications. Impressive examples in the field of vision are the one-pixel
attack.
Data is used to train a neural network. Depending on the application, this
data may contain sensitive information. Additionally, the question
arises how much of the information is contained in the weights of
the networks and can it be extracted.
The number of neural network weights has
increased enormously in recent years. Can neural networks scale
arbitrarily? Where are the limits. What is the argument for using
ever larger networks, and what is the argument that this does not
work.
"Too often, machine learning systems
overspecialize at individual tasks, when they could excel at
many. That’s why we’re building Pathways—a new AI architecture
that will handle many tasks at once, learn new tasks quickly and
reflect a better understanding of the world."
(https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/)
11. Meta Learning for NLP
An attempt to solve the limitations of
neural networks are the so-called meta learning. This involves
deriving rules or knowledge from data in order to be able to use
these rules to solve new problems. Or more strikingly expressed:
"Learning to learn".
12. NLP vs NLU
Processing language is not (always) language understanding.
Distribution of Topics:
On Monday 9.01.2023 at 12:30.
Virtual Meeting Room:
https://fh-wedel.zoom.us/j/81935321320?pwd=OHJIOE1nZEoxMjJZelZWREdJUFFRQT09