Zero and Few Shot Learning

Introduction

The field of NLP is getting more and more exciting each day. Until a few years ago, we were not able to fully leverage the vast sources of data available online. With the amazing success of unsupervised learning methods and transfer learning, the NLP community has built models which serve as a knowledge base for multiple NLP tasks.

However, we're still dependent on annotated data for fine-tuning on a downstream task. More often than not, getting labeled data is not handy and is a relatively expensive and time taking exercise. What can we do if we don't have any labeled data or have very less of it? The answer to this problem is zero-shot and few shot learning.

There is no single definition of zero and few shot methods. Rather, one can say that its definition is task dependent.

Zero shot classification means that we train a model on some classes and predict for a new class, which the model has never seen before. Obviously, the class name needs to exist in the list of classes, but there are no training samples for this class.

In the more broader sense, zero shot setting refers to using a model to do something it wasn't trained to do.

Let's assume we train a language model on a large text corpus (or use a pre-trained one like GPT-2). Our task is to predict whether a given article is about sports, entertainment or technology. Normally, we would formulate this as a fine tuning task with many labeled examples, and add a linear layer for classification on top of the language model. But with zero shot, we use the language model directly (without any explicit fine-tuning). We can give the article along with explanatory label names and get a prediction.

Does it still sound confusing? Fret not! and dive into the intuition behind these concepts to make things clearer. Also, the example in the end will ensure a deeper understanding of its usability.

Intuition behind zero and few shot learning

We as humans store a huge amount of information that we learn from every resource, be it books, news, courses, or just experience.

If we are asked to do the following task: "Translate from english to french": How are you? -> ?

From the task's description, it is quite clear to us what is to be done here. We have used our knowledge base to infer what translation means.

Another task can be as follows: "I loved the movie!" -> happy-or-sad

Reading the self explanatory task explanation (happy-or-sad), we understand that it is a classification task. Our knowledge base also helps us understand the sentence and infer that it is happy!

This is exactly how zero shot classification works. We have a pre trained model (eg. a language model) which serves as the knowledge base since it has been trained on a huge amount of text from many websites. For any type of task, we give relevant class descriptors and let the model infer what the task actually is.

Needless to say, the more labeled data we provide, the better the results would be. And many times, zero shot doesn't work very well. If we have a few samples of labeled data but not enough for fine tuning, few shot is the way to go. As used in GPT-3, "Language Models are Few Shot Learners", the authors prove that very large language models can perform competitively on downstream tasks with much lesser labeled data as compared to what smaller models would require.

Few Shot is simply an extension of zero shot, but with a few examples to further train the model.

FlairNLP and Huggingface to the rescue!

Both FlairNLP and Huggingface have zero shot classification pipelines for english (since they use bert as the model). Even though flairNLP uses bert-base-uncased for english as its base model, it works surprisingly well with simple indonesian text. In the below example, I'll walk you through the steps of zero and few shot learning using the TARS model in flairNLP on indonesian text.

The zero-shot classification pipeline implemented by huggingface has some excellent articles and demos. Check out this excellent blog and this live demo on zero shot classification by HuggingFace.

Since I am using Bahasa Indonesia, I will not use the huggingface zero-shot pipeline, but use the GPT-2 model (trained on Bahasa Indonesia by cahya and available on huggingface) and implement zero-shot classification.

Examples

FlairNLP

Use the zero shot classifier (TARS) model to classify whether the sentence is about sports (olahraga) or politics (politik)

from flair.models.text_classification_model import TARSClassifierfrom flair.data import Sentence

# 1. Load our pre-trained TARS model for Englishtars = TARSClassifier.load('tars-base')

# 2. Prepare a test sentencesentence1 = Sentence("Inggris mengalahkan Australia dengan 4 gawang")sentence2 = Sentence("Para menteri kabinet memainkan permainan yang tidak menyenangkan")

# 3. Define some classes that you want to predict using descriptive namesclasses = ["olahraga","politik"]

#4. Predict for these classestars.predict_zero_shot(sentence1, classes, multi_label=False)tars.predict_zero_shot(sentence1, classes, multi_label=False)

# Print sentence with predicted labelsprint(sentence1)print(sentence2)

Output

Sentence: "Inggris mengalahkan Australia dengan 4 gawang"   [− Tokens: 6  − Sentence-Labels: {'label': [olahraga (0.6423)]}]Sentence: "Para menteri kabinet memainkan permainan yang tidak menyenangkan"   [− Tokens: 8  − Sentence-Labels: {'label': [olahraga (0.0951)]}]

The english model works surprisingly well on Bahasa Indonesia! Although it predicts the second example incorrectly, the sentence "The cabinet ministers are playing a nasty game" has been categorized as sports and not politics. Probably because the word playing confusing the model! Let us make it learn with 4 examples (2 from each class) and re-evaluate on the same example using few shot classification.

from flair.data import Corpusfrom flair.datasets import SentenceDataset

# training dataset consisting of four sentences (2 labeled as "olahraga" (sports) and 2 labeled as "politik" (politics)train = SentenceDataset(    [        Sentence('Pemilu akan dimulai bulan depan').add_label('olahraga_atau_politik', 'politik'),        Sentence('Perdana menteri mengumumkan dana bantuan banjir besar-besaran').add_label('olahraga_atau_politik', 'politik'),        Sentence('Olimpiade akan diadakan di Tokyo').add_label('olahraga_atau_politik', 'olahraga'),        Sentence('Asian Games 2018 merupakan ajang multi sport yang diadakan di Jakarta').add_label('olahraga_atau_politik', 'olahraga')    ])

# test dataset consisting of two sentences (1 labeled as "olahraga" and 1 labeled as "politik")test = SentenceDataset(    [        Sentence('Para menteri kabinet memainkan permainan yang tidak menyenangkan').add_label('olahraga_atau_politik', 'politik'),        Sentence('Ayo pergi dan bermain basket').add_label('olahraga_atau_politik', 'olahraga')    ])

# make a corpus with train and test splitcorpus = Corpus(train=train, test=test)

Train the model on the new data

from flair.trainers import ModelTrainer

# 1. load base TARStars = TARSClassifier.load('tars-base')

# 2. make the model aware of the desired set of labels from the new corpustars.add_and_switch_to_new_task("OLAHRAGA_POLITIK", label_dictionary=corpus.make_label_dictionary())

# 3. initialize the text classifier trainer with your corpustrainer = ModelTrainer(tars, corpus)

# 4. train modeltrainer.train(base_path='resources/taggers/olahraga_politik', # path to store the model artifacts              learning_rate=0.02,               mini_batch_size=1,               max_epochs=10,               )

# 5. Load few-shot TARS modeltars = TARSClassifier.load('resources/taggers/olahraga_politik/final-model.pt')

# 6. Prepare a test sentencesentence = Sentence("Para menteri kabinet memainkan permainan yang tidak menyenangkan")

# 7. Predict for olahraga and politiktars.predict(sentence)print(sentence)

Output

Sentence: "Para menteri kabinet memainkan permainan yang tidak menyenangkan"   [− Tokens: 8  − Sentence-Labels: {'label': [politik (0.9968)]}]

Zero-Shot Classification using GPT-2 trained on Bahasa Indonesia

from torch.nn import functional as Ffrom transformers import GPT2Tokenizer, GPT2Model

# 1. Load pertained GPT-2 model and tokenizermodel_name='cahya/gpt2-small-indonesian-522M'tokenizer = GPT2Tokenizer.from_pretrained(model_name)model = GPT2Model.from_pretrained(model_name)

# 2. Prepare a test sentence and labelssentence = 'Para menteri kabinet memainkan permainan yang tidak menyenangkan'labels = ['olahraga', 'politik']

# 3. Since there is no padding token in this tokenizer, add a token. # A separate pad token can be added using the add_special_token# functiontokenizer.pad_token = tokenizer.eos_token

# 4. Concatenate sentence with lablesinputs = tokenizer([sentence]+labels, return_tensors='pt',padding='longest')

input_ids = inputs['input_ids']attention_mask = inputs['attention_mask']output = model(input_ids, attention_mask=attention_mask)[0]sentence_rep = output[:1].mean(dim=1)label_reps = output[1:].mean(dim=1)

# now find the labels with the highest cosine similarities to# the sentencesimilarities = F.cosine_similarity(sentence_rep, label_reps)closest = similarities.argsort(descending=True)for ind in closest:    print(f'label: {labels[ind]} t similarity: {similarities[ind]}')

Output

label: politik   similarity: 0.5492470860481262label: olahraga      similarity: 0.48411038517951965

Conclusion

Zero shot and few shot learning methods are reducing the reliance on annotated data. The GPT-2 and GPT-3 models have shown remarkable results to prove this. However, for low resource languages like Bahasa Indonesia, it is still an area of active research. However, thanks to valuable resources like cahya, we have a GPT-2 model trained specifically for Indonesian. However, since the data for indonesian is not humongous, few shot is preferred over zero-shot learning. Nonetheless, it is absolutely remarkable what we can achieve with only a few examples, and how libraries like flairNLP and huggingface have made life so much easier to implement cutting edge models!

Cheers!

Eram