`num_return_sequences` has to be smaller or equal to `num_beams`.

Package:

transformers

50617

Exception Class:

ValueError

Raise code

                eos_token_id=eos_token_id,
                **model_kwargs,
            )
        elif num_beams > 1:
            length_penalty = length_penalty if length_penalty is not None else self.config.length_penalty
            early_stopping = early_stopping if early_stopping is not None else self.config.early_stopping
            if num_return_sequences > num_beams:
                raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
            beam_scorer = BeamSearchScorer(
                batch_size=batch_size,
                num_beams=num_beams,
                device=self.device,
                length_penalty=length_penalty,
                do_early_stopping=early_stopping,
                num_beam_hyps_to_keep=num_return_sequences,

Links to the raise (3)

https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/models/rag/modeling_rag.py#L1550 https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/generation_utils.py#L1019 https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/generation_utils.py#L1096

Ways to fix

The generate method of a pretrained T5ForConditionalGeneration model generates sequences for models with a language modeling head. When calling this method on initialized model the parameter num_return_sequences which is used to specify the number of independently computed returned sequences for each element in the batch should be smaller or equal to parameternum_beans.If a value greater than the num_beamsis given This particular error is raised.

How to reproduce the error:

Setup and installation

pip install --user pipenv

mkdir test_folder

cd test_folder

pipenv shell

pipenv install transformers

Run Sample code

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
document = ("at least two people were killed in a suspected bomb attack on a passenger bus "
"in the strife-torn southern philippines on monday , the military said."
)
input_ids = tokenizer(document, return_tensors="pt").input_ids
# generate 3 independent sequences using beam search decoding (5 beams)
# with T5 encoder-decoder model conditioned on short news article.
outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=6)
print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

Output:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-4fdd15f341c2> in <module>()
      8 # generate 3 independent sequences using beam search decoding (5 beams)
      9 # with T5 encoder-decoder model conditioned on short news article.
---> 10 outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=6)
     11 print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---> 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30 

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, max_new_tokens, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
   1034 
   1035             if num_return_sequences > num_beams:
-> 1036                 raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
   1037 
   1038             if stopping_criteria.max_length is None:

ValueError: `num_return_sequences` has to be smaller or equal to `num_beams`.

How to fix the error:

Make sure `num_return_sequences` is smaller or equal to `n``um_beams when calling the generate method.`

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
document = ("at least two people were killed in a suspected bomb attack on a passenger bus "
"in the strife-torn southern philippines on monday , the military said."
)
input_ids = tokenizer(document, return_tensors="pt").input_ids
# generate 3 independent sequences using beam search decoding (5 beams)
# with T5 encoder-decoder model conditioned on short news article.
outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=3)
print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

Correct Output:

/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)


 Generated: ['at least two people were killed in a suspected bomb attack on a passenger bus in the', 'at least two people were killed in a suspected bomb attack on a passenger bus.', 'at least two people were killed in a suspected bomb attack on a passenger bus on mon']