`num_return_sequences` has to be smaller or equal to `num_beams`.

Package:

transformers

50617

Exception Class:

ValueError

Raise code

                eos_token_id=eos_token_id,
                **model_kwargs,
            )
        elif num_beams > 1:
            length_penalty = length_penalty if length_penalty is not None else self.config.length_penalty
            early_stopping = early_stopping if early_stopping is not None else self.config.early_stopping
            if num_return_sequences > num_beams:
                raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
            beam_scorer = BeamSearchScorer(
                batch_size=batch_size,
                num_beams=num_beams,
                device=self.device,
                length_penalty=length_penalty,
                do_early_stopping=early_stopping,
                num_beam_hyps_to_keep=num_return_sequences,

🙏 Scream for help to Ukraine

Today, 3rd July 2022, Russia continues bombing and firing Ukraine. Don't trust Russia, they are bombing us and brazenly lying in same time they are not doing this 😠, civilians and children are dying too! We are screaming and asking exactly you to help us, we want to survive, our families, children, older ones.
Please spread the information, and ask your governemnt to stop Russia by any means. We promise to work extrahard after survival to make the world safer place for all.

Links to the raise (3)

https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/models/rag/modeling_rag.py#L1550 https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/generation_utils.py#L1019 https://github.com/huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/generation_utils.py#L1096

Ways to fix

The generate method of a pretrained T5ForConditionalGeneration model generates sequences for models with a language modeling head. When calling this method on initialized model the parameter num_return_sequences which is used to specify the number of independently computed returned sequences for each element in the batch should be smaller or equal to parameternum_beans.If a value greater than the num_beamsis given This particular error is raised.

How to reproduce the error:

Setup and installation

pip install --user pipenv

mkdir test_folder

cd test_folder

pipenv shell

pipenv install transformers

Run Sample code

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
document = ("at least two people were killed in a suspected bomb attack on a passenger bus "
"in the strife-torn southern philippines on monday , the military said."
)
input_ids = tokenizer(document, return_tensors="pt").input_ids
# generate 3 independent sequences using beam search decoding (5 beams)
# with T5 encoder-decoder model conditioned on short news article.
outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=6)
print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

Output:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-4fdd15f341c2> in <module>()
      8 # generate 3 independent sequences using beam search decoding (5 beams)
      9 # with T5 encoder-decoder model conditioned on short news article.
---> 10 outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=6)
     11 print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---> 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30 

/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, encoder_no_repeat_ngram_size, num_return_sequences, max_time, max_new_tokens, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, forced_bos_token_id, forced_eos_token_id, remove_invalid_values, synced_gpus, **model_kwargs)
   1034 
   1035             if num_return_sequences > num_beams:
-> 1036                 raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
   1037 
   1038             if stopping_criteria.max_length is None:

ValueError: `num_return_sequences` has to be smaller or equal to `num_beams`.

How to fix the error:

Make sure `num_return_sequences` is smaller or equal to `n``um_beams when calling the generate method.`

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
document = ("at least two people were killed in a suspected bomb attack on a passenger bus "
"in the strife-torn southern philippines on monday , the military said."
)
input_ids = tokenizer(document, return_tensors="pt").input_ids
# generate 3 independent sequences using beam search decoding (5 beams)
# with T5 encoder-decoder model conditioned on short news article.
outputs = model.generate(input_ids=input_ids, num_beams=5,num_return_sequences=3)
print("\n\n Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

Correct Output:

/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)


 Generated: ['at least two people were killed in a suspected bomb attack on a passenger bus in the', 'at least two people were killed in a suspected bomb attack on a passenger bus.', 'at least two people were killed in a suspected bomb attack on a passenger bus on mon']