Trying to create optimizer slot variable under the scope for tf.distribute.Strategy ((param1)), which is different from the scope used for the original variable ((param1)). Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope

Package:

tensorflow

158813

Exception Class:

ValueError

Raise code

  initializer, shape=slot_shape, dtype=var.dtype)
      else:
        initial_value = initializer

      with self._distribution_strategy_scope():
        strategy = distribute_ctx.get_strategy()
        if not strategy.extended.variable_created_in_scope(var):
          raise ValueError(
              "Trying to create optimizer slot variable under the scope for "
              "tf.distribute.Strategy ({}), which is different from the scope "
              "used for the original variable ({}). Make sure the slot "
              "variables are created under the same strategy scope. This may "
              "happen if you're restoring from a checkpoint outside the scope"
              .format(strategy, var))

        wi

🙏 Scream for help to Ukraine

Today, 25th May 2022, Russia continues bombing and firing Ukraine. Don't trust Russia, they are bombing us and brazenly lying in same time they are not doing this 😠, civilians and children are dying too! We are screaming and asking exactly you to help us, we want to survive, our families, children, older ones.
Please spread the information, and ask your governemnt to stop Russia by any means. We promise to work extrahard after survival to make the world safer place for all.

Links to the raise (1)

https://github.com/tensorflow/tensorflow/blob/f26800a1e5b1199cfc1a5aca916edc836b541687/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L911

Ways to fix

Error code:

import tensorflow as tf
from tensorflow.python.keras.optimizer_v2 import optimizer_v2
from tensorflow.python.ops import variables
from tensorflow.python.framework import dtypes

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  x = variables.Variable([[1.0]], dtype=dtypes.float32) #<--- Inside strategy scope
slot_shape = [2]
optimizer_1 = optimizer_v2.OptimizerV2(name='test')
optimizer_1.add_slot(x, 'test_slot', 'ones', shape=slot_shape)

Explanation:

When we adding a slot to the optimizer, it creates a default strategy.

strategy = distribute_ctx.get_strategy()

But if we want to change strategy, we can define strategy before our Variable. We used MirroredStrategy.

So when we adding our Variable to our strategy, As you know it uses a different strategy than optimizer use.

Our variable strategy is MirroredStrategy but the optimizer use tf.distribute.Strategy

To solve it, we need to use both of them in the same strategy either inside of scope or outside of scope doesn't matter.

Fix code:

import tensorflow as tf
from tensorflow.python.keras.optimizer_v2 import optimizer_v2
from tensorflow.python.ops import variables
from tensorflow.python.framework import dtypes

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  x = variables.Variable([[1.0]], dtype=dtypes.float32)
  slot_shape = [2]
  optimizer_1 = optimizer_v2.OptimizerV2(name='test')
  optimizer_1.add_slot(x, 'test_slot', 'ones', shape=slot_shape)
  # Both are inside scope optimizer and variable

Jul 09, 2021

anonim 13.0k

Add a possible fix

Please authorize to post fix

Raise code

Links to the raise (1)

See also in the other packages (1)

Ways to fix

Add a possible fix