Trying to create optimizer slot variable under the scope for tf.distribute.Strategy ((param1)), which is different from the scope used for the original variable ((param1)). Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope

Package:

tensorflow

158813

Exception Class:

ValueError

Raise code

  initializer, shape=slot_shape, dtype=var.dtype)
      else:
        initial_value = initializer

      with self._distribution_strategy_scope():
        strategy = distribute_ctx.get_strategy()
        if not strategy.extended.variable_created_in_scope(var):
          raise ValueError(
              "Trying to create optimizer slot variable under the scope for "
              "tf.distribute.Strategy ({}), which is different from the scope "
              "used for the original variable ({}). Make sure the slot "
              "variables are created under the same strategy scope. This may "
              "happen if you're restoring from a checkpoint outside the scope"
              .format(strategy, var))

        wi

Links to the raise (1)

https://github.com/tensorflow/tensorflow/blob/f26800a1e5b1199cfc1a5aca916edc836b541687/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L911

Ways to fix

Error code:

import tensorflow as tf
from tensorflow.python.keras.optimizer_v2 import optimizer_v2
from tensorflow.python.ops import variables
from tensorflow.python.framework import dtypes

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  x = variables.Variable([[1.0]], dtype=dtypes.float32) #<--- Inside strategy scope
slot_shape = [2]
optimizer_1 = optimizer_v2.OptimizerV2(name='test')
optimizer_1.add_slot(x, 'test_slot', 'ones', shape=slot_shape)

Explanation:

When we adding a slot to the optimizer, it creates a default strategy.

strategy = distribute_ctx.get_strategy()

But if we want to change strategy, we can define strategy before our Variable. We used MirroredStrategy.

So when we adding our Variable to our strategy, As you know it uses a different strategy than optimizer use.

Our variable strategy is MirroredStrategy but the optimizer use tf.distribute.Strategy

To solve it, we need to use both of them in the same strategy either inside of scope or outside of scope doesn't matter.

Fix code:

import tensorflow as tf
from tensorflow.python.keras.optimizer_v2 import optimizer_v2
from tensorflow.python.ops import variables
from tensorflow.python.framework import dtypes

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  x = variables.Variable([[1.0]], dtype=dtypes.float32)
  slot_shape = [2]
  optimizer_1 = optimizer_v2.OptimizerV2(name='test')
  optimizer_1.add_slot(x, 'test_slot', 'ones', shape=slot_shape)
  # Both are inside scope optimizer and variable

Jul 09, 2021

anonim 13.0k

Add a possible fix

Please authorize to post fix

Raise code

Links to the raise (1)

See also in the other packages (1)

Ways to fix

Add a possible fix