【Linux】keras多GPU，分佈式訓練出錯

報錯內容：

Epoch 1/1
2020-01-13 21:39:19.392806: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-01-13 21:39:22.432074: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at tensor_array_ops.cc:122 : Not found: Resource __per_step_10/_tensor_arraysinput_ta_0_4/N10tensorflow11TensorArrayE does not exist.
Traceback (most recent call last):
  File "/home/hans/WorkSpace/client.py", line 90, in <module>
    hist = client.train(convertDict(dictItem), dictItem.get(f"DirName"), dictItem.get(f"round"), CRNNconfig["ESTIMATION"]["SAVE_CLIENT_MODEL"])
  File "/home/hans/WorkSpace/FLutils/client.py", line 44, in train
    verbose=1)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training.py", line 1732, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training_generator.py", line 220, in fit_generator
    reset_metrics=False)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training.py", line 1514, in train_on_batch
    outputs = self.train_function(ins)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in __call__
    run_metadata=self.run_metadata)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
    run_metadata_ptr)
  File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Resource __per_step_10/_tensor_arraysinput_ta_0_4/N10tensorflow11TensorArrayE does not exist.
	 [[{{node training/Adadelta/gradients/replica_0/model_1/rnn1_bgru1/while_1/TensorArrayReadV3_grad/TensorArrayGrad/TensorArrayGradV3}}]]
	 [[{{node training/Adadelta/gradients/b_count_26}}]]

花了一天時間，經歷重裝驅動到CUDA到系統到虛擬環境，最終最確定是因爲keras版本問題。我其他兩臺機器的版本是2.2.4，出問題這臺電腦是2.3.1，版本降到2.2.4就不報這個錯誤了。

我是第一個遇到這個問題的人，至少之前在網上根本搜不到相關問題的帖子。