一次GAN項目背景下的tensorflow_datasets中mnist數據集的下載筆記

起因:幫我可愛的妹子跑一個GAN的代碼(已有,github),這回是在我自己實驗室的服務器上(到了展現我男友力的時候了!激動!)

我的背景:第一次 | 正式使用tensorflow | 要開始排坑了 | 我愛Pytorch | 用之前碎碎念

項目地址:https://github.com/google/compare_gan(很基礎的一個GAN代碼,谷歌大佬發佈的)

項目論文:https://arxiv.org/pdf/1406.2661.pdf

一、TF環境安裝

1. 建立虛擬環境:conda create -n GAN python=3.6

2. 進入虛擬環境:source activate GAN

3. 確認本地Linux版本是否支持cuda:uname -m && cat /etc/*release【得到反饋 CentOS Linux release 7.3.1611 (Core)】

4. 檢查本地gcc是否安裝:gcc --version【得到反饋 gcc (GCC) 4.8.5】

5. 開始安裝tensorflow:pip install tensorflow-gpu==1.15【得到反饋如下】

ERROR: matplotlib 2.1.2 requires cycler>=0.10, which is not installed.
ERROR: matplotlib 2.1.2 requires pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1, which is not installed.
ERROR: matplotlib 2.1.2 requires pytz, which is not installed.
Installing collected packages: six, google-pasta, astor, numpy, h5py, keras-applications, markdown, absl-py, werkzeug, protobuf, grpcio, tensorboard, tensorflow-estimator, keras-preprocessing, gast, opt-einsum, termcolor, wrapt, tensorflow-gpu
Successfully installed absl-py-0.8.1 astor-0.8.1 gast-0.2.2 google-pasta-0.1.8 grpcio-1.25.0 h5py-2.10.0 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.4 opt-einsum-3.1.0 protobuf-3.11.1 six-1.13.0 tensorboard-1.15.0 tensorflow-estimator-1.15.1 tensorflow-gpu-1.15.0 termcolor-1.1.0 werkzeug-0.16.0 wrapt-1.11.2

6. 意思是缺少cycler、pyparsing、pytz所以沒安成matplotlib,於是:pip install matplotlib【得到反饋 安裝成功】

二、安裝項目要求環境

1. 按照項目網址提供的方式:https://github.com/google/compare_gan,運行安裝setup.py下的依賴

(GAN) [zsun@gpu09 compare_gan]$ pwd
/users4/zsun/tensorflow/compare_gan/compare_gan
(GAN) [zsun@gpu09 compare_gan]$ cd ..
(GAN) [zsun@gpu09 compare_gan]$ pip install -e .

得到反饋

Found existing installation: compare-gan 3.0
    Can't uninstall 'compare-gan'. No files were found to uninstall.
  Running setup.py develop for compare-gan
Successfully installed chardet-3.0.4 compare-gan enum34-1.1.6 future-0.18.2 gin-config-0.1.4 googleapis-common-protos-1.6.0 idna-2.8 pandas-0.25.3 promise-2.2.1 pstar-0.1.9 qj-0.1.6 requests-2.22.0 scipy-1.3.3 tensorflow-datasets-1.0.1 tensorflow-gan-0.0.0.dev0 tensorflow-hub-0.7.0 tensorflow-metadata-0.15.1 tqdm-4.40.2 urllib3-1.25.7

2. 檢查TF環境是否裝好【最後一行可知,安裝成功,這輸出這麼多,666】

(GAN) [zsun@gpu09 compare_gan]$ python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('HELLO!')
>>> with tf.Session() as sess:
...     print(sess.run(hello))
...
2019-12-12 11:19:36.965890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-12 11:19:51.072275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
2019-12-12 11:19:51.147632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
2019-12-12 11:19:51.225782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 2 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:07:00.0
2019-12-12 11:19:51.801456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-12 11:19:51.804722: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-12 11:19:51.808385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-12 11:19:53.037509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1, 2
2019-12-12 11:19:53.038331: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-12 11:19:53.061923: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200035000 Hz
2019-12-12 11:19:53.075273: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe6ada91810 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-12-12 11:19:53.075359: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2019-12-12 11:20:48.462150: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe6aa1f53e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2019-12-12 11:20:48.462228: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-12-12 11:20:48.462243: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-12-12 11:20:48.462254: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-12-12 11:20:52.105700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
2019-12-12 11:20:52.163494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
2019-12-12 11:20:52.230471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 2 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:07:00.0
2019-12-12 11:20:52.861860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-12 11:20:52.862299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-12 11:20:52.862343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-12 11:20:57.968244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1, 2
2019-12-12 11:20:57.968473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-12 11:20:59.756185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-12 11:20:59.756279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 1 2 
2019-12-12 11:20:59.756306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N Y Y 
2019-12-12 11:20:59.756323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1:   Y N Y 
2019-12-12 11:20:59.756339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 2:   Y Y N 
tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15190 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-12-12 11:21:01.088668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15190 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-12-12 11:21:01.417236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 5406 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0)
b'HELLO!'

3. 檢查 tensorflow_datasets 是否安裝成功,並輸出tfds都包含了哪些數據集:

>>> import tensorflow_datasets as tfds
>>> print(tfds.list_builders())
['bair_robot_pushing_small', 'cats_vs_dogs', 'celeb_a', 'celeb_a_hq', 'cifar10', 'cifar100', 'coco2014', 'diabetic_retinopathy_detection', 'dummy_dataset_shared_generator', 'dummy_mnist', 'fashion_mnist', 'image_label_folder', 'imagenet2012', 'imdb_reviews', 'lm1b', 'lsun', 'mnist', 'moving_mnist', 'nsynth', 'omniglot', 'open_images_v4', 'quickdraw_bitmap', 'squad', 'starcraft_video', 'svhn_cropped', 'tf_flowers', 'wmt_translate_ende', 'wmt_translate_enfr']

沒問題,安裝成功。

三、嘗試運行訓練代碼

1. 首次運行 python main.py --model_dir try --gin_config ../example_configs/dcgan_celeba64.gin

出現錯誤。

  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/download/downloader.py", line 133, in _sync_download
    response = session.get(url, stream=True)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe51008c2e8>: Failed to establish a new connection: [Errno 113] No route to host',))

於是懷疑問題出現在數據集無法下載上,直接嘗試能否下載數據集:

>>> ds_train = tfds.load(name="celeb_a", split="train")

報同樣的錯誤,定位成功。

2. 嘗試手動下載celeb_a數據集,發現官網不是很好找,於是換一個常用數據集mnist

更改代碼的gin文件即可。

python main.py --model_dir try --gin_config ../example_configs/mnist.gin

3. 檢查 tensorflow_datasets 是否可以使用:

【用tfds.load執行一系列的批量示例、轉換操作,然後再調用】

>>> ds_train = tfds.load(name="mnist", split="train")

報錯:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe51008c2e8>: Failed to establish a new connection: [Errno 113] No route to host',))

4. 嘗試一下另一種方法:【DatasetBuilder直接實例化或者用tfds.builder字符串讀取】【這份其實是上面tfds.load的源碼

# Fetch the dataset directly
mnist = tfds.image.MNIST()
# or by string name
mnist = tfds.builder('mnist')
mnist.download_and_prepare()

仍然報同一個錯誤:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe510110d68>: Failed to establish a new connection: [Errno 113] No route to host',))

使用mnist(有四個文件,train-img,train-label,t10k-img,t10k-label)檢查了一下到底有沒有成功下載,發現指定下載目錄下成功下載了兩個(兩個label文件,所以初步懷疑報錯是因爲兩個大文件太大下載不成功),並帶上了他們的info文件(dict形式,寫着名稱,下載地址等)

yann.lecu.com_exdb_mnis_t10k-labe-idx1-ubytlX-SfSvqLNNrV7AqoACgdSrEYP_4wsDz4T_rhacyC0o.gz       yann.lecu.com_exdb_mnis_trai-labe-idx1-ubyt7cc_IeM51G_ngIY2ORleKjMjLVCXd-TCUHlYvEiRiKI.gz
yann.lecu.com_exdb_mnis_t10k-labe-idx1-ubytlX-SfSvqLNNrV7AqoACgdSrEYP_4wsDz4T_rhacyC0o.gz.INFO  yann.lecu.com_exdb_mnis_trai-labe-idx1-ubyt7cc_IeM51G_ngIY2ORleKjMjLVCXd-TCUHlYvEiRiKI.gz.INFO

但是另外兩個image文件並沒有下載成功,上面的錯誤正是報在image文件無法下載這個問題上。

5.  tensorflow_datasets 爲什麼安裝成功但是不能使用?

參考博客:

https://blog.csdn.net/i8088/article/details/79126150

https://blog.csdn.net/abc13526222160/article/details/85109498

https://blog.csdn.net/jiaoyangwm/article/details/79254667

能查到的問題都是在解決input_data(如下)無法自動下載mnist文件的問題,而我們的代碼使用的不是input_data,而是tfds。所以我們面臨的是tfds無法自動下載mnist文件的問題。

我們的這個問題無法查到。

from tensorflow.examples.tutorials.mnist import input_data #這是TensorFlow 爲了教學Mnist而提前設計好的程序
mnist = input_data.read_data_sets('MNIST_data', one_hot=True) #TensorFlow 會檢測數據是否存在。當數據不存在時,系統會自動,在當前代碼py文件位置,自動創建MNIST_data文件夾,並將數據下載到該件夾內。當執行完語句後,讀者可以自行前往MNIST_data/文件夾下查看上述4 個文件是否已經被正確地下載
#若因網絡問題無法正常下載,可以前往MNIST官網http://yann.lecun.com/exdb/mnist/使用下載工具下載上述4 個文件, 並將它們複製到MNIST_data/文件夾中。

能查到的博客上說的都是同一個方法:自己手動下載數據 ,o好8,那就下載一個。

6. 手動下載數據集

在自己電腦上從網址 http://yann.lecun.com/exdb/mnist/ 手動下載數據集(四個,上文有提到)

這個時候需要注意⚠️,有些瀏覽器下載壓縮包會自動給解壓。

我的電腦就自動給解壓了,並且在瀏覽器裏沒有找到 [不要解壓] 的選項。

於是換一種下載方法,在數據集處右鍵獲得數據集鏈接,直接在命令行輸入 [wget + 鏈接] 下載未解壓版本。

此處也可以直接使用上面的代碼來下載未解壓文件:

from tensorflow.examples.tutorials.mnist import input_data #這是TensorFlow 爲了教學Mnist而提前設計好的程序
mnist = input_data.read_data_sets('MNIST_data', one_hot=True) #TensorFlow 會檢測數據是否存在。當數據不存在時,系統會自動,在當前代碼py文件位置,自動創建MNIST_data文件夾,並將數據下載到該件夾內。當執行完語句後,讀者可以自行前往MNIST_data/文件夾下查看上述4 個文件是否已經被正確地下載
#若因網絡問題無法正常下載,可以前往MNIST官網http://yann.lecun.com/exdb/mnist/使用下載工具下載上述4 個文件, 並將它們複製到MNIST_data/文件夾中。

7. 數據集放到指定位置

仔細觀察步驟4的報錯信息,發現其自動下載數據集的存放地址爲 [~/tensorflow_datasets/mnist/1.0.0]

於是新建一個文件夾,把剛剛下載好的未解壓的文件放到這裏。

再次運行步驟4的tfds.load('mnist')的代碼

手動下載數據集並放到正確位置後,url錯誤消失,但出現新的錯誤,仍然無法成功load數據集:

  File "/users4/zsun/tensorflow/compare_gan/compare_gan/datasets.py", line 247, in _load_dataset
    as_dataset_kwargs={"shuffle_files": False})  # \\\\\\\\\\\\\\\\\\\\\\
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 52, in disallow_positional_args_dec
    return fn(*args, **kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/registered.py", line 250, in load
    dbuilder = builder(name, data_dir=data_dir, **builder_kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/registered.py", line 149, in builder
    return _DATASET_REGISTRY[name](**builder_kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 52, in disallow_positional_args_dec
    return fn(*args, **kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_builder.py", line 531, in __init__
    super(GeneratorBasedBuilder, self).__init__(**kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 52, in disallow_positional_args_dec
    return fn(*args, **kwargs)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_builder.py", line 158, in __init__
    self.info.read_from_directory(self._data_dir)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_info.py", line 312, in read_from_directory
    parsed_proto = read_from_json(json_filename)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_info.py", line 543, in read_from_json
    dataset_info_json_str = f.read()
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 122, in read
    self._preread_check()
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 84, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512)
tensorflow.python.framework.errors_impl.NotFoundError: /users4/zsun/tensorflow_datasets/mnist/1.0.0/dataset_info.json; No such file or directory

報錯內容:tensorflow_datasets/mnist/1.0.0/dataset_info.json; No such file or directory

8. 這個 [dataset_info.json] 是什麼?

官網只提供了四個數據文件,並沒有這個json文件。

但想要成功運行tfds.load 或者 mnist.download_and_prepare(),看起來必須要這個文件。

那麼找一下

查看tensorflow源碼(根據tensorflow_datasets.core相關的報錯位置查看源碼)

https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/dataset_info.py

https://github.com/tensorflow/datasets/blob/v1.3.0/tensorflow_datasets/core/dataset_builder.py#L236-L308

查看tensorflow官方文檔 https://tensorflow.google.cn/datasets/api_docs/python/tfds/core/DatasetInfo 其中有關於數據集dataset的info文件,靈機一動,誒,會不會是他呢?

於是查找到dataset.info的輸出方式,輸出看看,嗯,格式差不多。。。

那我們把它存成這個json文件試一下在這裏https://www.tensorflow.org/datasets/api_docs/python/tfds/core/DatasetInfo找到了DatasetInfo有轉爲json的屬性,當然直接用啊!

轉爲json之後,寫到文件裏,整體代碼如下

>>> mnist = tfds.image.MNIST()
>>> mnist.info
<tensorflow_datasets.core.dataset_info.DatasetInfo object at 0x7fd6945811d0>
>>> print(mnist.info)
tfds.core.DatasetInfo(
    name='mnist',
    version=1.0.0,
    description='The MNIST database of handwritten digits.',
    urls=['http://yann.lecun.com/exdb/mnist/'],
    features=FeaturesDict({
        'image': Image(shape=(28, 28, 1), dtype=tf.uint8),
        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10)
    },
    total_num_examples=0,
    splits={

    },
    supervised_keys=('image', 'label'),
    citation='"""
        @article{lecun2010mnist,
          title={MNIST handwritten digit database},
          author={LeCun, Yann and Cortes, Corinna and Burges, CJ},
          journal={ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist},
          volume={2},
          year={2010}
        }

    """',
)

>>> print(mnist.info.as_json)
{
  "citation": "@article{lecun2010mnist,\n  title={MNIST handwritten digit database},\n  author={LeCun, Yann and Cortes, Corinna and Burges, CJ},\n  journal={ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist},\n  volume={2},\n  year={2010}\n}\n",
  "description": "The MNIST database of handwritten digits.",
  "location": {
    "urls": [
      "http://yann.lecun.com/exdb/mnist/"
    ]
  },
  "name": "mnist",
  "supervisedKeys": {
    "input": "image",
    "output": "label"
  },
  "version": "1.0.0"
}
>>>
>>> import json
>>> with open('/users4/zsun/tensorflow_datasets/mnist/1.0.0/dataset_info.json', 'w') as f:json.dump(mnist.info.as_json, f)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_info.py", line 546, in read_from_json
    dataset_info_pb2.DatasetInfo())
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 430, in Parse
    return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 450, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 592, in _ConvertFieldValuePair
    raise ParseError(str(e))
google.protobuf.json_format.ParseError: Message type "tensorflow_datasets.DatasetInfo" has no field named "{".
 Available Fields(except extensions): ['name', 'description', 'version', 'citation', 'sizeInBytes', 'location', 'downloadChecksums', 'schema', 'splits', 'supervisedKeys']

依舊有問題,說明自己建json文件不可行。

懷疑input_data與tfds所需要的數據集格式不同,inputdata的解決方案並不適用。

9. 本地使用tfds下載安裝數據集

之前報url的錯誤是是服務器上,由於服務器無法翻牆,又無法保證鏈接穩定;

目前來看其他方法也行不通。

所以最後一個辦法。

本地電腦安裝tensorflow==1.15 tensorflow_dataset==1.0.1

然後本地運行某一種方法:

mnist = tfds.builder('mnist')
mnist.download_and_prepare()

成功下載並處理。

查看處理後的文件樣式:

果然和input_data.py相差十萬八千里

再次運行python main.py --model_dir try --gin_config ../example_configs/mnist.gin,不再出現數據集的問題。

成功!!!

總結:input_data 和 tfds 的數據集調用方式 和 問題解決方式不一樣,目前來看mnist在tfds上只有換電腦這一種解決方式

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章