How to Use Shared Storage

If you want to use your own storage during using NNI, shared storage can satisfy you. Instead of using training service native storage, shared storage can bring you more convenience. All the information generated by the experiment will be stored under /nni folder in your shared storage. All the output produced by the trial will be located under /nni/{EXPERIMENT_ID}/trials/{TRIAL_ID}/nnioutput folder in your shared storage. This saves you from finding for experiment-related information in various places. Remember that your trial working directory is /nni/{EXPERIMENT_ID}/trials/{TRIAL_ID}, so if you upload your data in this shared storage, you can open it like a local file in your trial code without downloading it. And we will develop more practical features in the future based on shared storage.

Note

Shared storage is currently in the experimental stage. We suggest use AzureBlob under Ubuntu/CentOS/RHEL, and NFS under Ubuntu/CentOS/RHEL/Fedora/Debian for remote. And make sure your local machine can mount NFS or fuse AzureBlob and has sudo permission on your remote runtime. We only support shared storage under training service with reuse mode for now.

Example

If you want to use AzureBlob, add below to your config. Full config file see mnist-sharedstorage/config_azureblob.yml.

sharedStorage:
    storageType: AzureBlob
    localMountPoint: ${your/local/mount/point}
    remoteMountPoint: ${your/remote/mount/point}
    storageAccountName: ${replace_to_your_storageAccountName}
    storageAccountKey: ${replace_to_your_storageAccountKey}
    # If you did not set storageAccountKey, you need use `az login` with Azure CLI at first and set resourceGroupName.
    # resourceGroupName: ${replace_to_your_resourceGroupName}
    containerName: ${replace_to_your_containerName}
    # usermount means you have already mount this storage on localMountPoint
    # nnimount means nni will try to mount this storage on localMountPoint
    # nomount means storage will not mount in local machine, will support partial storages in the future
    localMounted: nnimount

If you want to use NFS, add below to your config. Full config file see mnist-sharedstorage/config_nfs.yml.

sharedStorage:
    storageType: NFS
    localMountPoint: ${your/local/mount/point}
    remoteMountPoint: ${your/remote/mount/point}
    nfsServer: ${nfs-server-ip}
    exportedDirectory: ${nfs/exported/directory}
    # usermount means you have already mount this storage on localMountPoint
    # nnimount means nni will try to mount this storage on localMountPoint
    # nomount means storage will not mount in local machine, will support partial storages in the future
    localMounted: nnimount