training guide¶

The guide is divided into three steps.

Upload the compressed package from the local to the platform personal data
Download platform personal data into the instance
Perform training with automatic upload of results and shutdown

The operation process is roughly as follows:

Upload personal data locally¶

Locally, the training data is first typed into a compressed package in the common format of zip and tar.gz. Then download and install the oss command line tool locally, and use the tool to upload the compressed package to the platform's personal data middle.

tip

In the process of uploading data to personal space. You can perform operations before starting the instance, saving the cost of running the instance.

The Windows system is used here as the local environment. For other systems, please refer to oss command line tool for installation.

Download the OSS (Windows) executable. Change the oss_windows_x86_64.exe filename to oss.exe after the download is complete. Open CMD (Command Prompt) or PowerShell under Windows.

Use the cd command to change to the directory where oss.exe is located, where oss.exe is saved under D:\Download. Here it is executed using the PowerShell terminal.

PS C:\Users\windows> cd D:\Download
PS D:\Download>

Execute .\oss in this directory to execute the command. You need to log in before uploading. The login account is the account name and password of the Gpushare cloud platform, not an instance. The account name is the mobile phone number. If it is a non-mainland China mobile phone number, you need to add the country number with +.

# execute .\oss login
PS D:\Download> .\oss login
Username: 139********
Password:**********
139******** login successfully!

After the login is successful, the file transfer operation will be performed, and the local compressed package will be uploaded to personal data. Note that the uploaded file extension only supports common archive formats. The path to the zip file here is D:\Datasets\MNIST.zip.

# Execute .\oss cp D:\Datasets\MNIST.zip oss://
PS D:\Download> .\oss cp D:\Datasets\MNIST.zip oss://

Start at 2021-10-22 06:21:48.2061924 +0000 UTC

[------------------------------------] 100.00% 52.34MB/s 76.55MB/76.55MB 1.53s

Upload successfully, 76.55MB, n/a, D:\Datasets\MNIST.zip --> oss://MNIST.zip, cost [7020], status [200], request id [0000017CA6A9BE01901395D26CE9A228]

After the upload is successful, the file can be viewed on the personal data page of the platform.

Example download personal data¶

After the platform creates or starts an instance, go to the terminal through the page of JupyterLab or connect to the terminal through SSH Client.

Also use oss in the terminal to log in first. The login account is the account name and password of the Gpushare cloud platform, not an instance. The account name is the mobile phone number. If it is a non-mainland China mobile phone number, you need to add the country number with +.

# execute oss login
 oss login
Username: 139********
Password:**********
139******** login successfully!

After successful login, the file download operation will be performed, and the file just uploaded will be downloaded to /hy-tmp (models with shared storage can use /hy-nas).

# Execute oss cp oss://MNIST.zip /hy-tmp
 oss cp oss://MNIST.zip /hy-tmp
Start at 2021-10-22 06:37:17.227649376 +0000 UTC

[------------------------------------] 100.00% 62.85MB/s 76.55MB/76.55MB 1.42s
Waiting to rename temporary file...

Download successfully, 76.55MB, n/a, oss://MNIST.zip --> /hy-tmp/MNIST.zip, cost [2422], status [206], request id [0000017CA6B7D6419012DB7767D11959]

After the download is successful, go to the download path and decompress the compressed package.

# Execute cd /hy-tmp to enter the directory
 cd /hy-tmp
# Execute unzip -q MNIST.zip to decompress the archive
/hy-tmp# unzip -q MNIST.zip

Perform training and automatically upload results and shut down¶

The data already exists in the instance, and the next stage is to perform training. Code can be written using tools such as JupyterLab or VSCode. Note that the data path referenced in the code needs to be changed to the data path within the instance. In the above example, /hy-tmp is used. Long training tasks need to be run in the background through Tmux, so that the local computer can be shut down without interrupting training.

For pay-as-you-go instances, you can upload the results to personal data after training is completed, and then shut down and terminate billing. This operation can save the usage cost of pay-as-you-go, and avoid the data of /hy-tmp cannot be viewed and downloaded due to the machine being occupied.

The operation of the sample script requires the following prerequisites, and some operations are adjusted according to actual needs.

The result of training is in /hy-tmp/result
In the instance, you need to use the platform account mobile phone number and password in advance to execute oss login to successfully log in

Create an upload.sh script using vim in the /root directory.

cd
vim upload.sh

The script content is as follows, use the vim editor, enter I to enter the insert mode, paste the entire script content, and then enter Esc , :wq and Enter to save.

#!/bin/bash
set -e

cd /hy-tmp
# archive name
file="result-$(date "+%Y%m%d-%H%M%S").zip"
# Make the result directory into a zip archive
zip -q -r "${file}" result
# Upload to the backup folder in personal data via oss
oss cp "${file}" oss://backup/
rm -f "${file}"

# Shut down after successful transfer
shutdown

Add execute permission to the script.

chmod u+x upload.sh

This script implements /hy-tmp/result into a zip archive, and then uploads this file to personal data. Perform a shutdown operation after successful execution.

The /root/upload.sh script can be called at the end of the training task code to run. It is recommended to test it for the first time. As quoted in Python code:

import os
os.system('/root/upload.sh')

When the training execution is completed and shut down, you can use the oss command line tool locally to download the packaged result file. When the instance machine is occupied and cannot be powered on, you can also create a new instance on another machine and download data to the new instance through oss. For details about oss related operations, refer to the documentation oss command tool.

# Log in to the Gpushare cloud account, use the account name and password of Gpushare cloud, the account name is the mobile phone number
# If it is a non-mainland China mobile phone number, you need to add the area code with +
 oss login
Username: 139********
Password:**********
139******** login successfully!

# View files and folders, -s means only display file names
 oss ls -s oss://
Folder list:
oss://
oss://backup/

Object list:
oss://backup/result-20211018-164323.zip

# Download the result-20211018-164323.zip file in the personal data to the current directory
 oss cp oss://backup/result-20211018-164323.zip .
Download successfully, 310B, n/a, oss://backup/result-20211018-164323.zip --> /root/result-20211018-164323.zip, cost [57], status [200], request id [0000017C92929EF49014BE16738685B7 ]