Some tips about data transferring between local and server

General practice:

• Compress the file (bz2, zip, or tar.gz) before transferring the data

  • find . -type d -maxdepth 1 -exec zip -r {}.zip {} \;

• Don’t send data from local to pure storage directly

• If it is just for sync, use rsync over scp to avoid sending the repetitive data to the server;

• If you prefer using rsync, -av would work; add -z to send folder directly (-z will compress the file)

• If you prefer using scp, please compress the folder first. It is being tested that `-p -C -o 'CompressionLevel 9' -o 'IPQoS throughput'` speeds up the transfer speed from 10% to 50%.

• If you found the transfer speed is super-slow,

  • Reconnect the VPN

  • retry

  • Test the network speed (by typing network speed test at google, and check the network speed) under VPN or without VPN. If the speed under VPN is about 50% slower than that without VPN, it looks normal; otherwise, contact the admin.

  • Use `df -h` to check the storage left at the server. If there is no much storage, please try to clean your local storage, and contact the admin.

• Leave the transferring task working under the backend.

• Use multiple threads if needed, but please check the md5sum in case some threads failed.

Others:

• What’s the speed difference between rsync/scp over FileZila?