> For the complete documentation index, see [llms.txt](https://lwang010.gitbook.io/longw/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lwang010.gitbook.io/longw/mlops/chap-6.-team-infra/todo/some-tips-about-data-transferring-between-local-and-server.md).

# Some tips about data transferring between local and server

General practice:

* Compress the file (bz2, zip, or tar.gz) before transferring the data
  * ```
    find . -type d -maxdepth 1 -exec zip -r {}.zip {} \;
    ```
* Don’t send data from local to pure storage directly
* If it is just for sync, use rsync over scp to avoid sending the repetitive data to the server;
* If you prefer using rsync, -av would work; add -z to send folder directly (-z will compress the file)
* If you prefer using scp, please compress the folder first. It is being tested that \`-p -C -o 'CompressionLevel 9' -o 'IPQoS throughput'\` speeds up the transfer speed from 10% to 50%. 
* If you found the transfer speed is super-slow, 
  * Reconnect the VPN
  * retry
  * Test the network speed (by typing network speed test at google, and check the network speed) under VPN or without VPN. If the speed under VPN is about 50% slower than that without VPN, it looks normal; otherwise, contact the admin.
  * Use \`df -h\` to check the storage left at the server. If there is no much storage, please try to clean your local storage, and contact the admin.
* Leave the transferring task working under the backend.
* Use multiple threads if needed, but please check the md5sum in case some threads failed.

Others:

* What’s the speed difference between rsync/scp over FileZila?