dlprof

Docker comes with dlprof tool see details at https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/#profiling.

Run dlprof train.sh:

You will get a set of files, open nsys_profile.qdrep using the Nsight Systems GUI using vnc (outside your docker)

References:

https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/clara_faq.html