Profiling PyTorch training without accidentally stalling the GPU [D]

Profiling PyTorch training has an interesting measurement problem: the more you measure, the more you can change the behavior of the run itself.

A simple example is torch.cuda.synchronize(). It gives cleaner timing boundaries, but it also inserts synchronization points into an otherwise asynchronous CUDA workload.

An alternative is to use CUDA events around selected boundaries and read them later, so timing can be captured without forcing synchronization in the hot path. This does not replace PyTorch Profiler or Nsight, but it can work as a lightweight first pass before deeper operator-level profiling.

I wrote a short technical note about this while working on an open-source PyTorch training diagnostics tool:

https://medium.com/p/19adf1054bcf