Hello,

I thin that you must use the cudaDeviceSynchronize() funtion to synchronise between CUDA kernels. Otherwise, you risk to lose the result of some CUDA threads.

I hope it helped.

Sidi,

Thank you dear all for your replies. CudaDeviceSynchronize() is important in host code when we have multiple streams or when we want to debug the code. I think that when we did not specify the stream's number in the call of a kernel from host , it refer to the default stream, hence we don't need CudaDeviceSynchronize() because all the instructions in the same stream are serialized until we call explicitly asynchronous operations. Do you agree with me?

I agree with you if we launch multiple kernels from host, but if we launch just one kernel, I think that the host will wait until the kernel is finished which is the case for cudaMemcpy. What do you think?

I find a link which can be useful for others that have the same problem I have. Actually, when a kernel is launched it returns control to host which continue the following instructions. Fortunately cudaMemcpy operations ensure synchronization.

https://devblogs.nvidia.com/parallelforall/how-implement-performance-metrics-cuda-cc/

Thank you for the answer and question,but I still don't understand what is the difference between cudaMemcpy and cudaDeviceSynchronize. Could you tell me what is the difference please?