public static extern cudnnStatus cudnnActivationForward( cudnnHandle handle, cudnnActivationMode mode, ref float alpha, cudnnTensorDescriptor srcDesc, CUdeviceptr srcData, ref float beta, cudnnTensorDescriptor destDesc, CUdeviceptr destData );
public static extern cudnnStatus cudnnActivationBackward( cudnnHandle handle, cudnnActivationMode mode, ref double alpha, cudnnTensorDescriptor srcDesc, CUdeviceptr srcData, cudnnTensorDescriptor srcDiffDesc, CUdeviceptr srcDiffData, cudnnTensorDescriptor destDesc, CUdeviceptr destData, ref double beta, cudnnTensorDescriptor destDiffDesc, CUdeviceptr destDiffData );
/* Function to perform backward activation */
public void ActivationBackward(cudnnActivationMode mode,
float alpha,
TensorDescriptor srcDesc,
CudaDeviceVariable<float> srcData,
TensorDescriptor srcDiffDesc,
CudaDeviceVariable<float> srcDiffData,
TensorDescriptor destDesc,
CudaDeviceVariable<float> destData,
float beta,
TensorDescriptor destDiffDesc,
CudaDeviceVariable<float> destDiffData
)
{
res = CudaDNNNativeMethods.cudnnActivationBackward(_handle, mode, ref alpha, srcDesc.Desc, srcData.DevicePointer, srcDiffDesc.Desc, srcDiffData.DevicePointer, destDesc.Desc, destData.DevicePointer, ref beta, destDiffDesc.Desc, destDiffData.DevicePointer);
Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "cudnnActivationForward", res));
if (res != cudnnStatus.Success) throw new CudaDNNException(res);
}
/* Function to perform backward activation */
public void ActivationBackward(cudnnActivationMode mode,
double alpha,
TensorDescriptor srcDesc,
CudaDeviceVariable <double> srcData,
TensorDescriptor srcDiffDesc,
CudaDeviceVariable <double> srcDiffData,
TensorDescriptor destDesc,
CudaDeviceVariable <double> destData,
double beta,
TensorDescriptor destDiffDesc,
CudaDeviceVariable <double> destDiffData
)
{
res = CudaDNNNativeMethods.cudnnActivationBackward(_handle, mode, ref alpha, srcDesc.Desc, srcData.DevicePointer, srcDiffDesc.Desc, srcDiffData.DevicePointer, destDesc.Desc, destData.DevicePointer, ref beta, destDiffDesc.Desc, destDiffData.DevicePointer);
Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "cudnnActivationForward", res));
if (res != cudnnStatus.Success)
{
throw new CudaDNNException(res);
}
}
private void DoActivation(Volume <double> result, cudnnActivationMode mode)
{
var resultStorage = result.Storage as VolumeStorage;
if (resultStorage == null)
{
throw new ArgumentException($"{nameof(result)} storage should be VolumeStorage", nameof(result));
}
// Copy to device if not already done
this._volumeStorage.CopyToDevice();
resultStorage.CopyToDevice();
// Synchro
this._context.DefaultStream.Synchronize();
// Relu
using (var activationDesc = new ActivationDescriptor())
using (var srcDesc = new TensorDescriptor())
using (var resultDesc = new TensorDescriptor())
{
var n = result.Shape.GetDimension(3);
var c = result.Shape.GetDimension(2);
var h = result.Shape.GetDimension(1);
var w = result.Shape.GetDimension(0);
srcDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Double, n, c, h, w);
resultDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Double, n, c, h, w);
activationDesc.SetActivationDescriptor(mode, cudnnNanPropagation.NotPropagateNan, 0.0);
this._context.CudnnContext.ActivationForward(activationDesc.Desc, 1.0, srcDesc, this._volumeStorage.DeviceBuffer, 0.0,
resultDesc, resultStorage.DeviceBuffer);
}
resultStorage.CopiedToDevice = true;
}
private void DoActivationGradient(Volume <float> input, Volume <float> outputGradient, Volume <float> inputGradient, cudnnActivationMode mode)
{
var inputStorage = input.Storage as VolumeStorage;
var inputGradientStorage = inputGradient.Storage as VolumeStorage;
var outputStorage = this._volumeStorage;
var outputGradientStorage = outputGradient.Storage as VolumeStorage;
// Copy to device if not already done
outputStorage.CopyToDevice();
outputGradientStorage.CopyToDevice();
inputGradientStorage.CopyToDevice();
// Synchro
this._context.DefaultStream.Synchronize();
using (var activationDesc = new ActivationDescriptor())
using (var srcDesc = new TensorDescriptor())
using (var srcDiffDesc = new TensorDescriptor())
using (var destDesc = new TensorDescriptor())
using (var destDiffDesc = new TensorDescriptor())
{
var n = this.Shape.GetDimension(3);
var c = this.Shape.GetDimension(2);
var h = this.Shape.GetDimension(1);
var w = this.Shape.GetDimension(0);
srcDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Float, n, c, h, w);
srcDiffDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Float, n, c, h, w);
destDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Float, n, c, h, w);
destDiffDesc.SetTensor4dDescriptor(cudnnTensorFormat.NCHW, cudnnDataType.Float, n, c, h, w);
activationDesc.SetActivationDescriptor(mode, cudnnNanPropagation.NotPropagateNan,
0.0);
this._context.CudnnContext.ActivationBackward(activationDesc, 1.0f,
srcDesc, outputStorage.DeviceBuffer,
srcDiffDesc, outputGradientStorage.DeviceBuffer,
destDesc, inputStorage.DeviceBuffer,
0.0f,
destDiffDesc, inputGradientStorage.DeviceBuffer);
}
}
public static extern cudnnStatus cudnnSetActivationDescriptor(
cudnnActivationDescriptor activationDesc,
cudnnActivationMode mode,
cudnnNanPropagation reluNanOpt,
double reluCeiling);
/// <summary>
/// This function queries the parameters of the previouly initialized activation descriptor object.
/// </summary>
/// <param name="mode">Enumerant to specify the activation mode.</param>
/// <param name="reluNanOpt">Nan propagation option for the relu.</param>
/// <param name="reluCeiling">The ceiling for the clipped relu.</param>
public void GetActivationDescriptor( ref cudnnActivationMode mode,
ref cudnnNanPropagation reluNanOpt,
ref double reluCeiling)
{
res = CudaDNNNativeMethods.cudnnGetActivationDescriptor(_desc, ref mode, ref reluNanOpt, ref reluCeiling);
Debug.WriteLine(String.Format("{0:G}, {1}: {2}", DateTime.Now, "cudnnGetActivationDescriptor", res));
if (res != cudnnStatus.Success) throw new CudaDNNException(res);
}
