The source code and experimental data of

Benchmarking State-of-the-Art Deep Learning Software Tools

(Version 2, 26 August 2016)

Declaration: The runtime performance of each software tool depends not only on the hardware platform, but also on the third-party libraries and the network configuration files. Our results only reflect the performance of the tested networks with the associated configuration files and the specified third-party libraries on our testing machines, which are not necessarily the best performance that can be achieved by the software tool.

Go to newest version



Fully Connected Networks (FCNs)

Caffe

Source Code: CaffeFCNs.zip
Usage example: caffe time -model=fcn5-b32.prototxt -gpu=0

Data: created by the script: createFakeDataForCaffeFCN.py

CNTK

Source Code: CNTKFCNs.zip
Usage example: cntk configFile=fcn5.cntk configName=fcn5 deviceId=0 minibatch=32

Data: created by the script: createDataForCNTKFCN.py

TensorFlow

Source Code: TensorFlowFCNs.zip
Usage example: python fcn5.py -e 2 -b 32 -i 4 -d 0

Data: random generated in the source code.

Torch

Source Code: TorchFCNs.zip
Usage example: th fcn5.lua -nGPU 1 -deviceId 0 -batchSize 32 -nEpochs 2 -nIterations 4

Data: random generated in the source code.

Convolutional Neural Networks (CNNs)

AlexNet

Caffe

Source Code: CaffeAlexNet.zip
Usage example: caffe time -model=alexnet-b256.prototxt -iterations=10 -gpu=0

Data: created by the script: createFakeImageNetForCaffeCNN.py

CNTK

Source Code: CNTKAlexNet.zip
Usage example: cntk configFile=alexnet.cntk configName=alexnet deviceId=0 minibatchSize=256 epochSize=2560 maxEpochs=4

Data: created by the script: createFakeImageNetDataForCNTKCNN.py and createLabelMapForCNTKCNN.py

TensorFlow

Source Code: TensorFlowAlexNet.zip
Usage example: python alexnet.py -e 4 -b 256 -i 10 -d 0

Data: random generated in the source code.

Torch

Source Code: TorchAlexNet.zip
Usage:

Data: random generated in the source code.

ResNet-50

Caffe

Source Code: CaffeResNet.zip
Usage example: caffe time -model=resnet-b32.prototxt -iterations=4 -gpu=0

Data: the same as Caffe AlexNet data.

CNTK

Source Code: CNTKResNet.zip
Usage example: cntk configFile=resnet.cntk configName=resnet deviceId=0 minibatchSize=32 epochSize=2 maxEpochs=2

Data: the same as CNTK AlexNet data.

TensorFlow

Source Code: TensorFlowResNet.zip
Usage example: python resnet.py -e 2 -b 32 -i 4 -d 0

Data: random generated in the source code.

Torch

Source Code: TorchResNet.zip
Usage example: th resnet.lua -depth 50 -nGPU 1 -deviceId 0 -batchSize 256 -nEpochs 4 -nIterations 10 -dataset imagenet

Data: random generated in the source code.

Recurrent Neural Networks (RNNs)

LSTM

CNTK

Source Code: CNTKLSTM.zip
Usage example: cntk configFile=lstm.cntk configName=lstm deviceId=0 minibatchSize=256 epochSize=8192 maxEpochs=1

Data: from CNTK.

TensorFlow

Source Code: TensorFlowLSTM.zip
Usage example: python lstm.py --batchsize 256 --iters 10 --seqlen 32 --numlayer 2 --hiddensize 256 --device 0 --data_path ~/data/PennTreeBank

Data: from CNTK.

Torch

Source Code: TorchLSTM.zip
Usage example: th lstm.lua --seqlen 32 --batchsize 256 --iters 10 --hiddensize {256,256} --cuda --lstm --startlr 1 --cutoff 5 --maxepoch 1 --device 0

Data: generated by the source code.

Versions of Softwares

Acknowledgements