Navigation :

GTDLBench

With its effective and wide real applications, deep learning attracts great attention nowadays. Many deep learning frameworks emerged to facilitate deep learning research, development and applications. However, few research has been conducted to effectively measure these frameworks. And it still remains a critical task on choosing the optimal framework for specific deep learning models and applications. GTDLBench aims at introducing a hybrid benchmark to systematically measure the performance of these deep learning frameworks as well as the performance of different hardware devices. The performance metrics derived from both deep learning and systems field are introduced to present an easy and holistic approach for deep learning benchmarking.

Metrics

End-to-end Benchmark

Training Time (s)
Testing Time (s)
Accuracy (%)

Micro Benchmark

Latency (ms)

Tutorials

Deep Learning Frameworks

Frameworks

Datasets

Deep Learning Models

Please cite the following papers:

Papers under GTDLBench

Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond

A Comparative Measurement Study of Deep Learning as a Service Framework

Experimental Characterizations and Analysis of Deep Learning Frameworks

@INPROCEEDINGS{GTDLBenchICDCS, 
    author={{Liu}, Ling and {Wu}, Yanzhao and {Wei}, Wenqi and {Cao}, Wenqi and {Sahin}, Semih and {Zhang}, Qi}, 
    booktitle={2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)}, 
    title="{Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond}",
    year={2018},
    pages={1258-1269}, 
    doi={10.1109/ICDCS.2018.00125}, 
    ISSN={2575-8411}, 
    month={July},
}

@ARTICLE{GTDLBencharXiv,
    author = {{Wu}, Yanzhao and {Liu}, Ling and {Pu}, Calton and {Cao}, Wenqi and
    {Sahin}, Semih and {Wei}, Wenqi and {Zhang}, Qi},
    title = "{A Comparative Measurement Study of Deep Learning as a Service Framework}",
    journal = {arXiv e-prints},
    keywords = {Computer Science - Performance, Computer Science - Machine Learning},
    year = 2018,
    month = Oct,
    eid = {arXiv:1810.12210},
    pages = {arXiv:1810.12210},
    archivePrefix = {arXiv},
    eprint = {1810.12210},
    primaryClass = {cs.PF},
    adsurl = {https://ui.adsabs.harvard.edu/\#abs/2018arXiv181012210W},
    adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@INPROCEEDINGS{GTDLBenchBigData, 
    author={{Wu}, Yanzhao and and {Cao}, Wenqi and {Sahin}, Semih and {Liu}, Ling}, 
    booktitle={2018 IEEE 38th International Conference on Big Data}, 
    title="{Experimental Characterizations and Analysis of Deep Learning Frameworks}", 
    year={2018},
    month={December},
}