Keras Benchmarks¶
Run the Keras benchmarks under hpvm/benchmarks/dnn_benchmarks/keras
Download CNN Model Files¶
Prior to running the benchmarks, ensure you download the CNN model data (inputs and weights) if not done in automatic build script.
wget https://databank.illinois.edu/datafiles/o3izd/download -O model_params.tar.gz
tar -xf model_params.tar.gz
Move extracted model_params
directory to /benchmarks/dnn_benchmarks/model_params
(Benchmarks expect data at this location)
Running Benchmaks¶
List of benchmarks and the expected accuracies:
Benchmark |
Accuracy |
---|---|
alexnet.py |
79.28 |
alexnet2.py |
84.98 |
alexnet_imagenet.py |
56.30 |
lenet.py |
98.70 |
mobilenet_cifar10.py |
84.42 |
resnet18_cifar10.py |
89.56 |
resnet50_imagenet.py |
75.10 |
vgg16_cifar10.py |
89.96 |
vgg16_cifar100.py |
66.50 |
vgg16_imagenet.py |
69.46 |
Synopsis¶
python3 ${BENCH_NAME}.py [hpvm_reload|keras_reload] [frontend] [compile]
Command-line Parameters
hpvm_reload
: Reloads HPVM weights (.bin
binary format used by HPVM weights - present in model_params
download directory) from directory path specified in the reload_dir
parameter set in code - this is described in “Parameters to Change in Code” (below).
keras_reload
: Alternatively, reload weights in Keras .h5
file format with path to file specified in keras_model_file
described in “Parameters to Change in Code” (below).
frontend
: Invokes the HPVM frontend and dumps weights (in HPVM .bin
format) in the output directory specified. The parameters that control where data and source files are dumped are specified by parameters data_dir
and src_dir
, respectively. These are described below.
compile
: Optional Parameter. When specified, it compiles the HPVM-C code generated by the frontend into an HPVM binary under the directory specified by src_dir
(described below). If src_dir
path exists, a unique directory (which appends a unique ID) is created.
The binary is built with the name HPVM_binary
.
NOTE: Before running HPVM_binary
necessary to set CUDA and CUDNN paths with:
source ${PATH_TO_YOUR_HPVM_ROOT}/hpvm/set_paths.sh
Parameters to Change in Code
The AlexNet source is commented with explanations on how to use the Keras frontend interface. AlexNet source is here.
NAME
: Benchmark Name - Can be set to any desired valuereload_dir
: Path to directory from where to reload weights in HPVM format. This directory is used to reload weights ifhpvm_reload
command-line option is used.keras_model_file
: Path to Keras .h5 model file to reload weigths from. Either ofreload_dir
orkeras_model_file
can be used.keras_model_file
is used whenkeras_reload
commad-line parameter is used with the Benchmark script.data_dir
: Directory to dump weights specified specified in constructor.src_dir
: Directory to dump ApproxHPVM sources in HPVM-C (C with HPVM compiler intrinsics) specified in constructor.num_classes
: number of output classes - dependent on the dataset used. For CIFAR10,num_classes
is 10, CIFAR100 has 100 classes, for ImageNet, number of classes is 1000.batch_size
: This parameter controls the size of each batch that is processed in HPVM. The batch size should be kept as large as the GPU memory can support. This parameter should be adapted according to the memory size of the deployed device.
Using the Frontend with Custom (New) Benchmarks¶
Any new benchmarks must inherit from the commom parent Benchmark
class
and override the virtual functions for building the model, training,
and data preprocessing. These methods are described below:
def buildModel(self)
:
Constructs and returns a keras model
def data_preprocess(self)
:
returns X_train, y_train, X_test, y_test, X_tuner, and y_tuner data (in that order):
These are described here:
X_train:
Training data (fp32) in NCHW formaty_train:
Training labels (int32)X_test:
Testing/Evaluation data in NCHW formaty_test:
Testing/Evaluation labelsX_tuner:
Data to be used for autotuningy_tuner:
Labels corresponding to tuning data
def trainModel(self, model, X_train, y_train, X_test, y_test)
:
Trains the Keras model constructed in buildModel
and is expected to return the
trained keras model - training parameters should be tuned here.
Directly using Keras Frontend API¶
Alternate to extending the Benchmark
class, users may directly invoke the Keras Frontend API. This can be done as:
from keras_frontend.approxhpvm_translator import translate_to_approxhpvm
# Construct and train your Keras Model (or load pre-trained weights)
translate_to_approxhpvm(model, data_dir, src_dir, test_data, test_labels, tune_data, tune_labels, batch_size, num_classes)
Running HPVM Binary¶
Run the HPVM_binary
generated under the directory specified by src_dir
(described above). Usage:
./HPVM_binary -t {test|tune} -c ${config_file_path}
test|tune
: Runs with either tune (autotuning data) or test set (for evaluation)
config_file_path
: Path to an HPVM tensor configuration file (includes approximation settings)
NOTE: The accuracy of the bennchmarks is dumped into a file named final_accuracy
in the current working directory - this includes accuracy averaged across batches
Automated Tests¶
/hpvm/hpvm/projects/keras/scripts/test_benchmarks.py
is an automated test script that evaluates the accuracy of each Benchmark in Keras and HPVM (after comilation using HPVM Compiler) and compares the accuracy of each binary to the known correct accuracy. Run from root of hpvm/benchmarks/dnn_benchmarks/keras
:
python3 test_benchmarks.py