I am referring to this study https://proceedings.neurips.cc/paper/2020/file/288cd2567953f06e460a33951f55daaf-Paper.pdf entitled "On Warm-Starting Neural Network Training". Here, the authors propose a technique to take a pretrained model and shrink its weights by multiplying the weights by 0.5 and then add a Gaussian noise with sigma of 0.01. I am trying to achieve this using a Keras custom function.
def shrink_perturb(model, lamda=0.5,sigma=0.01):
model.get_weights()
#multiply model weights by 0.5, add gaussian noise with sigma = 0.01
new_model =
return new_model
#apply to a pretrained model
model = load_model('weights/model.h5')
shrunk_model = shrink_perturb(model,lamda=0.5,sigma=0.01)
shrunk_model.summary()
I need support in completing the custom Keras function. Thanks.
Hi, I believe your problem can be solved by the following code. I have attached the dummy python files for your reference as well. Hope it helps
############################################
import tensorflow as tf
import numpy as np
import copy
def shrink_perturb(model):
# get model weight
weight = model.get_weights()
# loop thru weights of each layer
for i, w, in enumerate(weight):
# edit the noise based on your configuration
noise = np.random.normal(loc=0.0, scale=0.01, size=w.shape)
new_w = w * 0.5 + noise
weight[i] = new_w
# set the new weight to the model
model.set_weights(weight)
return model
# note that even if you do not return model
# the model in the main code has been updated with the new weights
# this is because list/array in python is passed by reference
# create a dummy model
model = tf.keras.applications.mobilenet_v2.MobileNetV2(
include_top=True,
weights='imagenet', classes=1000, classifier_activation='softmax',
)
# get original weight of the model, just for comparison purpose
ori_weight = copy.deepcopy(model.weights)
# shrinking and perturbing model weights
new_model = shrink_perturb(model)
# validate the changes in weight
print(ori_weight[-1][:10])
print(model.weights[-1][:10]) # the weight here has been updated, since list/array is passed by reference
print(new_model.weights[-1][:10])
############################################
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts