How to downsize a large pre-trained CNN model to a smaller size without losing performance?

More Vidit Kumar's questions See All

Which is the most efficient etching solution for nickel based alloys?

Which solution reveals the microstructure and features of nickel-based alloys?

02 March 2021 2,221 3 View

Which is the most convenient manufacturing software for Additive Manufacturing?

Which tools can be used to generate machine codes for additive manufacturing?

02 March 2021 7,220 2 View

Which is the best place in the world for the plant biotechnology research?

I am searching for a good place for the Post Doc,

02 March 2021 4,053 3 View

Which SAR dataset to use for generating coherency matrix and performing target decomposition (Description) ?

Working on chandrayaan-2 DFSAR data, there are three datasets available: 1) Slant range image data product: The slant range complex image file. Each pixel is represented by two 4-byte floating...

02 March 2021 8,481 3 View

How does one access the Cranet survey data on HRM practices?

Hello, The Cranet website does not provide any lead to the data obtained from their multinational survey on HRM practices. Any help on accessing the data would be most appreciated. Thank...

01 March 2021 9,536 3 View

Genome-wide gene identification protocol ?

Is there any book chapter/book, webpage or research article available to understand genome-wide gene identification?

28 February 2021 8,095 1 View

Can the problems of overfitting be solved without cross validation?

For an overfit data set the cross validation error will be much bigger than the training error. Is it true or false?

23 February 2021 3,616 2 View

Research Gate Does it help researches to publicize their scientific work ?

In my opinion, the RG site is useful in solving scientific problems and shows the researcher's ability. Besides, the separation of the H-index from the self-citations and the overall index is an...

22 February 2021 3,894 6 View

How to Study a proton NMR of Graphene prepared by Electrochemical Exfoliation technique?

I performed a Proton NMR of graphene powder prepared by electrochemical exfoliation technique. It shows some peaks. But I can't understand it.

21 February 2021 8,947 3 View

How to classify and recognize unseen image class in document sorting?

To sort the scanned document image I have to determine the class of document. But no of classes of image will keep on adding even after training. So at time of inference first I will detect unseen...

16 February 2021 4,501 3 View

Why should we choose deep learning to machine learning? Why is CNN better for image classification? Can I get Typical Answer?

What Characteristics makes CNN work better?

03 March 2021 1,458 4 View

What are some of the research gaps in machine learning and artificial intelligence in africa?

i would to know some of the research gaps in the artificial intelligence field in most african countries.

03 March 2021 6,145 3 View

Anyone plz suggest relevant topic for research in image processing+Deep learning.?

I have selected brain tumor images ...but now found that already lots of research done n this topic.

03 March 2021 5,774 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

Can you share me how teacher education is managed in your respective national educational system?

I feel that the practice in teacher education in my country is below the expected performance level due to very poor management system. Hope I will learn something from your experiences.

02 March 2021 1,516 4 View

Can NFL theorem be valid in infinite search space in ML?

NFL theorem is valid for algorithms training in fixed training set. However, the general characteristic of algorithms in expanded or open dataset has not been proved yet. Could you show your...

01 March 2021 1,189 3 View

What is the difference between L1 and L2 regularization in machine learning?

L1 and L2 regularization

28 February 2021 4,187 3 View

How to eliminate Steady state error from Reinforcement learning genetic algorithm code?

The following code (see 1st 2 images attached) is used to produce PID controller values that are designed to control the system (G). The code finds the PID controller values (noted as k) by using...

28 February 2021 6,560 14 View

Is there any Machine Learning/AI algorithm that lets you predict with incomplete features without missing value imputation process?

For instance, the model is trained with A,B,C,D,E features to predict F target. Then, I want to make prediction on a new data, but with only A,C,D,E is known without doing missing value imputation...

28 February 2021 1,556 6 View

How can I represent SDS-PAGE results for machine learning analysis?

I am trying to classify and analyze the results of an SDS-PAGE based array for bacterial detection using machine learning, but I have trouble finding the best way to represent the results with...

27 February 2021 9,176 3 View

Sheikh Md. Razibul Hasan Raj

Follow the link

https://medium.com/analytics-vidhya/reducing-deep-learning-model-size-without-effecting-its-original-performance-and-accuracy-with-a809b49cf519

Daniel Coelho

You should also check Knowledge Distillation.

Saif ul Islam

You can use the following techniques:

Network Pruning: Pruning involves removing less important weights or neurons from the network, resulting in a smaller network size with similar performance.

Quantization: Converting the network's parameters and activations from floating-point numbers to lower-precision fixed-point numbers.

Factorization: Decomposing the weight tensors in the network into smaller tensors and reducing the number of parameters.

Knowledge Distillation: Transferring knowledge from a large pre-trained model to a smaller model, where the large model acts as a teacher and the smaller model as a student.

Model Architecture: Choosing a smaller network architecture with fewer parameters, such as MobileNet, ShuffleNet, etc.

Janak Trivedi

Downsizing a pre-trained CNN model to a smaller size without losing performance can be achieved by several techniques, including:

Pruning: This involves removing neurons and connections in the model that have the least impact on performance. This can reduce the size of the model without sacrificing accuracy.

Quantization: This involves converting the weights and activations in the model from floating-point to integer representations. This reduces the size of the model and can also improve the performance of the model on hardware with limited memory.

Low-rank approximation: This involves approximating the weight matrices in the model with lower-dimensional matrices. This reduces the size of the model without sacrificing accuracy.

Architecture search: This involves using algorithms to find the optimal architecture for a given dataset and computational budget. This can result in a smaller model with improved accuracy.

Transfer learning: This involves using a pre-trained model as a starting point and fine-tuning it on a smaller dataset. This can result in a smaller model with improved accuracy that is specifically tailored to the new dataset.

It's important to note that downsizing a pre-trained model may come with trade-offs in terms of performance, and the best approach will depend on the specific requirements of the application. It may also require multiple iterations of fine-tuning and experimentation to achieve the desired balance between size and accuracy.

Yi Jie Wong

This question can be answered in two parts. Firstly, (A) DOWNSIZING PRETRAINED MODEL is different compared to (B) CREATE SMALLER CNN THAT MIMIC LARGER ONE. (A) can be done by pruning or quantization, while (B) can be done by knowledge distillation.

For (A):

Pruning: This removes less important weights or neurons from the CNN (or any types of AI model), to reduce the size of the model without sacrificing accuracy. However, pruning does not translate to higher inference speed.

Quantization: This involves converting the weights in the model from floating-point FP32 to FP16 or integer (INT8) representations. This can reduce the size of the model, and could speed up the model. For instance, FP16 model (may be) faster than FP16 in GPU, while INT8 model is faster than FP32 and FP16 in CPU.

For (B):

Knowledge Distillation: This is a process to transfer knowledge from a large pretrained model to a smaller model. The larger pretrained model acts as a teacher, while the smaller model acts as the student.

In practice, method (A) is easier to implement. There are various tools like Intel's OpenVINO and Google's TensorFlow Lite that can perform pruning and quantization automatically with little to no effort. You only need to pass the pretrained model to any of the toolkits, to downsize the model. On the other hand, method (B) requires more efforts. You can search the limitation of knowledge distillation in the internet.

Vidit Kumar

thanks Yi Jie Wong Janak Trivedi Saif ul Islam Daniel Coelho Sheikh Md. Razibul Hasan Raj for sharing useful information.