Welcome again to the Tiny Big sequence — a sequence the place I share what I realized about MobileNet architectures. Previously two articles I coated MobileNetV1 and MobileNetV2. Try references [1] and [2] if you happen to’re involved in studying them. In at the moment’s article I wish to proceed with the subsequent model of the mannequin: MobileNetV3.
MobileNetV3 was first proposed in a paper titled “Trying to find MobileNetV3” written by Howard et al. in 2019 [3]. Only a fast evaluation: the principle thought of the primary MobileNet model was changing full-convolutions with depthwise separable convolutions, which diminished the variety of params by almost 90% in comparison with its customary CNN counterpart. Within the second MobileNet model, the authors launched the so-called inverted residual and linear bottleneck mechanisms, which they built-in into the unique MobileNetV1 constructing blocks. Now within the third MobileNet model, the authors tried to push the efficiency of the community even additional by incorporating Squeeze-and-Excitation (SE) modules and onerous activation features into the constructing blocks. Moreover, the general construction of MobileNetV3 itself is partially designed utilizing NAS (Neural Structure Search), by which it primarily works considerably like a parameter tuning that operates on the architectural stage by maximizing accuracy whereas minimizing latency. Nonetheless, word that on this article I received’t go into how NAS works intimately. As an alternative, I’ll give attention to the ultimate design of MobileNetV3 proposed within the paper.
The Detailed MobileNetV3 Structure
The authors suggest two variants of this mannequin which they discuss with as MobileNetV3-Giant and MobileNetV3-Small. You possibly can see the small print of the 2 architectures in Determine 1 beneath.
Taking a more in-depth have a look at the structure, we will see that the 2 networks primarily encompass bneck (bottleneck) blocks. The configuration of the blocks themselves is described in columns exp measurement, #out, SE, NL, and s. The interior construction of those blocks in addition to the corresponding parameter configurations shall be mentioned additional within the following subsection.
The Bottleneck
MobileNetV3 makes use of the modified model of the constructing blocks utilized in MobileNetV2. As I’ve talked about earlier, what makes the 2 completely different is the presence of SE module and the usage of onerous activation perform. You possibly can see the 2 constructing blocks in Determine 2, with MobileNetV2 on the high and MobileNetV3 on the backside.
Discover that the primary two convolution layers in each constructing blocks are principally the identical: a pointwise convolution adopted by a depthwise convolution. The previous is used for increasing the variety of channels to exp measurement (enlargement measurement), whereas the latter is accountable to course of every channel of the ensuing tensor independently. The one distinction between the 2 constructing blocks lies within the activation features used, which they discuss with as NL (Nonlinearity). In MobileNetV2, the activation features positioned after the 2 convolution layers are set fastened to ReLU6, whereas in MobileNetV3 it might both be ReLU6 or hard-swish. The RE and HS you noticed earlier in Determine 1 principally refer to those two sorts of activations.
Subsequent, in MobileNetV3 we place the SE module after the depthwise convolution layer. In case you’re not but conversant in SE module, it’s primarily a form of constructing block we will connect in any form of CNN-based mannequin. This element is beneficial for giving weights to completely different channels, permitting the mannequin to pay extra consideration to the vital channels solely. I even have a separate article discussing the SE module intimately. Click on on the hyperlink at reference quantity [4] if you wish to learn that one. You will need to word that the SE module used right here is barely completely different, in that the final FC layer makes use of hard-sigmoid moderately than the usual sigmoid activation perform. (I’ll discuss extra concerning the onerous activations utilized in MobileNetV3 later within the subsequent subsection.) Actually, the SE module itself is just not all the time included in each bottleneck block. In case you return to Determine 1, you’ll discover that a few of the bottleneck blocks have a checkmark within the SE column, indicating that the SE module is utilized. However, some blocks don’t embrace the module, which could most likely be as a result of the NAS course of didn’t discover any efficiency enchancment from utilizing SE modules in these blocks.
Because the SE module has been related, we have to place one other pointwise convolution, which is accountable to regulate the variety of output channels in response to the #out column in Determine 1. This pointwise convolution doesn’t embrace any activation perform, aligning with the linear bottleneck design initially launched in MobileNetV2. I truly must make clear one thing right here. In case you check out the MobileNetV2 constructing block in Determine 2 above, you’ll discover that the final pointwise convolution has a ReLU6 positioned on it. I consider it is a mistake made by the authors, as a result of in response to the MobileNetV2 paper [6], the ReLU6 ought to be within the first pointwise convolution at first of the block as an alternative.
Final however not least, discover that there’s additionally a residual connection that skips throughout all layers within the bottleneck block. This connection is barely current when the output tensor has the very same dimensions because the enter, i.e., when the variety of enter and output channels is similar and when the s (stride) is 1.
Onerous-Sigmoid and Onerous-Swish
The activation features utilized in MobileNetV3 aren’t generally present in different deep studying fashions. To start out with, let’s have a look at the hard-sigmoid activation first, which is the one used within the SE module as a alternative for the traditional sigmoid. Check out Determine 3 beneath to see the distinction between the 2.
Right here you may most likely be questioning, why don’t we simply use the traditional sigmoid? Why do we actually want to make use of piecewise linear perform that seems much less clean as an alternative? To reply this query, we have to perceive the mathematical definition of a sigmoid perform prematurely, which I present in Determine 4 beneath.
We will clearly see within the above determine that the sigmoid perform initially includes an exponential time period within the denominator. Actually, this time period causes the perform to be computationally costly, which in flip makes the activation perform much less appropriate for low-power gadgets. Not solely that, the output of the sigmoid perform itself is a high-precision floating-point worth, which can also be not preferable for low-power gadgets as a consequence of their restricted assist for dealing with such values.
In case you have a look at Determine 3 once more, you may suppose that the hard-sigmoid perform is straight derived from the unique sigmoid. Actually, that’s truly not fairly proper. Regardless of having an identical form, hard-sigmoid is principally constructed utilizing ReLU6 as an alternative, which may formally be expressed in Determine 5 beneath. Right here you possibly can see that the equation is way easier because it solely consists of fundamental arithmetic operations and clipping, permitting it to be processed a lot sooner.
The following activation perform we’re going to make the most of in MobileNetV3 is the so-called hard-swish, which shall be applied after every of the primary two convolution layers within the bottleneck block. Similar to sigmoid and hard-sigmoid, the graph of the hard-swish perform seems to be just like the unique one.
The unique swish perform itself can mathematically be expressed within the equation in Determine 7. Once more, because the equation includes sigmoid, it’s going to undoubtedly decelerate the computation. Therefore, to hurry up the method, we will merely exchange the sigmoid perform with hard-sigmoid we simply mentioned. By doing so, we now have the onerous model of the swish activation perform as proven in Determine 8.
Some Experimental Outcomes
Earlier than we get into the experimental outcomes, you could know that there are two parameters in MobileNetV3 that enable us to regulate the mannequin measurement in response to our wants. These two parameters are width multiplier and enter decision, which in MobileNetV1 are generally known as α and ρ, respectively. Though we will technically modify the worth for the 2 freely, the authors already offered a number of numbers we will use. For the width multiplier, we will set it to both 0.35, 0.5, 0.75, 1.0, or 1.25, the place utilizing a price smaller than 1.0 causes the mannequin to have fewer variety of channels than these disclosed in Determine 1, successfully decreasing the mannequin measurement. As an example, if we set this parameter to 0.35, then the mannequin will solely have 35% of its default width (i.e., channel depend) all through your entire community.
In the meantime, the enter decision can both be 96, 128, 160, 192, 224, or 256, which because the identify suggests, it straight controls the spatial dimension of the enter picture. It’s price noting that regardless that utilizing a small enter measurement reduces the variety of operations throughout inference, it doesn’t have an effect on the mannequin measurement in any respect. So, in case your goal is to scale back mannequin measurement, you could modify the width multiplier, whereas in case your objective is to decrease computational value, you possibly can mess around with each the width multiplier and enter decision.
Now trying on the experimental leads to Determine 9, we will clearly see that MobileNetV3 outperforms MobileNetV2 by way of accuracy at comparable latency. The MobileNetV3-Small of default configuration (i.e., width multiplier 1.0 and enter decision 224×224) certainly has a decrease accuracy than the most important MobileNetV2 variant. However if you happen to take the default MobileNetV3-Giant into consideration, it received a straightforward win over the most important MobileNetV2 each by way of accuracy and latency. Moreover, we will nonetheless push the accuracy of MobileNetV3 even additional by enlarging the mannequin measurement by 1.25 instances (the blue datapoint on the high proper), however remember the fact that doing so considerably sacrifices computational velocity.
The authors additionally carried out a comparative evaluation with different light-weight fashions, of which the outcomes are proven within the desk in Determine 10.
The rows of the desk above are divided into two teams, the place the higher group is used to match fashions with complexity just like MobileNetV3-Giant, whereas the decrease group consists of fashions akin to MobileNetV3-Small. Right here you possibly can see that each V3-Giant and V3-Small obtained one of the best accuracy on ImageNet inside their respective teams. It’s price noting that though MnasNet-A1 and V3-Giant have the very same accuracy, the variety of operations (MAdds) of the previous mannequin is larger, which ends up in larger latency, as seen in columns P-1, P-2, and P-3 (measured in milliseconds). In case you’re questioning, the labels P-1, P-2, and P-3 primarily correspond to completely different Google Pixel sequence used to check the precise computational velocity. Subsequent, it’s essential to acknowledge that each MobileNetV3 variants have the best parameter depend (the params column) in comparison with different fashions of their group. Nonetheless, this doesn’t appear to be a serious concern for the authors as the first objective of MobileNetV3 is to attenuate computational latency, even when meaning having a barely larger mannequin.
The following experiment the authors carried out was concerning the results of worth quantization, i.e., a method that reduces the precision of floating-point numbers to hurry up computation. Whereas the networks already incorporate onerous activation features, that are appropriate with quantized values, this experiment takes quantization a step additional by making use of it to your entire community to see how a lot the velocity improves. The experimental outcomes when worth quantization was utilized are proven in Determine 11 beneath.
In case you examine the outcomes of V2 and V3 in Determine 11 with the corresponding fashions in Determine 10, you’ll discover that there’s a lower in latency, proving that the usage of low-precision numbers does enhance computational velocity. Nonetheless, it is very important remember the fact that this additionally results in a lower in accuracy.
MobileNetV3 Implementation
I feel all the reasons above cowl just about all the pieces you could know concerning the concept behind MobileNetV3. Now on this part I’m going to convey you into essentially the most enjoyable a part of this text: implementing MobileNetV3 from scratch.
As all the time, the very very first thing we do is importing the required modules.
# Codeblock 1
import torch
import torch.nn as nnAfterwards, we have to initialize the configurable parameters of the mannequin, particularly WIDTH_MULTIPLIER, INPUT_RESOLUTION, and NUM_CLASSES, as proven in Codeblock 2 beneath. I consider the primary two variables are simple as I’ve defined them completely within the earlier part. Right here I made a decision to assign default values for the 2. You possibly can undoubtedly change these numbers based mostly on the values offered within the paper if you wish to modify the complexity of the mannequin. Subsequent, the third variable corresponds to the variety of output neurons within the classification head. Right here I set it to 1000 as a result of the mannequin is initially educated on the ImageNet-1K dataset. It’s price noting that the MobileNetV3 structure is definitely not restricted to classification duties solely. As an alternative, it can be used for object detection and semantic segmentation as demonstrated within the paper. Nonetheless, because the focus of this text is to implement the spine, let’s simply use the usual classification head for the output layer to maintain issues easy.
# Codeblock 2
WIDTH_MULTIPLIER = 1.0
INPUT_RESOLUTION = 224
NUM_CLASSES = 1000What we’re going to do subsequent is to wrap the repeating parts into separate lessons. By doing this, we’ll later be capable to merely instantiate them at any time when wanted as an alternative of rewriting the identical code again and again. Now let’s start with the Squeeze-and-Excitation module first.
The Squeeze-and-Excitation Module
The implementation of this element is proven in Codeblock 3. I’m not going to get very deep into the code since it’s nearly precisely the identical because the one in my earlier article [4]. Nonetheless, usually talking, this code works by representing every enter channel with a single quantity (line #(1)), processing the ensuing vector with a sequence of linear layers (#(2–3)), then changing it right into a weight vector (#(4)). Take into account that within the unique SE module we usually use the usual sigmoid activation perform to acquire the load vector, however right here in MobileNetV3 we use hard-sigmoid as an alternative. This weight vector will then be multiplied with the unique tensor, which by doing so we will scale back the affect of channels that don’t give contribution to the ultimate output (#(5)).
# Codeblock 3
class SEModule(nn.Module):
def __init__(self, num_channels, r):
tremendous().__init__()
self.global_pooling = nn.AdaptiveAvgPool2d(output_size=(1,1))
self.fc0 = nn.Linear(in_features=num_channels,
out_features=num_channels//r,
bias=False)
self.relu6 = nn.ReLU6()
self.fc1 = nn.Linear(in_features=num_channels//r,
out_features=num_channels,
bias=False)
self.hardsigmoid = nn.Hardsigmoid()
def ahead(self, x):
print(f'originaltt: {x.measurement()}')
squeezed = self.global_pooling(x) #(1)
print(f'after avgpooltt: {squeezed.measurement()}')
squeezed = torch.flatten(squeezed, 1)
print(f'after flattentt: {squeezed.measurement()}')
excited = self.fc0(squeezed) #(2)
print(f'after fc0tt: {excited.measurement()}')
excited = self.relu6(excited)
print(f'after relu6tt: {excited.measurement()}')
excited = self.fc1(excited) #(3)
print(f'after fc1tt: {excited.measurement()}')
excited = self.hardsigmoid(excited) #(4)
print(f'after hardsigmoidt: {excited.measurement()}')
excited = excited[:, :, None, None]
print(f'after reshapett: {excited.measurement()}')
scaled = x * excited #(5)
print(f'after scalingtt: {scaled.measurement()}')
return scaledNow let’s examine if the above code works correctly by creating an SEModule occasion and passing a dummy tensor via it. See Codeblock 4 beneath for the small print. Right here I configure the SE module to simply accept a 512-channel picture for the enter. In the meantime, the r (discount ratio) parameter is about to 4, which means that the vector size between the 2 FC layers goes to be 4 instances smaller than that of its enter and output. It is perhaps price realizing that this quantity is completely different from the one talked about within the unique Squeeze-and-Excitation paper [7], the place r = 16 is claimed to be the candy spot for balancing accuracy and complexity.
# Codeblock 4
semodule = SEModule(num_channels=512, r=4)
x = torch.randn(1, 512, 28, 28)
out = semodule(x)If the code above produces the next output, it confirms that our SE module implementation is right because it efficiently handed the enter tensor via all layers inside the whole SE module.
# Codeblock 4 Output
unique : torch.Dimension([1, 512, 28, 28])
after avgpool : torch.Dimension([1, 512, 1, 1])
after flatten : torch.Dimension([1, 512])
after fc0 : torch.Dimension([1, 128])
after relu6 : torch.Dimension([1, 128])
after fc1 : torch.Dimension([1, 512])
after hardsigmoid : torch.Dimension([1, 512])
after reshape : torch.Dimension([1, 512, 1, 1])
after scaling : torch.Dimension([1, 512, 28, 28])The Convolution Block
The following element I’m going to create is the one wrapped within the ConvBlock class, which the detailed implementation might be seen in Codeblock 5. Actually, that is truly simply a normal convolution layer, however we don’t merely use nn.Conv2d as a result of in CNN we usually use the Conv-BN-ReLU construction. Therefore, it will likely be handy if we simply group these three layers collectively inside a single class. Nonetheless, as an alternative of really following this customary construction, we’re going to customise it to match the necessities for the MobileNetV3 structure.
# Codeblock 5
class ConvBlock(nn.Module):
def __init__(self,
in_channels, #(1)
out_channels, #(2)
kernel_size, #(3)
stride, #(4)
padding, #(5)
teams=1, #(6)
batchnorm=True, #(7)
activation=nn.ReLU6()): #(8)
tremendous().__init__()
bias = False if batchnorm else True #(9)
self.conv = nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
padding=padding,
teams=teams,
bias=bias)
self.bn = nn.BatchNorm2d(num_features=out_channels) if batchnorm else nn.Id() #(10)
self.activation = activation
def ahead(self, x): #(11)
print(f'originaltt: {x.measurement()}')
x = self.conv(x)
print(f'after convtt: {x.measurement()}')
x = self.bn(x)
print(f'after bntt: {x.measurement()}')
x = self.activation(x)
print(f'after activationt: {x.measurement()}')
return xThere are a number of parameters you could cross to instantiate a ConvBlock occasion. The primary 5 ones (#(1–5)) are fairly simple as they’re principally simply the usual parameters for the nn.Conv2d layer. Right here I set the teams parameter to be configurable (#(6)) in order that this class might be flexibly used not just for customary convolutions but in addition for depthwise convolutions. Subsequent, at line #(7) I create a parameter referred to as batchnorm, which determines whether or not or not a ConvBlock occasion implements a batch normalization layer. That is primarily executed as a result of there are some circumstances the place we don’t implement this layer, i.e., within the final two convolutions with NBN label (which stands for no batch normalization) in Determine 1. The final parameter we’ve got right here is the activation perform (#(8)). Afterward, there shall be circumstances that require us to set it to both nn.ReLU6(), nn.Hardswish() or nn.Id() (no activation).
Contained in the __init__() methodology, there are two issues occurring if we modify the enter argument for the batchnorm parameter. Once we set it to True, firstly, the bias time period of the convolution layer shall be deactivated (#(9)), and secondly, bn shall be an nn.BatchNorm2d() layer (#(10)). The bias time period won’t be used on this case as a result of making use of batch normalization after convolution will cancel it out. So, there may be principally no level of using bias within the first place. In the meantime, if we set the batchnorm parameter to False, the bias variable goes to be True since on this state of affairs it won’t be canceled out. The bn itself will simply be an id layer, which means that it received’t do something to the tensor.
Relating to the ahead() methodology (#(11)), I don’t suppose I want to clarify something as a result of what we do right here is simply passing a tensor via the layers sequentially. Now let’s simply transfer on to Codeblock 6 to see whether or not our ConvBlock implementation is right. Right here I attempt to create two ConvBlock situations, the place the primary one makes use of default batchnorm and activation, whereas the second omits the batch normalization layer (#(1)) and makes use of hard-swish activation perform (#(2)). As an alternative of passing a tensor via them, right here I would like you to see within the ensuing output that our code appropriately implements each buildings in response to the enter arguments we cross.
# Codeblock 6
convblock1 = ConvBlock(in_channels=64,
out_channels=128,
kernel_size=3,
stride=2,
padding=1)
convblock2 = ConvBlock(in_channels=64,
out_channels=128,
kernel_size=3,
stride=2,
padding=1,
batchnorm=False, #(1)
activation=nn.Hardswish()) #(2)
print(convblock1)
print('')
print(convblock2)# Codeblock 6 Output
ConvBlock(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): ReLU6()
)
ConvBlock(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(bn): Id()
(activation): Hardswish()
)The Bottleneck
Because the SEModule and the ConvBlock are executed, we will now transfer on to the principle element of the MobileNetV3 structure: the bottleneck. What we primarily do within the bottleneck is simply inserting one layer after one other which the overall construction is proven earlier in Determine 2. Within the case of MobileNetV2, it solely consists of three convolution layers, whereas right here in MobileNetV3 we’ve got an extra SE block positioned between the second and the third convolutions. Have a look at Codeblock 7a and 7b to see how I implement the bottleneck block for MobileNetV3.
# Codeblock 7a
class Bottleneck(nn.Module):
def __init__(self,
in_channels,
out_channels,
kernel_size,
stride,
padding,
exp_size, #(1)
se, #(2)
activation):
tremendous().__init__()
self.add = in_channels == out_channels and stride == 1 #(3)
self.conv0 = ConvBlock(in_channels=in_channels, #(4)
out_channels=exp_size, #(5)
kernel_size=1, #(6)
stride=1,
padding=0,
activation=activation)
self.conv1 = ConvBlock(in_channels=exp_size, #(7)
out_channels=exp_size, #(8)
kernel_size=kernel_size, #(9)
stride=stride,
padding=padding,
teams=exp_size, #(10)
activation=activation)
self.semodule = SEModule(num_channels=exp_size, r=4) if se else nn.Id() #(11)
self.conv2 = ConvBlock(in_channels=exp_size, #(12)
out_channels=out_channels, #(13)
kernel_size=1, #(14)
stride=1,
padding=0,
activation=nn.Id()) #(15)The enter parameters of the Bottleneck class look just like these of the ConvBlock class at a look. This undoubtedly is sensible as a result of we’ll certainly use them to instantiate ConvBlock situations contained in the Bottleneck. Nonetheless, if you happen to take a more in-depth have a look at them once more, you’ll discover that there are another parameters you haven’t seen earlier than, particularly se (#(1)) and exp_size (#(2)). Afterward, the enter arguments for these parameters shall be obtained from the configuration offered within the desk in Determine 1.
Contained in the __init__() methodology, what we have to do first is to examine whether or not the enter and output tensor dimensions are the identical utilizing the code at line #(3). By doing this, we may have our add variable containing both True or False. This dimensionality checking is vital as a result of we have to determine whether or not or not we carry out element-wise summation between the 2 to implement the skip-connection that skips via all layers inside the bottleneck block.
Subsequent, let’s now instantiate the layers themselves, of which the primary two are a pointwise convolution (conv0) and a depthwise convolution (conv1). For conv0, we have to set the kernel measurement to 1×1 (#(6)), whereas for conv1 the kernel measurement ought to match the one within the enter argument (#(9)), which may both be 3×3 or 5×5. It’s essential to use padding within the ConvBlock to stop the picture measurement from shrinking after each convolution operation. For kernel sizes of 1×1, 3×3, and 5×5, the required padding values are 0, 1, and a couple of, respectively. Speaking concerning the variety of channels, conv0 is accountable to broaden it from in_channels to exp_size (#(4–5)). In the meantime, the variety of enter and output channels of conv1 are precisely the identical (#(7–8)). Along with the conv1 layer, the teams parameter ought to be set to exp_size (#(10)) as a result of we wish every enter channel to be processed independently of one another.
After the primary two convolution layers are executed, what we have to instantiate subsequent is the Squeeze-and-Excitation module (#(11)). Right here we have to set the enter channel depend to exp_size, matching with the tensor measurement produced by the conv1 layer. Keep in mind that SE module is just not all the time used, therefore the instantiation of this element ought to be executed inside a situation, the place it’s going to truly be instantiated solely when the se parameter is True. In any other case, it’s going to simply be an id layer.
Lastly, the final convolution layer (conv2) is accountable to map the variety of output channels from exp_size to out_channels (#(12–13)). Similar to the conv0 layer, this one can also be a pointwise convolution, therefore we set the kernel measurement to 1×1 (#(14)) in order that it solely focuses on aggregating data alongside the channel dimension. The activation perform of this layer is about fastened to nn.Id() (#(15)) as a result of right here we’ll implement the concept of linear bottleneck.
And that’s just about all the pieces for the layers inside the bottleneck block. All we have to do afterwards is to create the movement of the community within the ahead() methodology as proven in Codeblock 7b beneath.
# Codeblock 7b
def ahead(self, x):
residual = x
print(f'originaltt: {x.measurement()}')
x = self.conv0(x)
print(f'after conv0tt: {x.measurement()}')
x = self.conv1(x)
print(f'after conv1tt: {x.measurement()}')
x = self.semodule(x)
print(f'after semodulett: {x.measurement()}')
x = self.conv2(x)
print(f'after conv2tt: {x.measurement()}')
if self.add:
x += residual
print(f'after summationtt: {x.measurement()}')
return xNow I wish to take a look at the Bottleneck class we simply created by simulating the third row of the MobileNetV3-Giant structure within the desk in Determine 1. Have a look at the Codeblock 8 beneath to see how I do that. In case you return to the architectural particulars, you’ll discover that this bottleneck accepts a tensor of measurement 16×112×112 (#(7)). On this case, the bottleneck block is configured to broaden the variety of channels to 64 (#(3)) earlier than ultimately shrinking it to 24 (#(1)). The kernel measurement of the depthwise convolution is about to three×3 (#(2)) and the stride is about to 2 (#(4)) which is able to scale back the spatial dimension by half. Right here we use ReLU6 for the activation perform (#(6)) of the primary two convolutions. Lastly, SE module won’t be applied (#(5)) since there isn’t a checkmark within the SE column within the desk.
# Codeblock 8
bottleneck = Bottleneck(in_channels=16,
out_channels=24, #(1)
kernel_size=3, #(2)
exp_size=64, #(3)
stride=2, #(4)
padding=1,
se=False, #(5)
activation=nn.ReLU6()) #(6)
x = torch.randn(1, 16, 112, 112) #(7)
out = bottleneck(x)In case you run the above code, the next output ought to seem in your display.
# Codeblock 8 Output
unique : torch.Dimension([1, 16, 112, 112])
after conv0 : torch.Dimension([1, 64, 112, 112])
after conv1 : torch.Dimension([1, 64, 56, 56])
after semodule : torch.Dimension([1, 64, 56, 56])
after conv2 : torch.Dimension([1, 24, 56, 56])This output confirms that our implementation is right by way of the tensor form, the place the spatial dimension halves from 112×112 to 56×56 whereas the variety of channels appropriately expands from 16 to 64 after which reduces from 64 to 24. Speaking extra particularly concerning the SE module, we will see within the above output that the tensor continues to be handed via this element regardless of we’ve got set the se parameter to False. Actually, if you happen to attempt to print out the detailed structure of this bottleneck like what I do in Codeblock 9, you will notice that semodule is simply an id layer, which successfully makes this construction behave as if we’re passing the output of conv1 on to conv2.
# Codeblock 9
bottleneck# Codeblock 9 Output
Bottleneck(
(conv0): ConvBlock(
(conv): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): ReLU6()
)
(conv1): ConvBlock(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), teams=64, bias=False)
(bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): ReLU6()
)
(semodule): Id()
(conv2): ConvBlock(
(conv): Conv2d(64, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): Id()
)
)The above bottleneck goes to behave in a different way if we instantiate it with the se parameter set to True. In Codeblock 10 beneath, I attempt to create the bottleneck block within the fifth row within the MobileNetV3-Giant structure. On this case, if you happen to print out the detailed construction, you will notice that semodule consists of all layers within the SEModule class we created earlier as an alternative of simply being an id layer like earlier than.
# Codeblock 10
bottleneck = Bottleneck(in_channels=24,
out_channels=40,
kernel_size=5,
exp_size=72,
stride=2,
padding=2,
se=True,
activation=nn.ReLU6())
bottleneck# Codeblock 10 Output
Bottleneck(
(conv0): ConvBlock(
(conv): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(72, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): ReLU6()
)
(conv1): ConvBlock(
(conv): Conv2d(72, 72, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), teams=72, bias=False)
(bn): BatchNorm2d(72, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): ReLU6()
)
(semodule): SEModule(
(global_pooling): AdaptiveAvgPool2d(output_size=(1, 1))
(fc0): Linear(in_features=72, out_features=18, bias=False)
(relu6): ReLU6()
(fc1): Linear(in_features=18, out_features=72, bias=False)
(hardsigmoid): Hardsigmoid()
)
(conv2): ConvBlock(
(conv): Conv2d(72, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(40, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activation): Id()
)
)The Full MobileNetV3
As all parts have been created, what we have to do subsequent is to assemble the principle class of the MobileNetV3 mannequin. However earlier than doing so, I wish to initialize a listing that shops the enter arguments used for instantiating the bottleneck blocks as proven in Codeblock 11 beneath. Take into account that these arguments are written in response to the MobileNetV3-Giant model. You’ll want to regulate the values within the BOTTLENECKS listing if you wish to create the small model as an alternative.
# Codeblock 11
HS = nn.Hardswish()
RE = nn.ReLU6()
BOTTLENECKS = [[16, 16, 3, 16, False, RE, 1, 1],
[16, 24, 3, 64, False, RE, 2, 1],
[24, 24, 3, 72, False, RE, 1, 1],
[24, 40, 5, 72, True, RE, 2, 2],
[40, 40, 5, 120, True, RE, 1, 2],
[40, 40, 5, 120, True, RE, 1, 2],
[40, 80, 3, 240, False, HS, 2, 1],
[80, 80, 3, 200, False, HS, 1, 1],
[80, 80, 3, 184, False, HS, 1, 1],
[80, 80, 3, 184, False, HS, 1, 1],
[80, 112, 3, 480, True, HS, 1, 1],
[112, 112, 3, 672, True, HS, 1, 1],
[112, 160, 5, 672, True, HS, 2, 2],
[160, 160, 5, 960, True, HS, 1, 2],
[160, 160, 5, 960, True, HS, 1, 2]]The arguments listed above are structured within the following order (from left to proper): in channels, out channels, kernel measurement, enlargement measurement, SE, activation, stride, and padding. Take into account that padding is just not explicitly said within the unique desk, however I embrace it right here as a result of it’s required as an enter when instantiating the bottleneck blocks.
Now let’s truly create the MobileNetV3 class. See the code implementation in Codeblocks 12a and 12b beneath.
# Codeblock 12a
class MobileNetV3(nn.Module):
def __init__(self):
tremendous().__init__()
self.first_conv = ConvBlock(in_channels=3, #(1)
out_channels=int(WIDTH_MULTIPLIER*16),
kernel_size=3,
stride=2,
padding=1,
activation=nn.Hardswish())
self.blocks = nn.ModuleList([]) #(2)
for config in BOTTLENECKS: #(3)
in_channels, out_channels, kernel_size, exp_size, se, activation, stride, padding = config
self.blocks.append(Bottleneck(in_channels=int(WIDTH_MULTIPLIER*in_channels),
out_channels=int(WIDTH_MULTIPLIER*out_channels),
kernel_size=kernel_size,
exp_size=int(WIDTH_MULTIPLIER*exp_size),
stride=stride,
padding=padding,
se=se,
activation=activation))
self.second_conv = ConvBlock(in_channels=int(WIDTH_MULTIPLIER*160), #(4)
out_channels=int(WIDTH_MULTIPLIER*960),
kernel_size=1,
stride=1,
padding=0,
activation=nn.Hardswish())
self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1)) #(5)
self.third_conv = ConvBlock(in_channels=int(WIDTH_MULTIPLIER*960), #(6)
out_channels=int(WIDTH_MULTIPLIER*1280),
kernel_size=1,
stride=1,
padding=0,
batchnorm=False,
activation=nn.Hardswish())
self.dropout = nn.Dropout(p=0.8) #(7)
self.output = ConvBlock(in_channels=int(WIDTH_MULTIPLIER*1280), #(8)
out_channels=int(NUM_CLASSES), #(9)
kernel_size=1,
stride=1,
padding=0,
batchnorm=False,
activation=nn.Id())Discover in Determine 1 that we initially begin from the usual convolution layer. Within the above codeblock, I discuss with this layer as first_conv (#(1)). It’s price noting that the enter arguments for this layer aren’t included within the BOTTLENECKS listing, therefore we have to outline them manually. Bear in mind to multiply the channel counts at every step by WIDTH_MULTIPLIER since we wish the mannequin measurement to be adjustable via that variable. Subsequent, we initialize a placeholder named blocks for storing all of the bottleneck blocks (#(2)). With a easy loop at line #(3), we’ll iterate via all gadgets within the BOTTLENECKS listing to really instantiate the bottleneck blocks and append them one after the other to blocks. Actually, this loop constructs the vast majority of the layers within the community, because it covers almost all parts listed within the desk.
Because the sequence of bottleneck blocks is finished, we’ll now proceed with the subsequent convolution layer, which I discuss with as second_conv (#(4)). Once more, because the configuration parameters for this layer aren’t saved within the BOTTLENECKS listing, we have to manually hard-code them. The output of this layer will then be handed via a worldwide common pooling layer (#(5)) which is able to drop the spatial dimension to 1×1. Afterwards, we join this layer to 2 consecutive pointwise convolutions (#(6) and #(8)) with a dropout layer in between (#(7)).
Speaking extra particularly concerning the two convolutions, it is very important know that making use of a 1×1 convolution on a tensor that has a 1×1 spatial dimension is basically equal to making use of an FC layer to a flattened tensor, the place the variety of channels will correspond to the variety of neurons. That is the rationale that I set the output channel depend of the final layer equal to the variety of lessons within the dataset (#(9)). The batchnorm parameter of each third_conv and output layers are set to False, as steered within the structure.
In the meantime, the activation perform of third_conv is about to nn.Hardswish(), whereas the output layer makes use of nn.Id(), which is equal to not making use of any activation perform in any respect. That is primarily executed as a result of throughout coaching softmax is already included within the loss perform (nn.CrossEntropyLoss()). Later within the inference part, we have to exchange nn.Id() with nn.Softmax() within the output layer in order that the mannequin will straight return the likelihood rating of every class.
Subsequent, let’s check out the ahead() methodology beneath, which I received’t clarify any additional since I feel it’s fairly simple to grasp.
# Codeblock 12b
def ahead(self, x):
print(f'originaltt: {x.measurement()}')
x = self.first_conv(x)
print(f'after first_convt: {x.measurement()}')
for i, block in enumerate(self.blocks):
x = block(x)
print(f"after bottleneck #{i}t: {x.form}")
x = self.second_conv(x)
print(f'after second_convt: {x.measurement()}')
x = self.avgpool(x)
print(f'after avgpooltt: {x.measurement()}')
x = self.third_conv(x)
print(f'after third_convt: {x.measurement()}')
x = self.dropout(x)
print(f'after dropouttt: {x.measurement()}')
x = self.output(x)
print(f'after outputtt: {x.measurement()}')
x = torch.flatten(x, start_dim=1)
print(f'after flattentt: {x.measurement()}')
return xThe code in Codeblock 13 demonstrates how we initialize a MobileNetV3 occasion and cross a dummy tensor via it. Keep in mind that right here we use the default enter decision, so we will principally consider the tensor as a batch of a single RGB picture of measurement 224×224.
# Codeblock 13
mobilenetv3 = MobileNetV3()
x = torch.randn(1, 3, INPUT_RESOLUTION, INPUT_RESOLUTION)
out = mobilenetv3(x)And beneath is what the ensuing output seems like, by which the tensor dimension after every block matches precisely with the MobileNetV3-Giant structure in Determine 1.
# Codeblock 13 Output
unique : torch.Dimension([1, 3, 224, 224])
after first_conv : torch.Dimension([1, 16, 112, 112])
after bottleneck #0 : torch.Dimension([1, 16, 112, 112])
after bottleneck #1 : torch.Dimension([1, 24, 56, 56])
after bottleneck #2 : torch.Dimension([1, 24, 56, 56])
after bottleneck #3 : torch.Dimension([1, 40, 28, 28])
after bottleneck #4 : torch.Dimension([1, 40, 28, 28])
after bottleneck #5 : torch.Dimension([1, 40, 28, 28])
after bottleneck #6 : torch.Dimension([1, 80, 14, 14])
after bottleneck #7 : torch.Dimension([1, 80, 14, 14])
after bottleneck #8 : torch.Dimension([1, 80, 14, 14])
after bottleneck #9 : torch.Dimension([1, 80, 14, 14])
after bottleneck #10 : torch.Dimension([1, 112, 14, 14])
after bottleneck #11 : torch.Dimension([1, 112, 14, 14])
after bottleneck #12 : torch.Dimension([1, 160, 7, 7])
after bottleneck #13 : torch.Dimension([1, 160, 7, 7])
after bottleneck #14 : torch.Dimension([1, 160, 7, 7])
after second_conv : torch.Dimension([1, 960, 7, 7])
after avgpool : torch.Dimension([1, 960, 1, 1])
after third_conv : torch.Dimension([1, 1280, 1, 1])
after dropout : torch.Dimension([1, 1280, 1, 1])
after output : torch.Dimension([1, 1000, 1, 1])
after flatten : torch.Dimension([1, 1000])As a way to be sure that our implementation is right, we will print out the variety of parameters contained within the mannequin utilizing the next code.
# Codeblock 14
total_params = sum(p.numel() for p in mobilenetv3.parameters())
total_params# Codeblock 14 Output
5476416Right here you possibly can see that this mannequin comprises round 5.5 million parameters, by which that is roughly the identical because the one disclosed within the unique paper (see Determine 10). Moreover, the parameter depend given within the PyTorch documentation can also be just like this quantity as you possibly can see in Determine 12 beneath. Primarily based on these info, I consider I can affirm that our MobileNetV3-Giant implementation is right.
Ending
Properly, that’s just about all the pieces concerning the MobileNetV3 structure. Right here I encourage you to really practice this mannequin from scratch on any datasets you need. Not solely that, I additionally need you to mess around with the parameter configurations of the bottleneck blocks to see whether or not we will nonetheless enhance the efficiency of MobileNetV3 even additional. By the way in which, the code used on this article can also be obtainable in my GitHub repo, which you could find within the hyperlink at reference quantity [9].
Thanks for studying. Be happy to achieve me via LinkedIn [10] if you happen to spot any mistake in my rationalization or within the code. See ya in my subsequent article!
References
[1] Muhammad Ardi. MobileNetV1 Paper Walkthrough: The Tiny Big. AI Advances. https://medium.com/ai-advances/mobilenetv1-paper-walkthrough-the-tiny-giant-987196f40cd5 [Accessed October 24, 2025].
[2] Muhammad Ardi. MobileNetV2 Paper Walkthrough: The Smarter Tiny Big. In the direction of Information Science. https://towardsdatascience.com/mobilenetv2-paper-walkthrough-the-smarter-tiny-giant/ [Accessed October 24, 2025].
[3] Andrew Howard et al. Trying to find MobileNetV3. Arxiv. https://arxiv.org/abs/1905.02244 [Accessed May 1, 2025].
[4] Muhammad Ardi. SENet Paper Walkthrough: The Channel-Smart Consideration. AI Advances. https://medium.com/ai-advances/senet-paper-walkthrough-the-channel-wise-attention-8ac72b9cc252 [Accessed October 24, 2025].
[5] Picture created initially by creator.
[6] Mark Sandler et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks. Arxiv. https://arxiv.org/abs/1801.04381 [Accessed May 12, 2025].
[7] Jie Hu et al. Squeeze and Excitation Networks. Arxiv. https://arxiv.org/abs/1709.01507 [Accessed May 12, 2025].
[8] Mobilenet_v3_large. PyTorch. https://docs.pytorch.org/vision/main/models/generated/torchvision.models.mobilenet_v3_large.html#torchvision.models.mobilenet_v3_large [Accessed May 12, 2025].
[9] MuhammadArdiPutra. The Tiny Big Getting Even Smarter — MobileNetV3. GitHub. https://github.com/MuhammadArdiPutra/medium_articles/blob/main/The%20Tiny%20Giant%20Getting%20Even%20Smarter%20-%20MobileNetV3.ipynb [Accessed May 12, 2025].
[10] Muhammad Ardi Putra. LinkedIn. https://www.linkedin.com/in/muhammad-ardi-putra-879528152/ [Accessed May 12, 2025].
