A more parameter-efficient bottleneck for EfficientNet!

Image for post
Image for post
Linear Bottleneck with Efficient Channel Attention instead of Squeeze Excitation

Introduction

Linear Bottleneck

Image for post
Image for post
Linear Bottleneck (cf MobileNetV2 paper)

EfficientNet: Linear Bottleneck + Squeeze Excitation

Image for post
Image for post
EfficientNet solving the “attention problem”: linearbottleneck_se. A better representation with hard-sigmoid can be found in MobileNetV3 paper (page 3)
Image for post
Image for post
Our reproduction of EfficientNetB0 after ~50 epochs (source for red line)

EfficientNetECA: Linear Bottleneck + Efficient Channel Attention

Image for post
Image for post
ECA layer : https://arxiv.org/pdf/1910.03151.pdf
Image for post
Image for post
Validation Accuracy we got with EfficientNetECA

Tl;dr

Conclusion / Further works

More plots

Image for post
Image for post
Partial Top1Acc = f(n_images_seen) — with curve smoothing
Image for post
Image for post
Partial Top1Acc = f(time) — with curve smoothing

Addendums

Code

Support us 💙

We work on statistics, machine learning, AI and we want to bring Artificial General Intelligence into the real world — Website: https://www.hyugen.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store