How Small Can Image Classifiers Get? My Experiments with Ultra-Lightweight Models Most image classification work today is about pushing accuracy higher and higher, usually by throwing more parameters at the problem. This project goes in the opposite direction: how small can the model get while still being useful? This post covers two tasks from TinyVision (v3): a cat vs dog classifier built around a handcrafted feature pipeline, and a CIFAR-10 classifier that ditches the filter bank entirely and just bets on compact CNN design. I am still writing the full v3 report, so the CIFAR-10 analysis here is partial, but there is enough to share the core ideas and the important numbers. Part 1: Cat vs Dog The Core Idea: Replace Learned Filters with Handcrafted Ones The central hypothesis is simple. CNNs spend a lot of their early capacity learning basic things like edges, textures, and gradients. What if you just give them those features upfront? Instead of feeding raw images into the network, every image goes through a fixed preprocessing pipeline that produces a 10-channel feature tensor. The model then only needs to learn higher-level patterns on top of these precomputed features. 10-channel feature tensor Here is what the pipeline looks like: Enhanced Grayscale (1 channel): Upscaled to 128x128 with Lanczos interpolation, denoised with non-local means, contrast-normalized with CLAHE, and sharpened with a Laplacian kernel. Otsu Threshold (1 channel): A binary foreground/background segmentation mask. The threshold value T is also passed forward to the next stage. Canny Edge Detection (2 channels): One with loose thresholds (0.3T, 0.7T) to catch subtle edges, one with strict thresholds (1.2T, 1.8T) to capture only dominant boundaries. Scharr Gradient Magnitude (2 channels): Scharr kernels have better rotational symmetry than Sobel. One channel at standard scale, one amplified to emphasize stronger transitions. Local Binary Pattern (2 channels): Texture descriptors. One at fine scale (P=16, R=2), one at coarser scale (P=24, R=4). Gabor Filters (2 channels): Oriented frequency filters mimicking early visual cortex. One sensitive to horizontal structure (0 degrees), one to vertical (90 degrees). Enhanced Grayscale (1 channel): Upscaled to 128x128 with Lanczos interpolation, denoised with non-local means, contrast-normalized with CLAHE, and sharpened with a Laplacian kernel. Enhanced Grayscale Otsu Threshold (1 channel): A binary foreground/background segmentation mask. The threshold value T is also passed forward to the next stage. Otsu Threshold Canny Edge Detection (2 channels): One with loose thresholds (0.3T, 0.7T) to catch subtle edges, one with strict thresholds (1.2T, 1.8T) to capture only dominant boundaries. Canny Edge Detection Scharr Gradient Magnitude (2 channels): Scharr kernels have better rotational symmetry than Sobel. One channel at standard scale, one amplified to emphasize stronger transitions. Scharr Gradient Magnitude Local Binary Pattern (2 channels): Texture descriptors. One at fine scale (P=16, R=2), one at coarser scale (P=24, R=4). Local Binary Pattern Gabor Filters (2 channels): Oriented frequency filters mimicking early visual cortex. One sensitive to horizontal structure (0 degrees), one to vertical (90 degrees). Gabor Filters The idea is that by making these inductive biases explicit, the model does not have to rediscover them from scratch, so it can do more with fewer parameters. The Architecture All four base architectures (Model0 through Model3) share a three-stage structure: Stage 1 processes each of the 10 input channels independently using grouped convolutions, expanding each channel to 2 and then compressing back to 1. No cross-channel mixing here, just per-feature extraction. Stage 2 groups the channels semantically into 5 pairs: [enhanced grayscale + Otsu], [Canny x2], [Scharr x2], [LBP x2], [Gabor x2]. Each pair gets its own small branch that reduces it to a single spatial map. The 5 outputs are concatenated to produce a 5-channel tensor. Stage 3 does global spatial reduction down to a 2x2 feature map, giving 20 scalar features total. Stage 1 processes each of the 10 input channels independently using grouped convolutions, expanding each channel to 2 and then compressing back to 1. No cross-channel mixing here, just per-feature extraction. Stage 1 Stage 2 groups the channels semantically into 5 pairs: [enhanced grayscale + Otsu], [Canny x2], [Scharr x2], [LBP x2], [Gabor x2]. Each pair gets its own small branch that reduces it to a single spatial map. The 5 outputs are concatenated to produce a 5-channel tensor. Stage 2 Stage 3 does global spatial reduction down to a 2x2 feature map, giving 20 scalar features total. Stage 3 Finally, a simple classification head: 20 -> 10 -> 5 -> 2. The four architectures differ in two dimensions: whether they use pooling or strided convolutions for downsampling, and whether they use channel expansion in the intermediate stages. HybridPoolingV0: A Lightweight Pseudo-Attention Module One of the more interesting design choices is a custom pooling module called HybridPoolingV0. The motivation was simple: MaxPool and AvgPool do very different things. MaxPool picks the sharpest local activation. AvgPool smooths out the neighborhood. What if you used one to decide where to look, and the other to decide what to read? where what The forward pass works like this: Apply both pool1 and pool2 to the input independently (both halve the spatial resolution). Pass pool1's output through BatchNorm and Sigmoid to generate a soft attention map. Multiply the attention map element-wise with pool2's output. Apply both pool1 and pool2 to the input independently (both halve the spatial resolution). Pass pool1's output through BatchNorm and Sigmoid to generate a soft attention map. Multiply the attention map element-wise with pool2's output. In the default configuration, pool1 is MaxPool (attention source) and pool2 is AvgPool (content). The sharp MaxPool output decides which regions matter, and the smooth AvgPool output provides the content at those regions. It adds zero parameters compared to a plain pooling layer, which makes it a very cheap upgrade. Results Architecture comparison (averaged across all pooling and ratio variants): Architecture comparison The most interesting result here is Model3 vs Model2. The only structural difference between them is pooling vs strided convolutions. Swapping to pooling bumps average accuracy from 77.38% to 81.61%, at the cost of just 320 extra parameters. At very small scales, pooling seems to carry a useful inductive bias that strided convolutions do not easily replicate. Pooling strategy comparison (Model0, all ratios): Pooling strategy comparison The flipped config (Avg->Max, where AvgPool generates the attention map and MaxPool provides content) achieves the single best result at 86.87%, which was the opposite of what I originally expected. A smoother attention map from AvgPool seems to generalize better than the sharper one from MaxPool. Plain AvgPool also consistently beats plain MaxPool, with a noticeably smaller generalization gap. MaxPool alone seems to latch onto sharp local features in a way that hurts generalization at this scale. Efficiency: If raw accuracy per parameter is what you care about, Model3 with Hybrid(Max->Avg) is the standout: 83.37% test accuracy with only 3.5k parameters. Efficiency Overall best: model15_0 at 86.87% test accuracy (Model0, Hybrid Avg->Max, ratio=3, batch size 40, 12.48k parameters). Overall best Part 2: CIFAR-10 I am still writing the full report for this section. What follows is based on the current draft and the available results. A proper analysis post will come once the report is finalized. I am still writing the full report for this section. What follows is based on the current draft and the available results. A proper analysis post will come once the report is finalized. A Different Approach The CIFAR-10 experiments drop the filter bank idea entirely. No handcrafted preprocessing pipeline. Instead, the focus shifts to iterative architecture search: try things, keep what works, remove what does not. The question here is how far compact CNN design alone can take you. Training Setup Architecture: BaseModelF32 The backbone is a modular four-block structure where each block is injected at construction time. The only fixed contract is input/output shape compatibility. The four blocks collectively produce a 240-dimensional feature vector, which feeds into a final Linear(240, 10) head. The design space is large. Block 1 expands 3 channels to 30 using depthwise convolutions, with three variants: the full depthwise-pointwise-depthwise-pointwise stack, a version where downsampling is moved to the end of the block rather than the middle, and a pointwise-free depthwise-only version. Block 2 doubles to 60 channels with the same three variant types. Block 3 is the most varied with 11 variants, exploring different channel trajectories: expanding to 120 and compressing back, compressing to 30 then expanding back, and flat 60-channel variants with different grouping strategies. Block 4 handles compression back down from 120 or 60 channels with 6 variants. Downsampling is handled by the same HybridPoolingV0 module from the cat vs dog experiments, with a full set of 12 postconv configurations varying the pooling type and the ordering of activation and pooling. The result is a large combinatorial space of model configurations. Instead of a grid search, the approach is trial and error: test configurations, identify what contributes positively, and build from there. Headline Numbers The full per-model results table is still being compiled, but the headline results are: A 22.11k parameter model reached 87.38% test accuracy A 31.15k parameter model reached 88.43% test accuracy A 22.11k parameter model reached 87.38% test accuracy 22.11k parameter model reached 87.38% test accuracy A 31.15k parameter model reached 88.43% test accuracy 31.15k parameter model reached 88.43% test accuracy For context, these are competitive numbers for models this size on CIFAR-10, where standard ResNet variants typically need 200k+ parameters to cross 90%. The full breakdown of which architectures drove these results will be in the next version of the report. Key Takeaways So Far Handcrafted feature pipelines are a viable way to reduce parameter count without completely sacrificing accuracy. The 10-channel input lets even a 3-4k parameter network do something reasonable on binary classification. Pooling beats strided convolutions at small scales, at least in these experiments. Model3 vs Model2 is the clearest evidence. HybridPoolingV0 consistently outperforms plain pooling at no additional parameter cost. The pseudo-attention design helps even in the simplest architecture. The Avg->Max configuration outperformed Max->Avg in Model0, which was surprising. Smoother attention maps seem to generalize better as a gate. Compact CNNs without any preprocessing can still be surprisingly capable, as the CIFAR-10 numbers suggest. More on that once the report is done. Handcrafted feature pipelines are a viable way to reduce parameter count without completely sacrificing accuracy. The 10-channel input lets even a 3-4k parameter network do something reasonable on binary classification. Handcrafted feature pipelines are a viable way to reduce parameter count Pooling beats strided convolutions at small scales, at least in these experiments. Model3 vs Model2 is the clearest evidence. Pooling beats strided convolutions at small scales HybridPoolingV0 consistently outperforms plain pooling at no additional parameter cost. The pseudo-attention design helps even in the simplest architecture. HybridPoolingV0 consistently outperforms plain pooling The Avg->Max configuration outperformed Max->Avg in Model0, which was surprising. Smoother attention maps seem to generalize better as a gate. The Avg->Max configuration outperformed Max->Avg Compact CNNs without any preprocessing can still be surprisingly capable, as the CIFAR-10 numbers suggest. More on that once the report is done. Compact CNNs without any preprocessing can still be surprisingly capable What's Next Future versions of TinyVision will include quantization experiments, proper ablation studies with multiple seeds, architecture diagrams, and a more thorough analysis of why certain design choices work. The CIFAR-10 report is the immediate next thing. If any of this sounds interesting, the full code is at https://github.com/SaptakBhoumik/TinyVision. If you find it interesting, leave a star or some feedback, and feel free to reach out on X at https://x.com/saptakbhoumik or via email at saptakbhoumik.acad@gmail.com. I would love to talk about it. https://github.com/SaptakBhoumik/TinyVision https://github.com/SaptakBhoumik/TinyVision https://x.com/saptakbhoumik saptakbhoumik.acad@gmail.com
