An Interpretable Multimodal Transfer Learning Decision-Support Framework for Multi-Crop, Multi-Disease Detection in Precision Agriculture

Abstract

Despite the widespread use of image-only convolutional models for plant disease diagnosis to provide global food security, image variability at the field level, visually similar symptoms, and stress due to soil nutrients or moisture conditions, can cause loss of accuracy. This paper compares classical machine learning classification models to custom convolutional neural networks and pretrained transfer-learning models for multi-crop and multi-disease classification and also introduces a late-fusion multimodal decision support model that could integrate data about images, soil, and weather. Experiments were conducted on the PlantVillage dataset which had 54,305 RGB images belonging to 38 different crop-condition classes (43,456 training, 10,849 validation, and 10,849 test images) across 14 different crops. The accuracy, macro-precision, macro-recall and macro-F1 score were computed for the five classical classifiers (KNN, Random Forest, Extra Trees, SGD-linear SVM, and SVC-RBF using PCA-reduced features), three custom CNN variants, and four pretrained models (ResNet50, MobileNetV2, GoogleNet, and EfficientNetB7). Of the classical models, SVC with RBF kernel yielded the highest accuracy (83.58%) and MacroF1 (82.96%). In the case of the custom CNN models, the deeper they became and the more dropout the higher accuracy they gained, with CNN-V3 attaining 95.71% accuracy and 95.66% macro-F1. Transfer learning yielded the best results with MobileNetV2 (99.30% accuracy and macro-F1) outperforming ResNet50 (98.60% accuracy and macro-F1) and GoogleNet (97.83% accuracy and 97.46% macro-F1). The findings serve as the basis for the introduction of a multimodal system for soil forecasting integrating a MobileNetV2 image encoder, a residual MLP for soil features, and two dual LSTMs for 48-hour and 168-hour time-series of the weather. The interpretability, modularity and the sufficient resistance towards loss of sensor information for probability-level late fusion with 0.80, 0.10, 0.05 and 0.05 respectively make the framework appropriate for precision-agriculture advisory systems, until it is tested in the field.