A multimodal deep learning approach for gravel road condition evaluation through image and audio integration

• This study investigates the combination of audio and image data to classify road conditions, particularly focusingon loose gravel scenarios. The dataset underwent binary categorisation, comprising audio segments capturinggravel sounds and corresponding images. Early feature fusion, utilising a pre-trained Very Deep ConvolutionalNetworks 19 (VGG19) and Principal component analysis (PCA), improved the accuracy of the Random Forestclassifier, surpassing other models in accuracy, precision, recall, and F1-score. Late fusion, involving decisionlevelprocessing with logical disjunction and conjunction gates (AND and OR) in combination with individualclassifiers for images and audio based on Densely Connected Convolutional Networks 121 (DenseNet121),demonstrated notable performance, especially with the OR gate, achieving 97 % accuracy. The late fusionmethod enhances adaptability by compensating for limitations in one modality with information from the other.Adapting maintenance based on identified road conditions minimises unnecessary environmental impact. Thismethod can help to identify loose gravel on gravel roads, substantially improving road safety and implementing aprecise maintenance strategy through a data-driven approach.

Subject headings

TEKNIK OCH TEKNOLOGIER  -- Samhällsbyggnadsteknik -- Arkitekturteknik (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Civil Engineering -- Architectural Engineering (hsv//eng)

Keyword

Gravel road maintenance

Data fusion

Sound analysis

Machine vision

Machine

Learning

Publication and Content Type

ref (subject category)

art (subject category)