Enhancing AI model U-Net for ocean remote sensing

U-Net, a convolutional neural network (CNN) originally intended for medical use, can potentially make waves in the ocean remote sensing field

There is seldom an issue in our modern world that cannot be solved or helped by technology and artificial intelligence (AI). In this instance, U-Net, a tool used to extract a desired “object” from a medical image, is looked at as a potential means of oceanographic research. Although it’s promising, U-Net is not perfect. A few key improvements in the model can make a huge difference when it comes to the field of ocean remote sensing.

Researchers published their findings in the Journal of Remote Sensing in August 2024.

The U-Net model appears to have a good enough structure to be a fine candidate for oceanographic research, but in its current state, it is not able to completely fulfill the needs of researchers.

To solve the challenges U-Net faces with pivoting to oceanographic research, three main categories need improvement: the model’s segmentation tasks, or ability to categorize each pixel in an image, forecasting tasks and super-resolution tasks.

“Through structural improvement and the introduction of new techniques, the U-Net model can gain significant improvement in small target detection, prediction accuracy and image reconstruction quality, further promoting the development of ocean remote sensing research,” said Haoyu Wang, author and researcher.

Improving semantic segmentation can improve the U-Net’s ability to detect and identify small targets in the ocean. This can be done by integrating the model with the ability to recognize and identify pixels at a distance away via attention mechanisms. For example, getting the model to recognize the difference between open water and ice formations in the ocean is integral, and U-Net can determine this difference.

Forecasting tasks refer to the model’s ability to logically predict an outcome based on physical knowledge and data-driven methods. Previous successes using the U-Net model for oceanic remote sensing include the Sea Ice Prediction Network (SIPNet), which predicts the sea ice concentration of the Antarctic.

SIPNet, the U-Net model, used another form of neural network architecture known as “encoder-decoder” that processes an input sequence (encoder) to later be reconstructed back to the original form (decoder). This is often used for summarizing or translating text, but in this case, SIPNet used 8 weeks of data about sea ice concentration to forecast the 8 weeks following. When the encoder-decoder architecture was combined with a temporal-spatial attention module (TSAM), the average difference between the prediction and the actual measurement was less than 3% for a 7-day forecast, showcasing the accuracy U-Net models can have when fully outfitted for the task.

Lastly, the improvements suggested for super-resolution tasks include the introduction of a diffusion model to reduce blurring in the images, or “noise.” To reduce noise in images, the correlation between high and low-resolution images has to be identified by taking note of the similarities observed in both resolutions. This also includes making improvements to the model’s capability of extracting features from images.

Researchers suggest utilizing a model, PanDiff, to blend the high-res panchromatic (sensitive to all visible colors in the spectrum) and low-resolution multispectral images (images that capture data through spectrums such as infrared and ultraviolet) to be reconstructed by U-Net via the random noise.

Further optimization of the U-Net model is necessary to support the goals of researchers in the long term.

“The U-Net model’s straightforward and understandable network architecture and superior model fitting capabilities have garnered the most popularity among researchers in the ocean remote sensing community, demonstrating great potential,” said Xiaofeng Li, researcher and author of the study.

In addition to the improvements researchers suggest for using U-Net in oceanic research, there is plenty of exploration to be done by combining U-Net with other systems or techniques to further extend an already wide application of the model.

U-Net, a convolutional neural network (CNN) originally intended for medical use, can potentially make waves in the ocean remote sensing field

There is seldom an issue in our modern world that cannot be solved or helped by technology and artificial intelligence (AI). In this instance, U-Net, a tool used to extract a desired “object” from a medical image, is looked at as a potential means of oceanographic research. Although it’s promising, U-Net is not perfect. A few key improvements in the model can make a huge difference when it comes to the field of ocean remote sensing.

Researchers published their findings in the Journal of Remote Sensing in August 2024.

The U-Net model appears to have a good enough structure to be a fine candidate for oceanographic research, but in its current state, it is not able to completely fulfill the needs of researchers.

To solve the challenges U-Net faces with pivoting to oceanographic research, three main categories need improvement: the model’s segmentation tasks, or ability to categorize each pixel in an image, forecasting tasks and super-resolution tasks.

“Through structural improvement and the introduction of new techniques, the U-Net model can gain significant improvement in small target detection, prediction accuracy and image reconstruction quality, further promoting the development of ocean remote sensing research,” said Haoyu Wang, author and researcher.

Improving semantic segmentation can improve the U-Net’s ability to detect and identify small targets in the ocean. This can be done by integrating the model with the ability to recognize and identify pixels at a distance away via attention mechanisms. For example, getting the model to recognize the difference between open water and ice formations in the ocean is integral, and U-Net can determine this difference.

Forecasting tasks refer to the model’s ability to logically predict an outcome based on physical knowledge and data-driven methods. Previous successes using the U-Net model for oceanic remote sensing include the Sea Ice Prediction Network (SIPNet), which predicts the sea ice concentration of the Antarctic.

SIPNet, the U-Net model, used another form of neural network architecture known as “encoder-decoder” that processes an input sequence (encoder) to later be reconstructed back to the original form (decoder). This is often used for summarizing or translating text, but in this case, SIPNet used 8 weeks of data about sea ice concentration to forecast the 8 weeks following. When the encoder-decoder architecture was combined with a temporal-spatial attention module (TSAM), the average difference between the prediction and the actual measurement was less than 3% for a 7-day forecast, showcasing the accuracy U-Net models can have when fully outfitted for the task.

Lastly, the improvements suggested for super-resolution tasks include the introduction of a diffusion model to reduce blurring in the images, or “noise.” To reduce noise in images, the correlation between high and low-resolution images has to be identified by taking note of the similarities observed in both resolutions. This also includes making improvements to the model’s capability of extracting features from images.

Researchers suggest utilizing a model, PanDiff, to blend the high-res panchromatic (sensitive to all visible colors in the spectrum) and low-resolution multispectral images (images that capture data through spectrums such as infrared and ultraviolet) to be reconstructed by U-Net via the random noise.

Further optimization of the U-Net model is necessary to support the goals of researchers in the long term.

“The U-Net model’s straightforward and understandable network architecture and superior model fitting capabilities have garnered the most popularity among researchers in the ocean remote sensing community, demonstrating great potential,” said Xiaofeng Li, researcher and author of the study.

In addition to the improvements researchers suggest for using U-Net in oceanic research, there is plenty of exploration to be done by combining U-Net with other systems or techniques to further extend an already wide application of the model.