变化检测综述
综述1：Change Detection Based on Artificial Intelligence: State-of-the-Art and Challenges
收录于：Remote Sens.2020
论文地址：Remote Sensing | Free Full-Text | Change Detection Based on Artificial Intelligence: State-of-the-Art and Challenges (mdpi.com)
解读：
变化检测综述：Change Detection Based on Artificial Intelligence: State-of-the-Art and Challenges_naath的博客-CSDN博客_变化检测发展历程
https://www.5axxw.com/wiki/content/y3bz0q#Ref-15
综述2：Deep learning for change detection in remote sensing images: comprehensive review and meta-analysis
收录于：IEEE Access，2020
论文地址：Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis | IEEE Journals & Magazine | IEEE Xplore
Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis | DeepAI
综述1，介绍变化检测流程、各种类型的CD数据集、分析不同的算法框架与当前AI主流网络，实际应用，机遇与挑战（无监督、异构数据、是否可靠），内容丰富。
综述2，包括，遥感领域中变化检测的应用，深度学习算法概述（历史、deep models、DBNs、SAEs、CNNs），已有的CD相关论文的汇集整理分析和涉及的期刊会议，大致分为三组（有监督、无监督CD、迁移学习），有前途的研究方向（强化学习、弱监督学习）。
相比之下，综述2适合简单的入门科普，综述1更为深刻，值得细读。
本篇博客主要围绕综述1所写，旨在记录CD相关代码、论文、CD数据集。具体有关CD的内容请详读论文。
正文
1. Introduction
基于遥感（RS）数据的变化检测是检测地球表面变化的重要方法，并且在城市规划，环境监测，农业调查，灾害评估和地图修订中具有广泛的应用。近年来，综合人工智能（AI）技术已成为开发新变化检测方法的研究重点。虽然一些研究人员声称AI的变化检测方法优于传统的变化检测方法，但它没有立即明显如何以及在多大程度上可以提高变化检测的性能。本综述重点是AI变化检测的最先进的方法，应用和挑战。具体地，首先引入基于AI的变化检测的实现过程。然后，呈现来自用于改变检测的不同传感器的数据，包括光学RS数据，合成孔径雷达（SAR）数据，街景图像和组合的异构数据，并且还列出了可用的开发数据集。系统地评估和分析了基于AI的变化检测方法的一般框架，进一步分析了AI基变化检测中使用的无监督方案。随后，描述了用于改变检测的AI中的常用网络。从实际的角度来看，基于AI的改变检测方法的应用领域基于其适用性进行分类。最后，讨论了AI用于变革检测的主要挑战和前景，包括（a）异构大数据处理，（b）无监督的ai，（c）AI的可靠性。本综述将有利于研究人员了解这一领域。
变化检测是通过在不同时间观察到来识别物体或现象状态的差异的过程。它们大致分为两类：传统和基于AI的。图1显示了传统变革检测的一般流程和基于AI的变化检测。
现有的变化检测主要集中在多时间超光谱图像（HSIS）和高空间分辨率图像中的变化检测技术的设计。他们审查的技术主要是传统的变化检测方法，可以概括为以下组：
- 目视分析：通过手动解释获得变更图，这可以根据专家知识提供高度可靠的结果，但是耗时和劳动密集型的效率;
- 基于Aalgebra的方法：通过对多时间数据执行代数操作或转换来获得变化图，例如图像差异，图像回归，图像比和改变向量分析（CV A）;
- Transformation：数据减少方法，例如原理分析（PCA），流苏帽（KT），多变量改变检测（MAD），Gramm-Schmidt（GS）和Chi-Square，用于抑制相关信息和突出显示多时间数据方差;
- 基于分类的方法：通过比较多个分类映射（即，分类后比较），或者使用训练分类器来直接对来自多个时段（即，多种分类或直接分类）直接对数据进行分类的更改;
- 高级模型：高级模型，如Li-Strahler反射模型，光谱混合模型和生物物理学参数方法，用于将多个时期数据的光谱反射率值转换为物理基础的参数或分数以进行变化分析，这更直观，具有物理意义，但它是复杂且耗时的;
- 其他：混合方法和其他人，例如基于知识，空间统计数据和集成的GIS和RS方法。
根据检测单元，这些方法也可以基于像素级，特征级，对象级别和三维（3D）对象级别分类。由于计算机技术的快速发展，传统变革检测方法的研究已经转向集成AI技术。在传统的变化检测流程和基于AI的流动中，第一步是数据采集，改变检测的目的是获得各种应用的变化检测图;在准备数据之后，传统方法通常由两个步骤组成，包括均质化过程和变化检测过程，而基于AI的方法通常需要额外的训练集生成过程和用于改变检测的AI模型训练过程。显然，基于AI的方法的关键组成部分是AI技术。
AI技术，人工智能，可以在各种数据处理任务中提供更好的性能。由于强大的建模和学习能力，AI技术可以尽可能地将图像对象与其现实世界地理特征之间的关系模拟，这使得能够检测更实际的变化信息。通常，它们利用多时间数据中的空间上下文信息来学习分层特征表示，并且这些高级特征表示在更改检测任务中更有效且稳健。
本文提供了对AI技术在RS改变检测处理中的应用的深度审查。它侧重于AI在多时间数据中改变检测的最先进的方法，应用和挑战。本文的主要贡献如下：
- 介绍了基于AI的变化检测的实施过程，我们总结了可以帮助初学者理解这一研究领域的常见实施策略;
- 我们详细介绍了用于AI的改变检测的不同传感器的数据，主要包括光学RS数据，SAR数据，街道视图图像和组合异构数据。更实际上，我们列出了带有注释的可用打开数据集，可以用作培训和评估未来改变检测研究的基准的基准;
- 通过系统地审查和分析基于AI的变化检测方法的过程，我们以实际的方式总结了他们的一般框架，这有助于将来设计变化检测方法。此外，分析了基于AI的变化检测中使用的无监督方案，以帮助解决实际应用中缺乏培训样本的问题;
- 我们描述了AI中的常用网络进行变化检测。分析其适用性有助于在实际应用中选择AI模型;
- 我们提供了在各种领域的基于AI的变化检测，并将其细分为不同的数据类型，这有助于那些对这些领域感兴趣以找到相关的AI的变化检测方法;
- 我们描绘并讨论AI对三个主要方向的变化检测的挑战和前景，即异构大数据处理，无监督的AI和AI的可靠性，为未来的研究提供有用的参考。
2. Implementation process
图1说明了基于AI的变化检测的一般流程。关键是获得高性能训练的AI模型。详细地说，如图2所示，基于AI的变更检测的实现过程包括以下四个主要步骤：
- 均质化：由于照明和大气条件的差异，季节和传感器态度在采集时，通常需要在改变检测之前均化均质。几何和辐射矫正是两个常用的方法[14,15]。前者旨在通过登记或共同登记来几何对准两个或更多个给定的数据。仅在两个时段数据中，只有当它们被覆盖时，相应位置之间的比较就会有意义[16]。后者旨在消除由传感器的数字化过程和由大气中的吸收和散射引起的大气衰减变形引起的辐射或反射差异[4]，这有助于减少由这些辐射误差引起的变化检测引起的误报。对于异构数据，可以设计特殊的AI模型结构以实现变化检测（参见第4.1.2节）;
- 训练集生成：要开发AI模型，需要一个大型高质量的培训集，可以帮助算法来了解某些模式或一系列结果具有给定的问题。使用某些技术（例如，手动注释[17]，使用主题数据[19]）来标记或注释多个时段数据（例如，手动注释[17]）以使AI模型简单地学习改变的特征对象。图2呈现了用于构建变更检测的注释示例，该示例由两个时段RS图像组成，并在像素级别的构建变化标记的相应地形真理组成。基于地面真理，即，先验知识，可以以监督方式培训AI模型。为了减轻缺乏培训数据的问题，被广泛使用的数据增强是一种很好的策略，例如水平或垂直翻转，旋转，尺度变化，作物，翻译或增加噪声，这可以显着提高多样性可用于培训模型的数据，而无需实际收集新数据;
- 模型培训：生成训练集后，通常可以根据样品或地理区域的数量分为两个数据集：用于AI模型训练的训练和用于训练过程中的准确性评估的测试集[ 20]。训练和测试过程是交替和迭代的。在培训过程中，该模型根据学习标准进行了优化，这可以是深度学习中的损失功能（例如，软MAX丢失[21]，对比丢失[22]，欧几里德丢失[23]或交叉熵损失[24]）。通过监视培训过程和测试精度，可以获得AI模型的收敛状态，可以帮助调整其超参数（例如学习率），也可以在判断模型性能是否已达到最佳状态（即，终止）条件;
- 模型服务：通过部署训练有素的AI模型，可以更智能地和自动为实际应用程序生成更改映射。此外，这有助于验证模型的泛化能力和鲁棒性，这是评估基于AI的变化检测技术的实用性的重要方面。
论文将基于AI的变化检测架构大致分为：
- 单流架构：直接分类结构、基于转换的结构
- 双流架构：孪生结构、基于迁移学习的结构、分类后结构
- 多模型集成结构
- 无监督方式
也写到，AI的主流网络包括：
AE、DBN、CNN、RNN、PCNN、GANs。 （还有近两年大火的Transformer。）
在实际应用中，根据变化信息，最终变化图 可以分为四种类型，binary maps, one-class maps, from–to maps, and instance maps，如图10所示。
- binary maps, 使用1和0表示变化，没有变化。它包含任何变化，无法为已变化的真实对象提供其他类型的信息。
- one-class maps,提供单类变化信息，指示特定类型的地面对象的外观和消失。例如，构建变化检测的结果是指示建筑物的添加和拆卸的单级地图，可用于城市管理。
- from–to maps， 提供变化传输信息，指示接地对象从一个类型改变为另一个类型，并且这些改变类型由分类器确定。
- instance maps为每个变化实例提供边界，这可能是基于对象的变化检测的结果。
可以通过训练的AI模型生成这些变化检测并在各种应用中使用。
2.1 Available codes for AI-based methods **
GitHub – MinZHANG-WHU/Change-Detection-Review: A review of change detection methods, including codes and open data sets for deep learning. From paper: change detection based on artificial intelligence: state-of-the-art and challenges.
Table 1. 深度学习变化检测方法。
| Methods | Keywords | Publication | (Re-)Implementation |
|---|---|---|---|
| SRCDNet | CNN; Siamese; Attention; Super-resolution; Optical RS | Super-resolution-based change detection network with stacked attention module for images with different resolutions, TGRS, 2021. [paper], [code, dataset] | Pytorch 1.2 |
| ESCNet | CNN; Siamese; Superpixel; Optical RS | An End-to-End superpixel-enhanced change detection network for Very-High-Resolution remote sensing images. TNNLS, 2021. [paper], [code] | Pytorch 1.3 |
| KPCAMNet | CNN; Siamese; KPCA; Unsupervised; Optical RS | Unsupervised change detection in multitemporal VHR images based on deep kernel PCA convolutional mapping network,TCYB, 2021. [paper], [code] | Python |
| SeCo | CNN (ResNet); Transfer Learning; Optical RS | Seasonal contrast: unsupervised pre-training from uncurated remote sensing data, arXiv, 2021. [paper], [code, dataset] | Pytorch 1.7 |
| CapsNet | Capsule Network(SegCaps); CVA; Siamese; Optical RS | Pseudo-siamese capsule network for aerial remote sensing images change detection, GRSL, 2020. [paper 1], Change Capsule Network for Optical Remote Sensing ImageChange Detection, RS, 2021. [paper 2], [code, dataset] | Keras |
| BIT_CD | CNN (ResNet18); Siamese; Attention; Transformer; Optical RS | Remote sensing image change detection with transformers, TGRS, 2021. [paper], [code, dataset, pre-trained model] | Pytorch 1.6 |
| IAug_CDNet | CNN (GauGAN+UNet); Siamese; GAN; Supervised; Optical RS | Adversarial instance augmentation for building change detection in remote sensing images, TGRS, 2021. [paper], [code, dataset] | Pytorch |
| DDNet | CNN; DI+FCM; Unsupervised; SAR | Change detection in synthetic aperture radar images using a dual-domain network, GRSL, 2021. [paper], [code, dataset] | Pytorch |
| SNUNet-CD | CNN (NestedUNet); Siamese; Attention; Supervised; Optical RS | SNUNet-CD: A densely connected siamese network for change detection of VHR images, GRSL, 2021. [paper], [code, dataset, pre-trained model] | Pytorch 1.4 |
| DSMSCN | CNN; Siamese; Multi-scale; Unsupervised/Supervised; Optical RS | A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images, arXiv, 2020. [paper], [code, dataset] | Tensorflow 1.9 |
| SiamCRNN | CNN+RNN; Siamese; Multi-source; Optical RS | Change Detection in Multisource VHR Images via Deep Siamese Convolutional Multiple-Layers Recurrent Neural Network, TGRS, 2020. [paper], [code, dataset] | Tensorflow 1.9 |
| DSIFN | CNN; Attention Mechanism; Optical RS | A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images, ISPRS, 2020. [paper], [code, dataset] | Pytorch & Keras |
| CEECNet | CNN; Attention Mechanism; Similarity Measure; Optical RS | Looking for change” />paper], [code, dataset] | MXNet + Python |
| LamboiseNet | CNN (Light UNet++); Optical RS | Change detection in satellite imagery using deep learning, Master Thesis. [code, dataset, pre-trained model] | Pytorch |
| DTCDSCN | CNN; Siamese | Building change detection for remote sensing images using a dual task constrained deep siamese convolutional network model, undergoing review. [code, dataset] | Pytorch |
| Land-Cover-Analysis | CNN (UNet); Post-Classification; Optical RS | Land Use/Land cover change detection in cyclone affected areas using convolutional neural networks. [report], [code, dataset, pre-trained model] | TensorFlow+Keras |
| CorrFusionNet | CNN; Scene-level; Siamese; Optical RS | Correlation based fusion network towards multi-temporal scene classification and change detection, undergoing review. [code, pre-trained model], [dataset] | TensorFlow 1.8 |
| SSCDNet | CNN (ResNet18); Siamese; Transfer Learning; Semantic; Streetview | Weakly supervised silhouette-based semantic scene change detection, ICRA, 2020. [paper] [code, dataset, pre-trained model] | Pytorch+Python3.6 |
| Heterogeneous_CD | AE (Code-Aligned AE); Unsupervised; Transformation; Heterogeneous; Optical RS | Code-aligned autoencoders for unsupervised change detection in multimodal remote sensing images, arXiv, 2020. [paper] [code, dataset] | TensorFlow 2.0 |
| FDCNN | CNN (VGG16); Transfer Learning; Pure-Siamese; Multi-scale; Optical RS | A feature difference convolutional neural network-based change detection method, TGRS, 2020. [paper] [code, dataset, pre-trained model] | Caffe+Python2.7 |
| STANet | CNN (ResNet-18); Attention Mechanism; Pure-Siamese; Spatial–Temporal Dependency; Optical RS | A spatial-temporal attention-based method and a new dataset for remote sensing image change detection, RS, 2020. [paper] [code, dataset] | Pytorch+Python3.6 |
| X-Net | CNN; Unsupervised; Transformation; Heterogeneous; Optical RS; SAR | Deep image translation with an affinity-based change prior for unsupervised multimodal change detection, 2020. [paper] [code, dataset] | Tensorflow 1.4 |
| ACE-Net | AE (Adversarial Cyclic Encoders); Unsupervised; Transformation; Heterogeneous; Optical RS; SAR | Deep image translation with an affinity-based change prior for unsupervised multimodal change detection, 2020. [paper] [code, dataset] | Tensorflow 1.4 |
| VGG_LR | CNN (VGG16); Transfer Learning; Pure-Siamese; SLIC; Low Ranks; Optical RS | Change detection based on deep features and low rank, GRSL, 2017. [paper] [re-implementation code, dataset, pre-trained model] | Caffe+Matlab |
| CDNet | CNN; Siamese; Multimodal Data; Point Cloud Data | Detecting building changes between airborne laser scanning and photogrammetric data, RS, 2019. [paper], [code] | Pytorch |
| SCCN | AE (DAE); Unsupervised; Heterogeneous; Optical RS; SAR | A deep convolutional coupling network for change detection based on heterogeneous optical and radar images, TNNLS, 2018. [paper] [re-implementation code] | TensorFlow 2.0 |
| cGAN | GAN (conditional GAN); Heterogeneous; Optical RS; SAR | A conditional adversarial network for change detection in heterogeneous images, GRSL, 2019. [paper] [re-implementation code] | TensorFlow 2.0 |
| DASNet | CNN (VGG16); Siamese; Attention Mechanism ; Optical RS | DASNet: Dual attentive fully convolutional siamese networks for change detection of high resolution satellite images, arXiv, 2020. [paper] [code, dataset, pre-trained model] | Pytorch+Python3.6 |
| UNetLSTM | CNN (UNet); RNN (LSTM); Integrated Model; Optical RS | Detecting Urban Changes With Recurrent Neural Networks From Multitemporal Sentinel-2 Data, IGARSS, 2019. [paper] [code, dataset, pre-trained model] and [code] | Pytorch+Python3.6 |
| CDMI-Net | CNN (Unet); Pure-Siamese; Multiple Instance Learning; Landslide Mapping; Optical RS | Deep multiple instance learning for landslide mapping, GRSL, 2020. [paper] [code, pre-trained model] | Pytorch+Python3.6 |
| DSFANet | DNN; Unsupervised; Pre-classification; Slow Feature Analysis; Optical RS | Unsupervised deep slow feature analysis for change detection in multi-temporal remote sensing images, TGRS, 2019. [paper] [code, dataset] | TensorFlow 1.7 |
| CD-UNet++ | CNN (improved UNet++); Direct Classification; Optical RS | End-to-end change detection for high resolution satellite images using improved UNet++, RS, 2019. [paper] [code] | TensorFlow+Keras |
| SiameseNet | CNN (VGG16); Pure-Siamese; Optical RS | Siamese network with multi-level features for patch-based change detection in satellite imagery, GlobalSIP, 2018. [paper] [code, dataset] | TensorFlow+Keras |
| Re3FCN | CNN (ConvLSTM); PCA; 3D convolution; Multi-class changes; Optical RS; Hyperspectral | Change detection in hyperspectral images using recurrent 3D fully convolutional networks, RS, 2018. [paper] [code, dataset] | TensorFlow+Keras |
| FC-EF, FC-Siam-conc, FC-Siam-diff | CNN (UNet); Pure-Siamese; Optical RS | Fully convolutional siamese networks for change detection, ICIP, 2018. [paper] [code, dataset] | Pytorch |
| CosimNet | CNN (Deeplab v2); Pure-Siamese; Streetview | Learning to measure changes: fully convolutional siamese metric networks for scene change detection, arXiv, 2018. [paper] [code, dataset, pre-trained model] | Pytorch+Python2.7 |
| Mask R-CNN | Mask R-CNN (ResNet-101); Transfer Learning; Post-Classification; Optical RS | Slum segmentation and change detection: a deep learning approach, NIPS, 2018. [paper] [code, dataset, pre-trained model] | TensorFlow+Keras |
| CaffeNet | CNN (CaffeNet); Unsupervised; Transfer Learning; Optical RS | Convolutional neural network features based change detection in satellite images, IWPR, 2016. [paper] [code, dataset] | TensorFlow+Keras |
| CWNN | CNN (CWNN); Unsupervised; Pre-Classification; SAR | Sea ice change detection in SAR images based on convolutional-wavelet neural networks, GRSL, 2019. [paper] [code, dataset] | Matlab |
| MLFN | CNN (DenseNet); Transfer learning; SAR | Transferred deep learning for sea ice change detection from synthetic aperture radar images, GRSL, 2019. [paper] [code, dataset] | Caffe+Matlab |
| GarborPCANet | CNN (PCANet); Unsupervised; Pre-Classification; Gabor Wavelets; SAR | Automatic change detection in synthetic aperture radar images based on PCANet, GRSL, 2016. [paper] [code, dataset] | Matlab |
| Ms-CapsNet | CNN (Ms-CapsNet); Capsule; Attention Mechanism; Adaptive Fusion Convolution; SAR | Change detection in SAR images based on multiscale capsule network, GRSL, 2020. [paper] [code, dataset] | Matlab+Keras2.16 |
| DCNet | CNN; Unsupervised; Pre-Classification; SAR | Change detection from synthetic aperture radar images based on channel weighting-based deep cascade network, JSTARS, 2019. [paper] [code, dataset] | Caffe |
| ChangeNet | CNN; Siamese; StreetView | ChangeNet: a deep learning architecture for visual change detection, ECCV, 2018. [paper] [code, dataset] | Pytorch |
| Others will be added soon! | |||
2.2 传统变化检测方法
2.2 Available codes for traditional methods
| Methods | Keywords | Publication | Implementation |
|---|---|---|---|
| Several Classical Methods | CVA; DPCA; Image Differencing; Image Ratioing; Image Regression; IR-MAD; MAD; PCAkMeans; PCDA; KMeans; OTSU; Fixed Threshold | A toolbox for remote sensing change detection. [code] | Matlab |
| Matlab Toolbox Change Detection | IR-MAD; IT-PCA; ERM; ICM | A toolbox for unsupervised change detection analysis, IJRS, 2016.[paper] [code] | Matlab |
| RFR,SVR,GPR | Unsupervised; Image Regression; Heterogeneous; Optical RS; SAR | Unsupervised image regression for heterogeneous change detection, TGRS, 2019. [paper] [code] | Matlab |
| HPT | Unsupervised; Transformation; Heterogeneous; Optical RS; SAR | Change detection in heterogenous remote sensing images via homogeneous pixel transformation, TIP, 2018. [paper] [re-implementation code] | Matlab |
| kCCA | Canonical Correlation Analysis; Cross-Sensor; Optical RS | Spectral alignment of multi-temporal cross-sensor images with automated kernel correlation analysis, IJPRS, 2015. [paper] [code] | Matlab |
| Ker. Diff. RBF | Unsupervised; K-means; Optical RS | Unsupervised change detection with kernels, GRSL, 2012. [paper] [code] | Matlab |
| FDA-RM | DI-based; Frequency-Domain Analysis; Random Multigraphs; SAR | Synthetic aperture radar image change detection based on frequency domain analysis and random multigraphs, JARS, 2018. [paper] [code] | Matlab |
| CD-NR-ELM | DI-based; Pre-Classification; Extreme Learning Machine; SAR | Change detection from synthetic aperture radar images based on neighborhood-based ratio and extreme learning machine, JARS, 2016. [paper] [code, dataset] | Matlab |
| None | Likelihood Ratio; Test Statistic; SAR | Change detection in polarimetric SAR images, 2015. [report] [code] | Python |
| PCA K-Means | Unsupervised; DI-based; PCA; K Means; Optical RS | Unsupervised Change Detection in Satellite Images Using Principal Component Analysis and k-Means Clustering, GRSL, 2009. [paper] [re-implementation code, dataset] or [re-implementation code] | Matlab |
| PTCD | Tensor; Hyperspectral Optical RS | Three-Order Tucker Decomposition and Reconstruction Detector for Unsupervised Hyperspectral Change Detection. JSTARS, 2021. [paper] [code, dataset] | Matlab |
| GBF-CD | Data Fusion; Graph; EM; KI; | Graph-Based Data Fusion Applied to: Change Detection and Biomass Estimation in Rice Crops. Remote Sensing, 2020 [paper] [code, dataset] | Matlab |
| Others will be added soon! | |||
3. Open datasets **
Table 3. CD开源数据集。
| Type | Data set | Description |
|---|---|---|
| Optical RS | DSIFN Dataset [25] | 6 bi-temporal high resolution images from Google Earth. There are 3600 image pairs with size of 512 × 512 for training, 340 for validation, and 48 for test. [Download] |
| S2MTCP [26] | 1520 Sentinel-2 level 1C image pairs focused on urban areas around the world, with 10m spatial resolution and the size of 600×600 pixels. Geometric or radiometric corrections are not performed. [Download] | |
| SYSU-CD [27] | 20000 pairs of 0.5-m aerial images of size 256×256 taken between the years 2007 and 2014 in Hong Kong, including 6 change types: (a) newly built urban buildings; (b) suburban dilation; (c) groundwork before construction; (d) change of vegetation; (e) road expansion; (f) sea construction. [Download] | |
| S2Looking [28] | Building change detection dataset consists of 5000 registered bitemporal image pairs (size of 1024*1024, 0.5 ~ 0.8 m/pixel) of rural areas throughout the world and more than 65,920 annotated change instances, separately indicating the newly built and demolished building [Download] | |
| Synthetic and real images Dataset [29] | The database contains 12,000 triples of synthetic images without object shift, 12,000 triples of model images with object shift and 16,000 triples of fragments of real remote sensing images. Performed tests have shown that the proposed CNN is promising and efficient enough in change detection on synthetic and real images [Download] | |
| SEmantic Change detectiON Dataset (SECOND) [24] | a pixel-level annotated semantic change detection dataset, including 4662 pairs of aerial images with 512 x 512 pixels from several platforms and sensors, covering Hangzhou, Chengdu, and Shanghai. It focus on 6 main land-cover classes, i.e. , non-vegetated ground surface, tree, low vegetation, water, buildings and playgrounds , that are frequently involved in natural and man-made geographical changes. [Download] | |
| Hyperspectral change detection dataset [1] | 3 different hyperspectral scenes acquired by AVIRIS or HYPERION sensor, with 224 or 242 spectral bands, labeled 5 types of changes related with crop transitions at pixel level. [Download] | |
| River HSIs dataset [2] | 2 HSIs in Jiangsu province, China, with 198 bands, labeled as changed and unchanged at pixel level. [Download] | |
| HRSCD [3] | 291 co-registered pairs of RGB aerial images, with pixel-level change and land cover annotations, providing hierarchical level change labels, for example, level 1 labels include five classes: no information, artificial surfaces, agricultural areas, forests, wetlands, and water. [Download] | |
| WHU building dataset [4] | 2-period aerial images containing 12,796 buildings, provided along with building vector and raster maps. [Download] | |
| SZTAKI Air change benchmark [5,6] | 13 aerial image pairs with 1.5 m spatial resolution, labeled as changed and unchanged at pixel level. [Download] | |
| OSCD [7] | 24 pairs of multispectral images acquired by Sentinel-2, labeled as changed and unchanged at pixel level. [Download] | |
| Change detection dataset [8] | 4 pairs of multispectral images with different spatial resolutions, labeled as changed and unchanged at pixel level. [Download] | |
| MtS-WH [9] | 2 large-size VHR images acquired by IKONOS sensors, with 4 bands and 1 m spatial resolution, labeled 5 types of changes (i.e., parking, sparse houses, residential region, and vegetation region) at scene level. [Download] | |
| ABCD [10] | 16,950 pairs of RGB aerial images for detecting washed buildings by tsunami, labeled damaged buildings at scene level. [Download] | |
| xBD [11] | Pre- and post-disaster satellite imageries for building damage assessment, with over 850,000 building polygons from 6 disaster types, labeled at pixel level with 4 damage scales. [Download] | |
| AICD [12] | 1000 pairs of synthetic aerial images with artificial changes generated with a rendering engine, labeled as changed and unchanged at pixel level. [Download] | |
| Database of synthetic and real images [13] | 24,000 synthetic images and 16,000 fragments of real season-varying RS images obtained by Google Earth, labeled as changed and unchanged at pixel level. [Download] | |
| LEVIR-CD [14] | 637 very high-resolution (VHR, 0.5m/pixel) Google Earth (GE) image patch pairs with a size of 1024 × 1024 pixels and contains a total of 31,333 individual change building instances, labeled as changed and unchanged at pixel level. [Download] | |
| Bastrop fire dataset [21] | 4 images acquired by different sensors over the Bastrop County, Texas (USA). It is composed by a Landsat 5 TM as the pre-event image and a Landsat 5 TM, a EO-1 ALI and a Landsat 8 as post-event images, labeled as changed and unchanged at pixel level, mainly caused by wildfire. [Download] | |
| Google data set [23] | 19 season-varying VHR images pairswith 3 bands of red, green, and blue, a spatial resolution of 0.55 m, and the size ranging from 1006×1168 pixels to 4936×5224 pixels. The image changes include waters, roads, farmland, bare land, forests, buildings, ships, etc. Buildings make up the main changes. acquired during the periods between 2006 and 2019, covering the suburb areas of Guangzhou City, China. [Download] | |
| Optical RS & SAR | California dataset [22] | 3 images, including a RS image captured by Landsat 8 with 9 channels on 2017, a SAR image captured by Sentinel-1A (recorded in polarisations VV and VH) after the occurrence of a flood, and a ground truth map. [Download] |
| Homogeneous CD Dataset [30] | 6 scenarios: Scenario 1 with two single-polarizationSAR data sets; Scenario 2 with two PolSAR data sets; Scenario 3 with two optical image data sets. HeterogeneousCD: Scenario 4 with two SAR/optical (multispectral) datasets; Scenario 5 with two multispectral data sets of differentbands acquired from different sensors; Scenario 6 with twoPolSAR/optical (multispectral) data sets. [Download] | |
| Street view | VL-CMU-CD [15] | 1362 co-registered pairs of RGB and depth images, labeled ground truth change (e.g., bin, sign, vehicle, refuse, construction, traffic cone, person/cycle, barrier) and sky masks at pixel level. [Download] |
| PCD 2015 [16] | 200 panoramic image pairs in “TSUNAMI” and “GSV” subset, with the size of 224 × 1024 pixels, label as changed and unchanged at pixel level. [Download] | |
| Change detection dataset [17] | Image sequences of city streets captured by a vehicle-mounted camera at two different time points, with the size of 5000 × 2500 pixels, labeled 3D scene structure changes at pixel level. [Download] | |
| CV | CDNet 2012 [18] | 6 video categories with 4 to 6 videos sequences in each category, and the groundtruth images contain 5 labels namely: static, hard shadow, outside region of interest, unknown motion (usually around moving objects, due to semi-transparency and motion blur), and motion. [Download] |
| CDNet 2014 [19,20] | 22 additional videos (∼70; 000 pixel-wise annotated frames) spanning 5 new categories that incorporate challenges encountered in many surveillance settings, and provides realistic, camera captured (without CGI), diverse set of indoor and outdoor videos like the CDnet 2012. [Download] | |
| ChangeSim [31] | a challenging dataset aimed at online scene change detection and more, collecting in photo-realistic simulation environments with the presence of environmental non-targeted variations, such as air turbidity and light condition changes, as well as targeted object changes in industrial indoor environments. [Download] | |
| More video datasets | ||
It can be seen that the amount of open datasets that can be used for change detection tasks is small, and some of them have small data sizes. At present, there is still a lack of large SAR datasets that can be used for AI training. Most AI-based change detection methods are based on several SAR data sets that contain limited types of changes, e.g., the Bern dataset, the Ottawa dataset, the Yellow River dataset, and the Mexico dataset, which cannot meet the needs of change detection in areas with complex land cover and various change types. Moreover, their labels are not freely available. Street-view datasets are generally used for research of AI-based change detection methods in computer vision (CV). In CV, change detection based on pictures or video is also a hot research field, and the basic idea is consistent with that based on RS data. Therefore, in addition to street view image datasets, several video datasets in CV can also be used for research on AI-based change detection methods, such as CDNet 2012 and CDNet 2014.
4. Applications
AI-based变化检测技术的发展极大地促进了许多应用，提高了它们的自动化和智能性。大多数AI-based变化检测生成的是二进制映射，而这些研究只关注算法本身，没有特定的应用领域。因此，可以认为它们一般适用于LULC变化检测。在本节中，我们将重点介绍与特定应用相关的技术，这些技术大致可分为四类：
- 城市环境：城市扩展、公共空间管理和建筑变化检测；
- 资源与环境：human-driven环境变化，hydro-environmental变化，海冰、地表水、森林监测；
- 自然灾害：滑坡测绘和破坏评估；
- 天文学：行星表面。
Table 4. Summary of main applications of AI-based change detection techniques.
| Applications | Data Types | Papers | |
|---|---|---|---|
| Urban contexts | Urban expansion | Satellite images | Lyu et.al (2018),Tong et.al (2007) |
| SAR images | Iino et.al (2017) | ||
| Public space management | Street view images | Varghese et.al (2018) | |
| Road surface | UAV images | Truong et.al (2020) | |
| Building change detection | Aerial images | Ji et.al (2019),Sun et.al (2019),Nemoto et.al (2017) | |
| Satellite images | Huang et.al (2019),Zhu et.al (2018) | ||
| Satellite/Aerial images | Jiang et.al (2020),Ji et.al (2018),Saha et.al (2020) | ||
| Airborne laser scanning data and aerial images | Zhang et.al (2019) | ||
| SAR images | Jaturapitpornchai et.al (2019) | ||
| Satellite images and GIS map | Ghaffarian et.al (2019) | ||
| Resources & environment | Human-driven environmental changes | Satellite images | Chen et.al (2016) |
| Hydro-environmental changes | Satellite images | Nourani et.al (2018) | |
| Sea ice | SAR images | Gao et.al (2019),Gao et.al (2019) | |
| Surface water | Satellite images | Song et.al (2019),Rokni et.al (2015) | |
| Forest monitoring | Satellite images | Khan et.al (2017),Lindquist et.al (2016),Deilmai et.al (2014),Woodcock et.al (2001),Gopal et.al (1996) | |
| Natural disasters | Landslide mapping | Aerial images | Fang et.al (2020),Lei et.al (2019) |
| Satellite images | Chen et.al (2018),Ding et.al (2016),Tarantino et.al (2006) | ||
| Damage assessment | Satellite images | caused by tsunami [Sublime et.al (2019),Singh et.al (2015)], particular incident [Hedjam et.al (2019)], flood [Peng et.al (2019)], or earthquake [Ji et.al (2019)] | |
| Aerial images | caused by tsunami [Fujita et.al (2017)] | ||
| SAR images | caused by fires [Planinšič et.al (2018)], or earthquake [Saha et.al (2018)] | ||
| Street view images | caused by tsunami [Sakurada et.al (2015)] | ||
| Street view images and GIS map | caused by tsunami [Sakurada et.al (2017)] | ||
| Astronomy | Planetary surfaces | Satellite images | Kerner et.al (2019) |
5. Software programs
Table 5. CD工具。
Table 5. A list of software for change detection.
| Type | Name | Description |
|---|---|---|
| Commercial | ERDAS IMAGINE | provides true value, consolidating remote sensing, photogrammetry, LiDAR analysis, basic vector analysis, and radar processing into a single product, including a variety ofchange detection tools. |
| ArcGIS | change detection can be calculate between two raster datasets by using theraster calculator toolordeep learning workflow. | |
| ENVI | provideschange detection analysis toolsand theENVI deep learning module. | |
| eCognition | can be used fora variety of change mapping, and by leveraging deep learning technology from the Google TensorFlow™ library, eCognition empowers customers with highly sophisticated pattern recognition and correlation tools that automate the classification of objects of interest for faster and more accurate results,more. | |
| PCI Geomatica | provideschange detection tools, and can be useful in numerous circumstances in which you may want to analyze change, such as: storm damage, forest-fire damage, flooding, urban sprawl, andmore. | |
| SenseTime | SenseRemote remote sensing intelligent solutions | |
| Open source | QGIS | provides manychange detection tools. |
| Orfeo ToolBox | change detection bymultivariate alteration detector (MAD) algorithm. | |
| Change Detection ToolBox | MATLAB toolbox for remote sensing change detection. |
6. Review papers for change detection
The following papers are helpful for researchers to better understand this field of remote sensing change detection, see table 6.
Table 6. A list of review papers on change detection.
| Published year | Review paper |
|---|---|
| 1989 | Digital change detection techniques using remotely sensed data, IJRS. [paper] |
| 2004 | Digital change detection methods in ecosystem monitoring: a review, IJRS. [paper] |
| 2004 | Change detection techniques, IJRS. [paper] |
| 2012 | Object-based change detection, IJRS. [paper] |
| 2013 | Change detection from remotely sensed images: From pixel-based to object-based approaches, ISPRS. [paper] |
| 2016 | 3D change detection–approaches and applications, ISPRS. [paper] |
| 2016 | Deep learning for remote sensing data a technical tutorial on the state of the art, MGRS. [paper] |
| 2017 | Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community, JRS. [paper] |
| 2017 | Deep Learning in Remote Sensing, MGRS. [paper] |
| 2018 | Computational intelligence in optical remote sensing image processing, ASOC. [paper] |
| 2019 | A review of change detection in multitemporal hyperspectral images: current techniques, applications, and challenges, MGRS. [paper] |
| 2019 | Deep learning in remote sensing applications: A meta-analysis and review, ISPRS. [paper] |
| 2020 | Deep Learning for change detection in remote sensing images: comprehensive review and meta-analysis, arXiv. [paper] |
| 2020 | Change detection based on artificial intelligence: state-of-the-art and challenges, RS. [paper] |
7. Reference
[1] Hyperspectral Change Detection Dataset. Available online:Hyperspectral Change Detection Dataset | Centro Singular de Investigación en Tecnoloxías Intelixentes – CiTIUS(accessed on 4 May 2020).
[2] Wang, Q.; Yuan, Z.; Du, Q.; Li, X. GETNET: A General End-to-End 2-D CNN Framework for Hyperspectral Image Change Detection. IEEE Trans. Geosci. Remote Sens. 2018, 57, 3–13. [Google Scholar] [CrossRef]
[3] Daudt, R.C.; Le Saux, B.; Boulch, A.; Gousseau, Y. Multitask learning for large-scale semantic change detection. Comput. Vis. Image Underst. 2019, 187, 102783. [Google Scholar] [CrossRef]
[4] Ji, S.; Wei, S.; Lu, M. Fully Convolutional Networks for Multisource Building Extraction from an Open Aerial and Satellite Imagery Data Set. IEEE Trans. Geosci. Remote Sens. 2018, 57, 574–586. [Google Scholar] [CrossRef]
[5] Benedek, C.; Sziranyi, T. Change Detection in Optical Aerial Images by a Multilayer Conditional Mixed Markov Model. IEEE Trans. Geosci. Remote Sens. 2009, 47, 3416–3430. [Google Scholar] [CrossRef]
[6] Benedek, C.; Sziranyi, T. A Mixed Markov model for change detection in aerial photos with large time differences. In Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA, 8–11 December 2008; pp. 1–4. [Google Scholar]
[7] Daudt, R.C.; Le Saux, B.; Boulch, A.; Gousseau, Y. Urban change detection for multispectral earth observation using convolutional neural networks. In Proceedings of the IGARSS 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 2115–2118. [Google Scholar]
[8] Zhang, M.; Shi, W. A Feature Difference Convolutional Neural Network-Based Change Detection Method. IEEE Trans. Geosci. Remote Sens. 2020, 1–15. [Google Scholar] [CrossRef]
[9] Wu, C.; Zhang, L.; Zhang, L. A scene change detection framework for multi-temporal very high resolution remote sensing images. Signal Process. 2016, 124, 184–197. [Google Scholar] [CrossRef]
[10] Fujita, A.; Sakurada, K.; Imaizumi, T.; Ito, R.; Hikosaka, S.; Nakamura, R. Damage detection from aerial images via convolutional neural networks. In Proceedings of the 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), Nagoya Univ, Nagoya, Japan, 08–12 May 2017; pp. 5–8 [Google Scholar]
[11] Gupta, R.; Goodman, B.; Patel, N.; Hosfelt, R.; Sajeev, S.; Heim, E.; Doshi, J.; Lucas, K.; Choset, H.; Gaston, M. Creating xBD: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 16–20 June 2019; pp. 10–17. [Google Scholar]
[12] Bourdis, N.; Marraud, D.; Sahbi, H. Constrained optical flow for aerial image change detection. In Proceedings of the 2011 IEEE International Geoscience and Remote Sensing Symposium, Vancouver, BC, Canada, 24–29 July 2011; pp. 4176–4179. [Google Scholar] [CrossRef]
[13] Lebedev, M.A.; Vizilter, Y.V.; Vygolov, O.V.; Knyaz, V.A.; Rubis, A.Y. Change detection in remote sensing images using conditional adversarial networks. ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2018, 565–571. [Google Scholar] [CrossRef]
[14] Chen, H.; Shi, Z. A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection. Remote Sensing, 12(10), 1662. [Google Scholar] [CrossRef]
[15] Alcantarilla, P.F.; Stent, S.; Ros, G.; Arroyo, R.; Gherardi, R. Street-view change detection with deconvolutional networks. Auton. Robot. 2018, 42, 1301–1322. [Google Scholar] [CrossRef]
[16] Sakurada, K.; Okatani, T. Change detection from a street image pair using CNN features and superpixel segmentation. In Proceedings of the British Machine Vision Conference (BMVC), Swansea, UK, 7–10 September 2015; pp. 61.1–61.12. [Google Scholar]
[17] Sakurada, K.; Okatani, T.; Deguchi, K. Detecting changes in 3D structure of a scene from multi-view images captured by a vehicle-mounted camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 137–144. [Google Scholar]
[18] Goyette, N.; Jodoin, P.-M.; Porikli, F.; Konrad, J.; Ishwar, P. Changedetection. net: A new change detection benchmark dataset. In Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA, 16–21 June 2012; pp. 1–8. [Google Scholar]
[19] Wang, Y.; Jodoin, P.-M.; Porikli, F.; Konrad, J.; Benezeth, Y.; Ishwar, P. CDnet 2014: An Expanded Change Detection Benchmark Dataset. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA, 23–28 June 2014; pp. 393–400. [Google Scholar]
[20] Goyette, N.; Jodoin, P.-M.; Porikli, F.; Konrad, J.; Ishwar, P. A Novel Video Dataset for Change Detection Benchmarking. IEEE Trans. Image Process. 2014, 23, 4663–4679. [Google Scholar] [CrossRef]
[21] Volpi, Michele; Camps-Valls, Gustau; Tuia, Devis (2015). Spectral alignment of multi-temporal cross-sensor images with automated kernel canonical correlation analysis; ISPRS Journal of Photogrammetry and Remote Sensing, vol. 107, pp. 50-63, 2015. [CrossRef]
[22] L. T. Luppino, F. M. Bianchi, G. Moser and S. N. Anfinsen. Unsupervised Image Regression for Heterogeneous Change Detection. IEEE Transactions on Geoscience and Remote Sensing. 2019, vol. 57, no. 12, pp. 9960-9975. [CrossRef]
[23] D. Peng, L. Bruzzone, Y. Zhang, H. Guan, H. Ding and X. Huang, SemiCDNet: A Semisupervised Convolutional Neural Network for Change Detection in High Resolution Remote-Sensing Images. IEEE Transactions on Geoscience and Remote Sensing. 2020. [CrossRef]
[24] Yang, Kunping, et al. Asymmetric Siamese Networks for Semantic Change Detection. arXiv preprint arXiv:2010.05687 (2020). [CrossRef]
[25] Zhang, C., Yue, P., Tapete, D., Jiang, L., Shangguan, B., Huang, L., & Liu, G. A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing. 2020. [CrossRef]
[26] LEENSTRA, Marrit, et al. Self-supervised pre-training enhances change detection in Sentinel-2 imagery. arXiv. 2021. [CrossRef]
[27] SHI, Qian, et al. A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection. IEEE Transactions on Geoscience and Remote Sensing. 2021. [CrossRef]
[28] SHEN, Li, et al. S2Looking: A Satellite Side-Looking Dataset for Building Change Detection. arXiv. 2021. [CrossRef]
[29] LEBEDEV, M. A., et al. CHANGE DETECTION IN REMOTE SENSING IMAGES USING CONDITIONAL ADVERSARIAL NETWORKS. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2018. [CrossRef]
[30] SUN, Yuli, et al. Structure Consistency-Based Graph for Unsupervised Change Detection With Homogeneous and Heterogeneous Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2021. [CrossRef]
[31] PARK, Jin-Man, et al. ChangeSim: Towards End-to-End Online Scene Change Detection in Industrial Indoor Environments. arXiv. 2021. [CrossRef]
Reference materials:
- I-Hope-Peace/ChangeDetectionRepository
- Michele Volpi personal research page
- llu025/Heterogeneous_CD
- wenhwu/awesome-remote-sensing-change-detection
- neverstoplearn/remote_sensing_change_detection
- Change Detection in GIS
- Gao Feng personal research page
- Bobholamovic/ChangeDetectionToolbox