Change & Anomaly Detection
The central task of detecting and identifying disruptions of critical infrastructures requires automated and robust methods for distinction between the desired condition and abnormal states. These methods need to be robust to seasonal or weather-related changes (fallen leaves, rain, snow) to ensure operation regardless of the geographic zone or season, and to reduce the rate of false alarms (false positives). At the same time, it is crucial that the methods are sensitive to minimize the rate of false negatives, i.e. failures to detect a disruption.The first method is change detection. This step can take place both on- and off-board, depending on the available resources, on the mission requirements for the timeliness of product generation (e.g., in real time on-board, or off-board upon user request or if bandwidth is available), and on the amount of data to be processed. The change detection step assumes the RSI and the current image (post-change, L-1 processing completed) to be available. Optimal accuracy requires co-registration with the RSI in order to make it superimposable to its corresponding spatial footprint in the RSI. Considering the rather innovative framework
(Ka-band, spatial resolution <=50 cm, on-/off-board processing, < 1h processing time), two kinds of artificial-intelligence-based approaches will be evaluated. The first one to be considered fast, based on CD thresholding techniques, with low computing requirements but possibly decreased accuracy; and the second one computationally more intensive, exploiting Deep Learning Neural Network architectures aimed at achieving high accuracies. This kind of CD methods exists in the state-of-the-art, but are designed for different applications and SAR sensors.
Given the limited resources available on-board it is beneficial to apply a simple, yet possibly effective, technique based on statistical thresholding of SAR change indices such as the ratio or the logratio images and semantic classification of linear features, involving strategies to deal with speckle noise contamination. However, semantic segmentation of linear features using SAR imagery is challenging since images typically differ with ground sample distance upon different overpasses. When standard CD methods are inadequate to detect multiple linear targets, deep learning provides more insight about features for better discrimination of linear infrastructures like railways.
An unsupervised deep learning model automatically learns semantic features using an (un)labelled training set, which might be composed of samples from the RSI only, or multitemporal pre-/post-change pairs. DL features are particularly suitable in modelling the challenging spatial context information in VHR images. For both approaches an analysis is required to understand applicability/adaptation to the infrastructure monitoring application, Ka band SAR data and VHR data (<50cm).
Contrasting change detection, which draws conclusions from comparing the present state to past states, anomaly detection identifies data with features not explainable by the probability distribution characterizing the desired state, i.e. an undisrupted infrastructure. While such outliers are easily found in true low-dimensional data, their reliable identification from images is challenging. This is because they need not be outliers with regard to the immediate properties of the image (such as histogram data), but analysis of the context of the image (a tree is not abnormal in an image per se; however, it is abnormal if it is lying on railway tracks).
As described above, deep-learning based methods excel at extracting and generalizing such features from data, and therefore they are ideally suited for the anomaly detection task. Different approaches facilitate this, including semi-supervised methods that train a neural network on an anomaly/obstacle-free data set (e.g. the RSI) by reconstructing samples from abstracted features. Anomalous samples are identified as those that cannot be correctly reconstructed. As stated before, the main task is to detect obstacles on or disruptions of railway tracks as the sensor passes over the very same. This may be achieved by detecting changes between SAR images and previously recorded SAR imagery of the tracks in question. However, it may also be achieved by segmenting aerial images acquired by VIS cameras by detecting railway tracks in those images. Such a segmentation may be used for track discontinuity detections and for comparison with railway maps to detect inconsistencies with those maps.
Both types of detections are features of obstacles or disruptions of railway tracks. Change to: Optical cameras and SAR have different sensing characteristics, e.g. active vs. passive illumination, different wavelengths and thus different interaction with the atmosphere as well as objects on ground, different fields of view, and very different sensing geometries. SAR pixel coordinates correspond to the sensor position along its trajectory and the range to the reflecting surface respectively, whereas the pixels of optical camera correspond to the angles of the rays through the camera centre and the reflecting points. Therefore, the two different types of sensors will likely detect different railway obstructions with different probabilities.
In IIMEO, we will exploit this complementarity by fusing data of both types of sensors into a joint map of obstacles and disruptions of railway tracks. We will put particular emphasis on handling conflicting sensor information, e.g. an obstacle being detected in optical images but not in the SAR image.
Data will be fused using two different strategies. The first is to produce independent obstacle feature maps per sensor, each spatially encoding the probability of the presence of a disruption or an obstacle. These maps are then fused into a map of obstacles using standard probabilistic techniques, e.g. "noisy or" operations or Generalized Linear Models. We think this approach will lead to results superior to using SAR data only, which are also relatively easy to explain. At the end of IIMEO, this sensor data fusion approach will have reached TRL 5.
The second approach to fuse SAR data and aerial images will be to use a neural network (CNN), which jointly processes both SAR data and images directly into on obstacle map, skipping the intermediate, per-sensor maps. In contrast to the first approach, this may find and exploit correlations of lower-level image features, potentially further improving the accuracy of the results. This network will be both trained and validated using the results of the first approach, however, it will also be harder to inspect upon the potential occurrence of odd results and thus will remain experimental during IIMEO.
The project proposes an end-to-end system for instantaneous monitoring of infrastructures. It includes elements from the acquisition platform to the delivery of final product. In this context the proposal focuses more on the novel SAR sensor as acquisition system (no seen before spatio-temporal resolution at that acquisition band) and railways as application. We will test the capabilities for "real time" on-board processing chain for one application on 2 chains (VIS and SAR) thus demonstrating the project and contributing to literature with a processing chain for Ka-band 50cm resolution change detection.
General on-board processing for satellites has been already proven in a lot of satellites. After acquisition, data is streamed from the SAR pre-processor responsible for SAR image formation to the on-board-processing hardware unit at a small but consistent time delay.
Time for further processing and the change detection as well as anomaly detection will be estimated within this project. We will adapt or choose candidate algorithms for the on-board processor to be in line with the planned time of the fast availability of one hour. Once the data has been processed from the on-board processor, we would have multiple options for providing the data to the pilot user. The data can either be directed to the cloud infrastructure for further processing, enhancements and visualisations or directly provided for the customer resulting in a faster availability of the data.
Direct downlink of the data from a potential satellite to antenna like X-band antenna on ground is the most common way for satellite communication. To give an example one possible antenna among many X band antennas from DLR in Germany. This antenna’s provides visibility of satellites over Europe being able to direct downlink captures images for our use case for Serbia. Therefore, a potential downlink can be established from the satellite to the DLR antenna and through terrestrial network to the cloud infrastructure and/or the pilot customer.
The latency of a LEO satellite to the antenna is about 10-15 ms. A data size after on-board processing of roughly 100mb the downlink will take an estimated maximum of 1 second for X-Band for a nanosatellite if the connection is established. The data rate necessary fits perfectly to microsat X-Band New space transmitters as the SYRLINKS XONOS transmitter. In X-Band, with DVB-S2 and 350Mhz bandwidth, the possible data rates are between 500 and 1000 Mbps. With the on board processing, including masking of not-on-interest areas, the necessary data rate can be reduced at min with an factor of 10, which results on required downlink rates of 100-500 Mbps, which is fulfilled by current available hardware and current ITU regulation for X-Band data downlink. The latency of a terrestrial network to Serbia (about 1000 km) is about 15 ms with a much higher data rate. Therefore, the overall delay from signal generation to the pilot user potentially should take less than a few seconds.
For a satellite constellation the fast availability of data can be archived through a connection to a telecommunication satellite constellation like Inmarsat or Starlink. These constellations are able to receive a signal from a satellite and route the data to fixed antennas or mobile antenna directly at customer's premise. Through intersatellite Links these constellations are able route the data to a satellite with active contact to an antenna and therefore no delay with respect to contact times of antennas are existing. Starlink is expected to provide have a latency of 20ms to 40ms. With a mentioned data rate of 50 to 150 MB/s this option should take less than a few seconds as well. But this approach is only necessary for areas/customer outside Europe, because inside Europe a typical LEO (800 km) microsat is in contact range of a German Groundstation (e.g. Neustrelitz) from south of Egypt till north Norway (typical 3000 to 4000 km radius around the station). Within this Project European customers are baseline of the business case.
Therefore the standard option is to downlink the data from the satellite directly to an antenna network on ground receiving and redirecting the data to the pilot user via internet for fast availability. The so-called user data ground station as a service provide antenna all around the world with different Communication bands. X-Band data downlink is standard for all station participating in ESA projects (e.g. SENTINEL). As the satellite needs to be in sight of the antenna for a downlink the available antennas in the region of interest depending on the service need further investigation, but typical receiving radius is up to 4000 km around the ground station (pending min elevation angle of the antenna).
Which groundstation suites best for this use case will be elaborated in the satellite conception phase, but most likely it’s a combination of different X-Band stations which are also used for Sentinel (and offers commercial service), as Matera (Italy), Neustrelitz (Germany), Maspalomas (Spain) and Svalbard. During the conception phase an end-to-end analysis will be conducted for our concept of the fast availability, taking into account a common timing for space to ground communication with standardised data rates. Overall, the time to the pilot user is mostly determined by the change detection.
Within the demonstration via flight campaign during project execution it is planned to use a wireless connection to connect the airplane for the tests with the internet. This will provide an end-to-end chain from sensor capturing through on-board processing to the pilot user and the cloud infrastructure. Depending on the availability of wireless network in Serbia the best connection link needs further investigation. Since the available data rates are not clear at this state the downlink time cannot be stated right now