ESA-sponsored FDL Europe yields inspiring results in AI space applications

At a recent showcase event, FDL Europe presented the fruits of this year’s summer research sprint. With technical and financial support from ESA Φ-lab, the team demonstrated how digital twins and foundation models can enable solutions in the areas of early detection of solar events, data gathering for disaster management, and facilitating downstream applications based on Synthetic Aperture Radar (SAR) data.

The Frontier Development Lab (FDL) Europe is a collaboration that works with ESA and a number of commercial partners such as Google Cloud and NVIDIA. The initiative focuses on accelerated research that applies artificial intelligence (AI) to space science in order to push frontiers and develop new tools that help solve major human challenges.

Φ-lab spearheads the collaboration within ESA, giving strategic and expert input into FDL Europe’s annual development cycle and individual projects. Each year, FDL Europe puts machine learning specialists together with Earth and planetary scientists for an intensive summer research sprint, drawing on bold thinking and rapid iteration and prototyping to produce stimulating outcomes.

For the 2023 run, the FDL Europe experts’ focus was heavily driven by the power of emerging AI technologies, as Head of the Φ-lab Explore Office Pierre Philippe Mathieu explains: “Foundation and large language models are examples of exciting new methodologies that are radically changing the way we – and machines – retrieve and interface with data. Part of FDL Europe’s remit is to apply these techniques safely and ethically for the benefit of humanity, and the three challenges this year were selected for their potentially wide-ranging impact in protecting our planet.”

Challenge 1: creating an SSA twin for space weather

The first FDL Europe team was tasked with creating a Space weather Situational Awareness (SSA) twin for solar events. Early detection of space weather is crucial for safeguarding both terrestrial and orbiting infrastructure, an endeavour that ESA’s Vigil mission will take on by providing an unprecedented view of the sun from its orbit at the fifth Lagrange point. But understanding the propagation of coronal mass ejections (CMEs) is complex, and Vigil’s distance from the Earth will limit its connectivity.

The team on this challenge developed a machine learning pipeline that leverages the advantages of edge computing in space with a CME-aware compression model, significantly reducing the downlink bandwidth requirements and the delay in alerting dangerous events. On the ground, their SSA twin can provide a three-dimensional reconstruction of solar activity through a combination of physics-based equations and Neural Radiance Fields, a deep learning method for synthesising 3D views of complex scenes by optimising an underlying continuous volumetric scene function using a sparse set of 2D input views. The result amply attests to the power of onboard intelligence paired with digital models.

Challenge 2: building a foundation model adaptor for disasters

As the recent growth in AI chatbots has shown, Large Language Models (LLMs) are having an enormous effect on society as a whole, but their ability to act as a knowledge store, summarise a large array of inputs and adapt to unseen tasks is also profoundly influencing software development. The second FDL team took on the challenge of harnessing the possibilities of LLMs for collating and interpreting information on current and past disasters, providing reliable and detailed reporting to authorities.

The team chose to focus on floods as one of the most prevalent and damaging classes of extreme weather events. While flood relief decision-making can often be hampered by the time and resources required to bring all the relevant insights together, the FDL team’s foundation model combines Earth observation (EO) data, flood prediction and mapping models, human-in-the-loop inputs and weather monitoring systems to create accurate, instantaneous flood reports through natural language processing. A prototype of the interface can be seen at floodbrain.com.

Challenge 3: a foundation model for analysing SAR EO data

With its all-weather capabilities, SAR remote sensing is an essential tool for Earth observation. However, in addition to the extensive preprocessing required, SAR data is difficult for deep learning to adapt to, and existing deep learning models struggle both to remain robust and to function reliably over new regions or different time frames.

The third FDL Europe team sought to develop SAR foundation models that could be generalised to a multitude of analysis tasks. The resulting pipeline was flexible enough to handle the huge breadth of data from SAR imagery and was also fine-tuned on a variety of labelled downstream tasks, including processing complex data and extending models across time and space domains. As a forerunner of a full foundation model for SAR, this pipeline has pioneered the way for a potentially transformative asset for the data-science community.

Giuseppe Mandorlo is the Project Manager for Vigil at ESA and so naturally was following the development of Challenge 1 particularly closely: “The results achieved by the FDL Europe Heliophysics team are extremely impressive – even more so when you take into account the short duration of the sprint. Some of the findings are truly astounding and could influence numerous aspects of the Vigil Project, from the selection of instrument imaging detector bands right up to the design of the overall Space Weather Network architecture.”

“All three of these challenges can be seen as precursor projects that lay the groundwork for the further development of key AI technologies for EO and science more generally,” added Φ-lab data scientist Nicolas Longépé. “The combination of multi-disciplinary research teams and internationally recognised mentors has led to remarkable outcomes and much food for thought, not least the fact that the user community for EO data will now include not only humans but also chatbots.”

A video of the showcase is now available, and the full report will be published on fdleurope.org shortly. Researchers interested in joining FDL Europe 2024 can now register their interest.

To know more: Φ-lab, ESA Vigil mission