ESA title

Φ-lab leads the way for new ChatGPT-style tools for Earth observation

As recently announced, ESA Φ-lab, in conjunction with its technology partners, is leading activities to develop AI foundation models, in a ChatGPT style, aimed at intelligent information retrieval in Earth observation (EO). With the launch of further initiatives exploring large language models, now is a good time to look back at the new and existing work Φ-lab is doing in this field in more detail.

ESA, other space agencies and New Space enterprises operate Earth observation missions for the benefit of science, commerce and society as a whole, but the volume of satellite data available far exceeds the capacity of humans to process and derive actionable insight in a timely manner.

Progress with more traditional AI can however be hampered by the need for a pool of labelled data to train AI models. Foundation models help to circumvent this limitation through generally self-supervised learning from large and varied sources of unlabelled data, in addition to supervised ones that are still necessary. Foundation models also deliver tools that can be adapted to a broad range of tasks, and since their inception in 2018 foundation models have contributed to a huge transformation in machine learning, even leading to chatbots with impressive natural language capabilities and several other emerging properties.

Φ-lab has a proven pedigree in disruptive innovation in Earth observation, with a particular focus on AI4EO and innovative computation paradigms. As covered in an article in March, given the enormous potential of foundation models for rapid, self-supervised learning, Φ-lab is undertaking various initiatives to create foundation models exploiting EO and remote sensing datasets.

The PhilEO project has been running for over a year. Developed by Φ-lab in conjunction with e-GEOS and Leonardo Labs, and exploiting the davinci-1 supercomputer, PhilEO is a geospatial foundation model trained on global Copernicus Sentinel-2 data. The model uses metadata from Sentinel-2 images and is trained to identify geographical features around the Earth, enabling it to learn general features and perform land cover classification, estimation of density and proximity between buildings and road segmentation regression.

In a major milestone, the PhilEO team is now releasing the model itself and associated resources to further research and testing throughout the EO community. PhilEO Bench, an evaluation benchmark that allows the performance of various models to be compared, can already be found on GitHub, and PhilEO Globe, the Sentinel-2 dataset, has been uploaded to Hugging Face. The code for the model will be available on the Hugging Face page in the coming weeks.

Two new activities supported by Φ-lab have also just been launched. A consortium comprising DLR, FZ Jülich, KP Labs and IBM will develop a European foundation model that is expected to significantly progress the state of the art. This project, which is named FAST-EO (Fostering Advancements in foundation models via unsupervised and Self-supervised learning for downstream Tasks in Earth Observation), will develop a multi-modal foundation model. Incorporating both Sentinel-1 SAR and Sentinel-2 optical, worldwide datasets, the model will integrate natural language capabilities and undergo validation through a range of environmentally critical applications such as methane leaks, biomass estimation and landcover change.

A second initiative has commenced in the last months. Foundation Models for Climate and Society is led by the Norwegian Computing Center, along with various national meteorological offices. This project, which is named FM4CS (Foundation Models for Climate and Society), will develop a foundation model that will focus on climate adaptation and extreme-weather-event mitigation. This enterprise will also benefit from the use of LUMI (Large Unified Modern Infrastructure), a petascale, world-class supercomputer.

AI foundation models serve as the engines of digital assistants, whereby the core processing of the foundation model is integrated with natural language models and interactive user interfaces. The general idea of a digital assistant is for all users – from non-technical to EO experts – to be able to perform a query on EO data archives such as “How many different crop types are in this Sentinel-1 image?”, ask more generic questions linked to EO and Earth science such as “How can EO help to monitor urban heat islands?”

To mature the human-interfacing aspect of a digital assistant, Φ-lab has just launched a new project with Pi School to build an EO Virtual Expert (EOVE). The team is exploring a set of large language models (LLMs), which will be trained and fine-tuned on specific and crafted documents related to EO and Earth science. A web platform with a simple graphical user interface and an application programming interface will be created as the gateway for the trained LLM.

The end game for Φ-lab’s various ventures in foundation models and LLMs is to set a path towards an EO digital assistant that can respond to information and knowledge queries posed in natural language and that will produce reliable, validated content.

“Foundation models are bringing a paradigm shift in AI, thanks to the scaling in training data and model size. These intelligent agents can be adapted to several specific applications and are showing impressive emerging properties, unlocking the potential of AI like never seen before. In this field, LLMs are currently disrupting the way humans interact with intelligent agents via natural language,” comments Head of ESA Φ-lab Division Giuseppe Borghi. “The integration of these models with EO and other heterogenous data will ultimately place a dedicated ChatGPT-style tool at the fingertips of EO end users in many sectors.”

To know more: Φ-lab, Norwegian Computing Center, Pi School

Photo courtesy of Pexels/ThisIsEngineering

Global storm surge forecasting: creating early warning models with AI4EO

ESA Φ-lab, a key point of reference for groundbreaking innovation in Earth observation, was recently invited to present its work on storm surge forecasting to an audience at Google Research. Φ-lab’s initiative follows a state-of-the-art approach, combining deep learning and data fusion of satellite imaging, tide gauge measurements and weather forecasting models. The goal of this enterprise is the optimal prediction of surge-related natural disasters, especially in under-served areas.

According to the United Nations Development Programme, the effects of climate change on coastal flooding will increase up to five times over the century, leaving more than 70 million people in the way of expanding floodplains. Latin America, the Caribbean, the Pacific and Small Island Developing States are expected to be among the most affected areas. Following the UN Sustainable Development Goals framework, ESA supports a rapid and resilient crisis response and Copernicus has its own Emergency Management Service.

Storm surges are one of the most critical climate change consequences. They are ocean dynamics driven by extreme weather, superimposing temporary rises on the mean sea level and causing coastal floods. The short-term prediction of these phenomena is accomplished via tidal gauges – in situ sensors that provide hourly records of sea level changes with high accuracy. They are widely deployed in Europe and in the US, but are sparse in other world regions, especially in developing countries.

As part of its efforts to fight climate change and its impact, ESA Φ-lab, a globally recognised entity in groundbreaking innovation in Earth observation (EO), is conducting research on the use of machine learning to predict storm surges. At the beginning of May, Google Research Flood Forecasting invited Patrick Ebel, an internal research fellow at Φ-lab and lead for the storm surge forecasting project, to give a talk about the recent work and approaches developed in the lab.

To counteract the difficulties presented by traditional prediction methods, this Φ-lab project introduces a global dataset of in situ tidal gauge time series paired with satellite-derived atmospheric and ocean state reanalysis products and global land-sea masks. Neural networks, a machine learning approach designed to recognise patterns, can then assimilate these data to provide forecasts with large spatial coverage. The novelty in this research lies in the generalisation of prediction to locations where tidal gauges are not available, assisting under-served communities with less in situ monitoring infrastructure and aligning with the UN Early Warnings for All initiative goals.

Patrick Ebel presents the work about storm surge forecasting developed at Φ-lab

Google software engineer Oren Gilon was one of the organisers of the session: “We at Google have been very excited to hear about Patrick’s work. Understanding that approaches similar to those that have been applied to riverine floods can be applied to coastal floods changes the way we look at this problem. We hope to find ways to collaborate on this matter in the future.”

The next steps for the project include replacing retrospective reanalysis products with recently developed forecasting models, incorporating data from satellite altimetry, modelling of impact at landfall, and translating storm surges into predictive flood maps. This work will be further discussed at the next IEEE/CVF Computer Vision and Pattern Recognition Conference and MedCyclones Workshop & Training School.

“I would like to thank Google Research for having organised the gathering and for the enthusiastic discussion on the storm surge topic. Our innovative approach to this research brings a fundamental change to the way natural hazards are predicted,” comments Giuseppe Borghi, Head of Φ-lab. “I am keen to see this dialogue between Google and ESA continue, as we work together to address some of our most pressing societal challenges.”

Further details on the research and its initial results can be found on arXiv.

To know more: Φ-lab, Google Research Flood Forecasting

Photo courtesy of Pexels/George Desipris and ESA/Patrick Ebel

Φ-lab gives major impetus to ESA-ECMWF ML4ESOP workshop

With record attendance and a host of new topics covered, the 2024 joint workshop on Machine Learning for Earth System Observation and Prediction (ML4ESOP) has just drawn to a close. Organised by ESA Φ-lab and ECMWF, the workshop explores synergies between machine learning, deep learning and conventional tools as applied to satellite observations and weather and climate models.

Machine learning and deep learning (ML/DL) are already proven technologies for enhancing our understanding of the planet’s complex dynamics, leveraging new methodologies and high-performance computing to process and analyse enormous datasets automatically. The intersection between Earth observation (EO), Earth System Prediction and AI is a point where a number of disciplines can come together and experts from each field can synergistically advance the state of the art.

This forms the guiding principle of the ESA-ECMWF ML4ESOP workshop, which each year gathers a diverse community of practitioners to look at the progress and challenges in the fusion of traditional ESOP techniques with ML/DL. This year’s event was held from 7 to 10 May and boasted almost 800 registrations online and in person, comprising academia, public institutions and industry.

Apart from hosting the 2024 workshop at ESA’s ESRIN establishment, Φ-lab played a fundamental role through its participation in the organising and scientific committees, the chairing of sessions, the presentation of a number of research initiatives and the lively discussions throughout the four days.

ECMWF’s Massimo Bonavita, who organised the event with Φ-lab AI Applications Lead Rochelle Schneider, was impressed by the turnout: “Each year we see a higher number of participants and an ever-broader range of research subject areas. The new and exciting thematic areas that were selected as the framework of the workshop ensured that the key elements and hurdles of ML4ESOP were addressed, with engaging contributions on weather and climate from a truly international cohort of researchers.”

Indeed, this year’s thematic areas (TAs) gave an enticing structure to the proceedings. TA1 was a new addition and covered Destination Earth (DestinE), reflecting the importance of the initiative’s platform, digital twin and data lake components. ESA and ECMWF are both development partners of DestinE, as is EUMETSAT, another contributor at the workshop.

TA2, entitled Multidomain ML4ESOP, delved into the integration of natural language processing and DL techniques into satellite image analysis. TA3 and TA4 both looked into weather and climate prediction, the former with respect to end-to-end ML and the latter in terms of hybrid approaches that combine ML with physics-based methodologies. Last but certainly not least, TA5 dealt with the emerging enablers of powerful ML4ESOP: high-performance, edge and quantum computing.

A wide range of applications were presented in the sessions, including volcanic cloud tracking, sea-ice detection, permafrost carbon dynamics, precipitation nowcasting, tree height estimation and European summer drought prediction to name but a few. Φ-labbers gave talks on the ESA-UNICEF child connectivity project and vision-based solar forecasting models.

Two poster sessions were distributed across the four days. Over 70 posters were on display, giving representatives from both the private and public sectors the opportunity to showcase ML4ESOP research that in many cases focused on extreme weather modelling and prediction.

Researchers showcasing their projects at the poster session

One of the guests at the gathering was Stephan Hoyer, Applied Sciences Research Lead at Google, who was also treated to a visit to Φ-lab: “This workshop has given us great exposure to other perspectives on EO, weather forecasting and AI, including some fascinating conversations with delegates from a huge variety of backgrounds.”

Each TA had an accompanying working group to discuss the topics presented, and the final afternoon featured a plenary session that brought the event to a conclusion. “The plenary underlined just how much had been covered during the workshop, together with the strategies we need to adopt as a community to propel ML4ESOP forward,” Rochelle commented. “It’s clear that machine learning is an indispensable tool in our quest to understand and predict the behaviour of Earth systems, and I’m confident that the work discussed will help determine the optimum approaches for future research in this domain.”

The workshop content is available on the event website.

To know more: Φ-lab, ECMWF, EUMETSAT

Photos courtesy of ESA and ECMWF/Mike Morrissey


Dedicated InCubed national calls give rise to innovative EO development activities

Following last year’s calls for proposals in the United Kingdom and Spain, seven new InCubed activities have now commenced. Addressing needs over the full range of the Earth observation value chain, the activities demonstrate how customer-oriented Earth observation solutions provide technological advances and tangible societal benefits.

With its co-funding of market-oriented development activities, the ESA InCubed programme is a key enabler for the financing of ventures that improve European competitiveness in Earth observation (EO). In cooperation with its participating Member States, InCubed periodically issues national calls in order to boost EO innovation and investment in specific countries.

InCubed national calls were launched in Spain and the UK last year. Both were met with an enthusiastic response, and InCubed then initiated a comprehensive evaluation process in conjunction with the space agencies of each country. The result so far is that five Spanish and two British companies have been awarded InCubed co-funding for new activities.

These activities cover both upstream and downstream applications, from instruments, data transmission and onboard software in the space segment to data processing and even an entire satellite mission. They respond to significant technical challenges, such as achieving timely intelligence extraction from Synthetic Aperture Radar (SAR) imagery, the need for higher resolution sensing in Thermal Infrared (TIR) cameras and the difficulty of reducing aerosol interference in methane detection.

Commercial EO España-style

The new activities from Spain encompass important developments in satellite technology. SATLANTIS’s TALISMAN is a 16U cubesat mission carrying a short-wave infrared (SWIR) optical instrument coupled with a liquid crystal polarimeter. The combination is designed to enable point-source methane detection – a vital activity for climate change mitigation – while minimising the impact of aerosol scattering. EODDL-LYNX from SENER builds on the company’s previous InCubed activity to integrate an active electronically steerable antenna with a payload data transmitter. The antenna will not only enhance downlink throughput and flexibility, but also provide a more compact, lightweight solution compared with mechanically steered variants.

Madrid-based Crisa will develop the power conditioning module of its NICE instrument control unit for satellites. The power conditioning module will complete the jigsaw of NICE’s core functions, alongside the payload controller and standard discrete interface module. Deimos Space’s Insight4EO picks up on the emerging trend of edge computing onboard satellites by providing a processing and intelligence services package for LEO EO missions. The activity is aimed at facilitating realtime decision-making, increased mission autonomy and higher data throughput.

In the downstream data processing segment, ATR4PAZ from Hisdesat will tackle the thorny issue of rapidly deriving meaningful insight from SAR data. Focusing on the security and intelligence sector, ATR4PAZ will simulate objects of interest on existing SAR imagery in order to train a machine learning to detect those features.

UK-side sustainability solutions

Like its Spanish counterparts, the British contingent also includes an innovative instrument in the form of SPIRIT, an ultra-compact, high-resolution, wide-field-of-view thermal infrared camera to be developed by Supersharp. SPIRIT imagery will be used to produce heat maps with an unparalleled revisit rate to monitor the energy usage of buildings worldwide.

London company Messium will develop an application for monitoring the Nitrogen concentration in wheat crops. Based on satellite-derived hyperspectral data and custom machine learning models, the interface will complement in-situ measurements and help in the adoption of variable-rate fertiliser distribution for more efficient nutrient management and reduced nitrous oxide production.

“Launching targeted national calls enables us to work hand in hand with our Participating States to stimulate EO innovation in individual countries,” commented Michele Castorina, InCubed Programme Manager and head of the ESA Φ-lab Invest Office. “These latest Spanish and British calls have yielded groundbreaking activities that not only include contributions to the green agenda, but have also captured the direction of travel in EO technologies with such topics as edge compute, payload downsizing and high-capacity data transmission. I therefore expect these developments to make a valid contribution to Europe’s commercial space arena.”

To know more: ESA InCubed, Spanish Space Agency, UK Space Agency

Photo courtesy of Pixabay/Efraimstochter, Pixabay/Nerivill

Applying artificial intelligence to raw satellite imagery for time-critical applications

An ESA Φ-lab-funded project is showing the benefits of using raw Copernicus Sentinel-2 data and artificial intelligence to improve response times in early warning systems for catastrophic events.

An abundance of useful satellite imagery is now available, for a realm of applications. The Sentinel-2 mission of the European Copernicus programme provides high-resolution optical imagery with good re-visit times over large areas, that are invaluable for land monitoring. However, for applications that require rapid response, such as the monitoring of catastrophic events or the detection of illegal vessels, traditional data handling and processing schemes usually exhibit excessive latency. The processing is typically encompassed within the ground segment, which causes delay.

Read the full article on sentinel.esa.int.

Copernicus Sentinel-2 image courtesy of ESA

AI in Earth observation: a force for good

The upcoming launch of the Φsat-2 mission is a prime example of the pioneering work that ESA does in the field of AI in Earth observation. But when it comes to AI, hopes and fears abound in equal measure. In this interview, ESA’s Rochelle Schneider sets the record straight on how this transformational technology is improving access to crucial information on the state and future of our planet.

The success of generative AI tools like ChatGPT has brought with it new questions of what awaits us if the power and capabilities of AI continue to grow. Fortunately, conferences such as AI For Good, in which ESA often participates, demonstrate that rather than causing harm, AI has a highly positive impact on society and sustainable development.

As an expert in machine learning, ESA Φ-lab AI Applications Lead Rochelle Schneider is well placed to explain why AI is a force for good in Earth observation. Rochelle has extensive experience in retrieving vital information from Earth observation data for the benefit of disease prevention and child development.

Read the full article on www.esa.int.

Φ-lab’s continuing EO innovation drives AI-derived methane detection

ESA Φ-lab, an established prime mover for disruptive innovation in European Earth observation (EO), is supporting a project on identifying methane pipeline leakage using satellite data. The research moves the state of the art forward through a data-fusion approach, employing machine learning algorithms to combine various remote sensing image and other time-series data sources.

Φ-lab has a proven track record for driving innovation in the Earth observation sector, including through the financial and technical sponsorship of cutting-edge research via PhDs.

Methane detection from space is one such area of Φ-lab’s research. Albeit with a shorter lifespan in the atmosphere, methane gives rise to up to 30 times more heat retention than carbon dioxide and so is proving to be a significant contributor to global warming. Detecting and fixing methane leaks is therefore a priority in terms of climate-change mitigation, but in the case of gas pipelines, management of leaks can be hampered by the vast expanses of remote locations that the networks of pipes often traverse.

In recent years Φ-lab has overseen two projects, STARCOP and LEO-GEO4GHG, both of which explored how methane plumes could be detected from satellites using onboard AI. In new research that Φ-lab is jointly supervising with German aerospace company OHB Digital Connect and the Technical University of Munich, PhD student and OHB employee Enno Tiemann is exploiting machine learning to analyse a combination of data sources for plume detection and quantification.

“Currently, methane detection often relies on experts who manually inspect the EO data and create a map of the plume that is then used to determine the emission rate – sometimes with a high degree of uncertainty,” Enno explains. “Whereas previous work has adopted a cascading approach, whereby initial AI detection of large plumes is followed by human analysis of higher-resolution optical or infrared data, our research involves algorithms that simultaneously interrogate multiple satellite datasets to improve detection and reduce uncertainty.”

Although Sentinel-5P is a well-established methane detector, giving daily coverage of the entire planet, it has a fairly coarse resolution that means the minimum measurable emission rate is around 25 tonnes per hour. Sentinel-2 and Landsat 8 and 9 by contrast have a threshold of 1.5-1.8 tonnes per hour, but lack daily coverage. By fusing time series data from multiple satellites, in conjunction with wind-vector data, the new research is expected to yield more robust estimates of plume location and gas flows, together with insight on the evolution of plumes over time.

Φ-lab provides essential support to the project through technical supervision, expertise sharing and joint funding with OHB. Φ-lab AI Applications Lead Rochelle Schneider is the project supervisor for ESA: “In addition to giving vital input to the project, Φ-lab is able to grant external researchers access to our pool of world-leading AI experts and data scientists. We’re very happy to help guide this research initiative, which like STARCOP and LEO-GEO4GHG is an example of AI-driven EO that is sure to make a significant contribution to combatting methane emissions in the energy sector.”

To know more: Φ-lab, OHB, TUM

Photo courtesy of Robzor

Φsat-2 gets two new AI apps

Φsat-2, ESA’s groundbreaking cubesat scheduled for launch in June, will now include two new AI-driven apps destined to demonstrate a crucial role in future environmental monitoring from space. The apps, which focus on marine pollution and wildfires, were the winning entries in ESA’s OrbitalAI Challenge.

AI is already established as a vital tool in Earth observation, helping to sift through terabytes of satellite data to provide useful insight for scientists, policy makers and commercial operators. But while most AI processing of information is carried out on the ground once the data have been downloaded, there are significant advantages to be had from integrating AI capabilities on the satellite.

Read the full article on www.esa.int.

Building ChatGPT-style tools with Earth observation

Imagine being able to ask a chatbot, “Can you make me an extremely accurate classification map of crop cultivation in Kenya?” or “Are buildings subsiding in my street?” And imagine that the information that comes back is scientifically sound and based on verified Earth observation data. ESA, in conjunction with technology partners, is working to make such a tool a reality by developing AI applications that will revolutionise information retrieval in Earth observation.

A digital helping hand for data

Earth observation generates vast volumes of vital data every day, but it is difficult for humans alone to ensure that we obtain the best value from that data. Fortunately, AI helps in interacting with such large and complex datasets, identifying key features and presenting the information in a user-friendly format.

I*STAR for example, an activity co-funded by the ESA InCubed programme, developed a platform that uses AI to monitor current events like earthquakes or volcano eruptions so that satellite operators can automatically plan the next data acquisitions for customers. The SaferPlaces AI tool, again supported by InCubed, creates flood maps for disaster response teams by merging in situ measurements with satellite data. SaferPlaces was crucial to damage assessment efforts during last year’s floods in Emilia-Romagna in Italy.

Read the full article on www.esa.int.

Hello Major TOM: ESA Φ-lab releases largest ML-ready Sentinel-2 dataset ever published

ESA Φ-lab has launched Major TOM (Terrestrial Observation Metaset), a community-oriented project that allows researchers to share, use and combine large Earth observation (EO) datasets. The Major TOM framework will help unlock the huge potential of satellite imagery by offering users the largest ever quality-controlled and globally distributed sample of data, with future expansions to multiple satellites and modalities planned from both Φ-lab and the wider community.

Recent years have seen a marked trend towards larger, more general EO and geospatial models, known as foundation models, which require massive volumes of high-quality training data. These large models present unique opportunities in that they have the potential to help solve many pressing scientific and societal problems.

But there are also challenges, including the risk of deepening the reproducibility crisis seen in AI research, whereby published models are often difficult to recreate due to closed data sources and opaque technical details. Bias is another issue, since all models are skewed by the data they learn from, and this may lead to biases being embedded into the systems that foundation models form part of.

ESA Φ-lab believes that these issues can be alleviated through the creation of high-quality globally distributed and collaborative ML-ready datasets, and has begun to integrate them under the moniker Major TOM. These ML-ready datasets are a means to steer the development of large models in a positive direction, democratising them and helping to make systems that are more reproducible and with a lower bias by virtue of the dataset’s global sampling. To achieve this, Φ-lab has partnered with Hugging Face to host and freely distribute Major TOM on the Hugging Face Hub. With its open and community-driven platform for datasets and models, Hugging Face is a leading light for the democratisation of machine learning technology.

The creation of such a large dataset presented the team with several technical hurdles. “Satellite data is often held and delivered in very large products – over 100 km across – which many people find difficult to work with for machine learning applications, especially when trying to combine different satellites whose products overlap to differing extents,” explains Φ-lab research fellow Alistair Francis. “By contrast, Major TOM uses a fixed, 10 km grid across the entire globe, meaning that data from one Major TOM dataset will fit neatly on top of another.”

Whilst the sheer volume of data processing involved was a challenge, the need to ensure its quality was equally difficult. For example, optical satellite imagery often contains clouds that hide the surface below. Although not eliminated from the Major TOM dataset entirely, cloudy imagery was minimised by using Φ-lab’s state-of-the-art AI cloud mask, soon to be released publicly.

Illustration of the global coverage of Major TOM Core. Regions in colour denote sampled areas (green for land and light blue for sea).

Major TOM’s inaugural core dataset has now been released on Hugging Face. It constitutes the largest ML-ready collection of Copernicus Sentinel-2 images ever published. Covering over 50% of the Earth’s surface (including almost all dry land) with nearly 50 TB of data and 2.5 trillion pixels, Major TOM Core is a game-changer for those seeking to train large models with satellite data. It is expected that future expansions from the broader EO community, spearheaded by ESA Φ-lab, will spawn a diverse ecosystem of combinable datasets that will be invaluable in creating the next generation of large deep learning models from satellite data.

Giuseppe Borghi is the Head of ESA Φ-lab: “We want to build an open community of contributors and end users who can create a data landscape that ensures EO derives the largest benefits from the AI revolution. If we want to make sure that EO models are reliable, reproducible, traceable and in turn, trustworthy, then it stands to reason that we need to start with high-quality trustable data.”

An interview on Major TOM with Alistair Francis and fellow Φ-lab researcher Mikolaj Czerkawski can be found here.

To know more: Φ-lab, Hugging Face, Major TOM paper preprint

Header image contains modified Copernicus Sentinel data (2022), processed by ESA