ESA title

Category: Philab

MAPTCHA – a platform where citizen science meets Earth observation for fast emergency responses

Posted on by mavelar

Every second counts when disaster strikes. Whether it is a wildfire or an earthquake, extracting crucial information from satellite data in a fast way is paramount for an effective emergency response. Supported by ESA Φ-lab, the MAPTCHA project will revolutionise the way first responders make informed decisions and potentially save lives during natural catastrophes.

Imagine you are about to sign up for a new service or make a purchase online, and suddenly you face the following challenge – “Select all the images with traffic lights”. Sounds familiar, correct? This is Human CAPTCHA (hCAPTCHA – Completely Automated Public Turing test to tell Computers and Humans Apart), a digital gatekeeper designed to distinguish human users from automated bots.

Now, consider the following scenario: you are looking at a satellite-derived image and you see a fire starting to develop. What if there was a quick way to flag this occurrence, enabling fast responses and informed decisions from emergency teams? This is exactly what MAPTCHA, a citizen science-based project, is trying to accomplish. MAPTCHA – just like “CAPTCHA”, but for “Maps” – is being developed by Osir.io and RSS-Hydro, in collaboration with ESA Φ-lab.

So, let’s take a look at the general MAPTCHA system: it starts by screening natural disaster events from various Internet sources and then collect relevant information such as the area of interest (i.e., the region affected), date of the event and event type (e.g., wildfire) and progression.

Next, the system identifies and retrieves new Sentinel-2 data that are associated with the detected events. These data are pre-processed to convert the multispectral data into false-colour composition that can be better interpretable by the human eyes, tiled in small patches, and then distributed to ‘Human Platform’. This service provided by Intuition Machines allows hCAPTCHA-based annotations, accessible to the entire global population with Internet access. The annotations provided by ‘Human Platform’ are stored and can then be visualised through a mapping tool that enables interactive map exploration.

Annotations are a crucial part of the system. In machine learning (ML) and artificial intelligence (AI), image annotation is the process of labelling images to train AI and ML models. This practice often involves human annotators that assign relevant classes to different entities in an image. For instance, a person can be asked to identify vehicles in a set of images – just like in an hCAPTCHA test. The resulting data from that identification (or annotation) can help train AI/ML models that can recognise and detect vehicles and discriminate them from pedestrians, traffic lights, or potential obstacles on the road to navigate safely.

By taking advantage of the massive volume of crowd-sourced image annotations, MAPTCHA seeks to significantly speed up the process of gathering information on damaged areas, resource allocation and displaced populations, empowering first responders and allowing informed decision making.

Its value goes beyond emergency situations: the massive volume of annotations generated by the platform will serve as a rich training dataset for ML models. This can significantly improve the capabilities of AI-powered Earth Observation applications, leading to advancements in fields such as environmental monitoring, resource management and climate change analysis.

In a collaboration with NASA for the FireCapture project, MAPTCHA is focused, for now, on early fire detection. Several Copernicus Emergency Management Service wildfire events in Bolivia, Spain, Chile and Mexico were used as study cases.

For the events in Chile, for instance, preliminary tests showed that around 60 annotations per minute were being collected, up to a total of 73 000 annotations. These tests showed that automated bots have difficulties accessing and interpreting Sentinel-2 imaging after the pre-processing step, as compared with what happens when a bot tries to identify traffic lights, demonstrating the reliability and added value of MAPTCHA.

“MAPTCHA can do both: adapt rapidly and process very large amounts of Earth Observation data to provide actionable insights for first responders,” says Ron Hagensieker, Founder of Osir.io. “Essentially, this project represents a significant leap forward in Earth Observation, offering a powerful tool not only for crisis response, but also to proactively safeguard our planet for the future generations.”

Guy Schumann, Founder and CEO of RSS-Hydro also states that “Rapid and accurate data extraction from satellite imagery is crucial during catastrophes such as wildfires, floods or earthquakes. By harnessing the collective intelligence of a vast network of users, MAPTCHA demonstrated how online antibot technology can boost Earth Observation data uptake, with huge potential to optimise disaster response.”

“The MAPTCHA initiative paves the way for a future where citizen science and AI converge to unlock the full potential of Earth Observation data”, comments Nicolas Longépé, Earth Observation Data Scientist at Φ-lab. “By facilitating the access to image annotation and continuously improving AI models, we gain a deeper understanding of our planet, developing sustainable solutions for the challenges we currently face”.

To know more: ESA Φ-lab, Osir.io, RSS-Hydro

Photo courtesy of Unsplash/Nejc Soklic

EO MakerSpace: crafting the future of smarter Earth observation systems

Posted on by mavelar

The Earth observation (EO) domain is undergoing a disruptive transformation featuring onboard intelligence and innovative sensing capabilities. With the support of InCubed, ESA EOP co-funding programme led by Φ-lab, IngeniArs is implementing EO MakerSpace – an initiative dedicated to AI on edge for the development of EO smart sensors.

The space sector is currently experiencing a technological and business evolution driven by increased demand and specialised advancements, such as onboard intelligence. Through the use of advanced AI techniques to process information directly on spacecraft, onboard intelligence promises to deliver more efficient, agile, autonomous, and reconfigurable Earth observation (EO) systems. ESA Φ-lab has a track record of developments in this area, as depicted below.

Timeline of Φ-lab-powered satellites and constellations

The latest research efforts have been committed to exploring the use of AI on edge for EO applications, such as the early detection of natural disasters, vessel incidents, and gas leaks. Onboard intelligence is also capable of identifying low-quality data, like cloud-covered satellite data and remote sensing images with limited information of interest, discarding them to save downlink data bandwidth to the ground station.

Leveraging the full capabilities of AI on edge in EO systems needs the development or enhancement of both space-related software and hardware. These improvements are essential for seamless integration with innovative sensors, as well as for preprocessing, calibrating, and correcting sensory data to guarantee the accurate performance of AI algorithms. To cope with the limited power and energy budget in space, it is essential for hardware and software design to work in tandem to maximise energy efficiency.

To achieve these goals and with a focus on the Italian ecosystem, ESA InCubed is partnering up with IngeniArs, a company specialised in the development of innovative high-tech electronic and informatics systems, in the aerospace, healthcare, cybersecurity and AI domains. IngeniArs has been developing GPU@SAT, a hardware/software ecosystem for space systems dedicated to AI and Computer Vision applications.

This contract is focused on the rapid and responsive prototyping of core elements of smart EO payloads within the GPU@SAT ecosystem, based on joint software and hardware developments. IngeniArs will coordinate the elements of individual (sub-)activities, while ESA InCubed will oversee and expedite the approval process. IngeniArs will be responsible for managing each (sub-)activity development with suitable entities.

Italian individuals, start-ups, SMEs, LSIs, university spin-offs and any innovative entities are welcome to apply and participate in the development of EO payloads.

Activities can be developed within the following domains: algorithm optimisation and software abstraction, sensor selection and integration, onboard data compression, smart sensor management and AI-based autonomous navigation. To facilitate the development of these activities, the GPU@SAT ecosystem will be implemented on a representative device, creating a dedicated development kit – the GPU@SAT devkit. To know more about these activities and to apply, visit the dedicated IngeniArs project webpage.

Giuseppe Gentile, CEO of IngeniArs, comments: “At IngeniArs we are very proud of leading this project. It will be a very important moment to gather the Italian ecosystem around the hot topic of ‘enabling Artificial Intelligence on-board’. We will make our GPU@SAT platform available for the development of applications and modules within a fully representative space environment, and we are very excited to see the results.”

This initiative will offer the EO ecosystem a report with the overall findings, including improved mechanisms to spur innovation and possible demonstrations or pilots, followed by a technological roadmap with the remaining open questions and technical gaps.

“The integration of AI algorithms with specialised hardware and software co-design can accelerate the future of Earth observation missions, leading to substantial reductions in time, costs, and the need for human resources while improving performances, latency time, and missions’ autonomy” comments Michele Castorina, Head of the Φ-lab Invest Office. “This collaboration with IngeniArs aligns with the InCubed mission to foster innovative technologies in the space industry, improving the efficiency and agility of autonomous EO systems through cutting-edge AI techniques.”

To know more: ESA InCubed, IngeniArs

Photo courtesy of Pexels/SpaceX

Catalysing new space ventures with ESA’s Earth Observation Commercialisation Forum

Posted on by mavelar

Taking place at KAP Europa in Frankfurt, Germany, from 27 to 28 November 2024, the second ESA Earth Observation Commercialisation Forum will bring together stakeholders from the Earth observation and space commercialisation sectors, including end users, space industry players, entrepreneurs, private and public investors and policymakers to discuss and promote commercial opportunities in Earth observation.

The event is an important part of ESA’s broader strategy to enhance the economic return from Earth observation investments to ensure that space technology contributes to sustainable economic growth.

The ESA Earth Observation Commercialisation Forum (ESA CommEO) is a collaborative initiative by ESA’s Earth Observation Programme along with the Commercialisation, Industry and Competitiveness directorate to foster the commercialisation of Earth observation space technologies and services.

To support the growth of space-related businesses and startups, ESA CommEO features dedicated events, workshops and networking opportunities to connect different stakeholders, providing information and insights on market trends, showcasing success stories and innovative solutions and offering platforms for pitching ideas and business plans to potential investors.

Read the full article on www.esa.int.

Unlocking Earth Observation commercial opportunities in Spain: new ESA InCubed funding cycle

Posted on by mavelar

Spain-based companies of any size are invited to participate in the latest Φ-lab InCubed co-funding call, by submitting proposals for the development of groundbreaking and commercially successful products in the Earth observation (EO) field. The call opens on 15 September and the deadline for submissions is 28 October 2024 at 14:00 CET.

InCubed is an ESA co-funding partnership programme run by ESA Φ-lab that aims to fill the gap between business ideas and the Earth observation market, with the support of its signatory Participating States. Focusing on leveraging the benefits of EO data and services, InCubed provides co-funding for the development of any elements of the EO value chain, namely full satellites and constellations, platforms, payloads and instruments, ground segment-based systems, and downstream applications and value-added services.

Entities can apply for different levels of co-funding, depending on the type of activity, and will be guided by ESA top-tier experts to create sound products/services from a technical, commercial, and financial standpoint. In this partnership, ESA will act as the partner of the proposing company, with the aim to reduce its development and business risks.  

In collaboration with the Spanish Space Agency (AEE), this latest call will open on 15 September 2024 and has a budget of €11 million, of which up to 30% can be allocated to de-risking activities (see its definition here). Proposals should be focused in developing EO innovative and commercially successful products and services. Proposals for Contract Change Notices for ongoing InCubed contracts may be submitted as well and will follow the same evaluation criteria of the new proposals.

Proposals may include non-Spanish suppliers, as long as they are essential for the success of the project. The goal is to demonstrate a strong economical return to the Spanish space sector, including both the lead applicant and their partners and supply chains. Proposed activities may be funded up to 50% of the total cost in the Product Development Cycle, and up to 75% in the De-risking Cycle. Universities and research institutes with no commercial interest in the project may be funded up to 100% of their costs if those do not surpass 30% of the total activity funding. Ideas currently being funded by other ESA/AEE programmes are not eligible for this call.

Differing from previous standard procedures, this specific call will not require pitch presentations from applicants. Spanish companies must submit directly an Outline Proposal (Part 1) on the dedicated InCubed platform, which will be ranked based on defined criteria. Those who score above the cut-off point will be invited to submit a full proposal to ESA, together with the letter of support from AEE, which they must do within 4 to 6 weeks after the invitation. Successful applicants identified by the Tender Evaluation Board will be contacted directly by ESA to discuss further contract negotiations.

Interested entities can find out more about this call during the AEE event in Madrid, on 3 September, and enrol via the AEE dedicated link. The call opens on 15 September and the closing date for submissions is 28 October 2024 at 14:00 CET.

To know more: ESA InCubed, Spanish Space Agency, InCubed Spanish Call event by AEE

Photo courtesy of ESA

Two Φ-lab supported satellites, ESA Φsat-2 and SmartSat CRC Kanyini, take AI to new heights

Posted on by mavelar

Φsat-2, ESA’s recent venture to advance Earth observation (EO) capabilities through AI on edge, was launched on a Space X Falcon 9 rocket from the Vanderberg Space Force Base in California, USA, on 16 August 2024. As part of the Transporter 11 rideshare mission, Φsat-2 will perform several AI-powered tasks on-board, generating actionable insight from raw data directly on the satellite, and improving autonomy and near real-time decision making. Also, Kanyini, from SmartSat CRC (Australia), has been successfully launched. Other three Φ-lab-supported satellites will follow in November 2024 on Transporter 12.

AI has a proven track record in the analysis of big volumes of satellite data, with most of the processing taking place on the ground after the data has been downloaded. Following the ‘AI-success’ of Φsat-1, launched in 2020, the Φsat-2 mission stands out from more traditional methods of analysing EO data, allowing direct processing at the very source: on-board of the satellite.

How is this possible? Φsat-2 is a 22x10x33 cm CubeSat, designed and developed by Open Cosmos as the prime contractor for ESA, and equipped with a novel multispectral camera and a powerful AI-based edge computer that will analyse and process EO imagery while in orbit. Φsat-2 has six applications running on-board, which will be used for cloud detection, street map generation, maritime vessel detection, on-board image compression and reconstruction, marine anomaly detection and wildfire detection.

The two latter applications, which concern marine anomaly detection and wildfire detection, were the winners of ESA’s OrbitalAI Challenge, coordinated by Nicolas Longépé, EO Data Scientist at Φ-lab. These two applications were developed by IRT Saint Exupéry Technical Research and by Thales Alenia Space, respectively. The remaining applications were developed and supported by an industrial consortium including CGI, Simera, Ubotica, CEiiA, GEO-K and KP Labs.

Φsat-2 will showcase the potential of developing, installing and operate custom AI apps on the satellite while it is in space. Its adaptability allows a better response to evolving needs, maximising its value for scientists, businesses and governments. Alongside Φsat-2 and other 114 Cubesats and Smallsats, the Kanyini satellite was also launched during the Transporter 11 mission. Developed by SmartSat CRC, Myriota and Inovor Technologies, Kanyini is the first South Australia government-funded satellite that will deliver critical sustainability and climate data to various institutions, detect bushfires faster than traditional methods, incorporate predictive AI for landslides and flooding, and sense urban heat islands.

To accomplish this, Kanyini resorts to two different payloads. First, the HyperScout 2 hyperspectral imager from Cosine (NL), which will provide detailed EO imagery to support crop health, forestry and inland and coastal water management research. Second, a Myriota IoT Space Services device that will enable the data transfer ability on the satellite.

The development of the Kanyini mission also counted with ESA Φ-lab’s expertise and support, including an exchange programme at the SmartSat CRC premises in Adelaide, Australia, (i.e., with Roberto Del Prete, a Visiting Researcher at the time and now Research Fellow at Φ-lab) for the creation of a set of comprehensive documentation to help SmartSat CRC partners understand data products and derivatives from Level 0 data. Roberto further examined quality control issues within data processing sequences, and together with the Kanyini team developed software for executing on-board AI based on Φsat-2 and other CubeSat standards.

Φ-lab will see soon (in November 2024, date yet to be confirmed) the launch of other three EO satellites, on Transporter 12, whose development was fully supported by the ESA InCubed programme, managed by ESA Φ-lab. The three satellites are HiVE from Constellr (DE) and OHB (DE), Forest 3 from OroraTech (DE), and a precursor of AIX from Planetek (IT), AIKO (IT) and D-Orbit (IT).  This will bring the count of ‘Φ-lab-powered’ satellites and constellations to 11 by the end of 2024.

Timeline of current ‘ESA Φ-lab-powered’ satellites and future launches

More information about the launch of these satellites can be found here and here. You can watch the full SpaceX Transporter 11 mission launch here.

To know more: Φsat-2, Kanyini, SpaceX Transporter 11 mission

Photo courtesy of SpaceX

AgriKOPA is enhancing smallholder agriculture with Earth observation

Posted on by mavelar

InCubed, a co-funding programme managed by ESA Φ-lab, is supporting agriKOPA – a platform that provides financial services and real-time crop monitoring, being an easy and convenient tool for end users during the agricultural season. This platform is led by agriBORA, a Kenyan-German agri-fin-tech company that aspires to empower small farming businesses in Africa.

ESA InCubed, an ESA EOP co-funding programme managed by Φ-lab, has a proven reputation for establishing contracts to develop pioneering technologies, services, and applications within the Earth observation (EO) domain. Φ-lab and the World Food Programme Innovation Accelerator set up the EO & AI for SDGs Innovation Initiative in 2021 to find commercially viable EO and AI-based solutions to counteract global hunger issues. As a result of the selection process, agriBORA, a Kenyan-German agri-fin-tech company that strives to transform the agricultural business model in African countries, received a grant to demonstrate a proof of concept within the initiative’s theme. AgriBORA further matured its solutions and was then selected for a contract in the context of an InCubed funding call.  

Agriculture is a key point for food security in Kenya, and it also provides a source of income. Climate change endangers crop development, which puts the farmers’ way of living at risk. With the support of InCubed, agriBORA is developing agriKOPA, an initiative that relies on the use of data analytics powered by EO satellite data, enabling local agri-merchants to work with agriHUBs – providers that offer climate-smart advisory services, linking financial services, input suppliers, farmers, and the market. The novelty in the use of EO data by agriKOPA is the creation of a score for each farmer, enabling Financial Service Providers (FSPs) to lend credit with more confidence. The loans will then allow farmers to purchase the inputs needed for production.

Albin Lacroix, a Φ-lab InCubed officer, shared his thoughts on the latest mission in Kenya, from 10 to 12 June 2024. This mission included visits and meetings with different stakeholders – agriBORA, Kenya Commercial Bank, and the Kenyan Space Agency. 

The mission started with a visit to the agriBORA premises, in Nairobi, focusing on one of the milestones of the project – the factory acceptance test (FAT). This represents the finalisation of the technical development of the service, and the kick-off of the pilot phase, rendering the service operational for test users. “Our meeting about FAT was particularly important for the project, as the service relies on many interactions between farmers, agriHUBs, loan providers, and agriBORA. Together with Kizito Odhiambo, Founder and CEO of agriBORA, and the Kenyan team members, we went through the intricate process pipeline of agriKOPA”, comments Albin.

The second day was spent on two different agriHUBs in Kisumu. The visits to these agriHUBs were the key point of the mission, given that there was direct contact between the InCubed programme, hub managers and farmers, who are the end users of the service. These meetings were held half in English, and half in Swahili, providing feedback on the first pilot tests conducted by agriBORA.

Albin Lacroix says “InCubed is a market-oriented programme and end users are the center of our solution design. Meeting them in person was very important. It was incredibly interesting to hear their daily concerns and the advantages they get from agriKOPA. The added value is immense, from getting good quality material inputs on time to ensuring loans and insurance in the case of bad crop years. The EO aspect is not directly visible to the farmers, but it is crucial for the estimation of the score provided by agriBORA to FSPs, allowing them to trust the smallholder they will lend credit to.” The next step will be the successful conduction of the pilot phase, followed by scaling up.

Albin Lacroix visits two agriHUBs in Kisumu

The third and final day of the mission began with a visit to the Kenya Commercial Bank (KCB) premises in Nairobi, and a meeting with a KCB representative. KCB is an FSP for agriBORA. After receiving requests for loans, agriBORA does a first round of scanning and filtering of applicants. KCB does a second round of scanning before releasing the loan to the farmer through the agriKOPA platform.

This was followed by a visit to the Kenya Space Agency (KSA). “The goal of the visit was for agriBORA to present their updates. KSA is very excited about this project, given that the topic of smallholder farmers is of great relevance to public authorities in Kenya. These farmers need funds to succeed in their activities and, with the support of InCubed, agriKOPA is the bridge between them and FSPs. KSA representatives also had the opportunity to get familiar with the work developed at Φ-lab and the main purpose of InCubed. Charles Mwangi, Head of the Earth Observation, Research, Education and Outreach Programmes at KSA was present in that meeting with three of his colleagues from KSA”, Albin remarks.

During this three-day mission, Albin Lacroix was accompanied by Kizito Odhiambo, Founder and CEO of agriBORA: “Unlocking access to finance through agriKOPA is a great milestone for us and aligns with our vision of making Africa the agricultural powerhouse of the world. The incredible support from ESA through InCubed and the strong partnerships with local financial service providers have been instrumental in the development phase, helping to de-risk the entire process. We are very excited to start the validation phase during the upcoming short rain season in August.”

Michele Castorina, Head of the Φ-lab Invest Office, comments: “this collaboration between InCubed and agriBORA has accelerated innovation in the EO, agriculture-related domain, providing agriBORA with cutting-edge satellite technology and expertise. Together, we are revolutionising farming, empowering smallholdings, and boosting productivity and sustainability. This is a leap towards a smarter, more resilient agricultural future.”

To know more: ESA InCubed, agriBORA, Kenya Space Agency

Photo courtesy of ESA

ESA and UK Space Agency announce new funding call

Posted on by mavelar

ESA and the UK Space Agency are pleased to announce a new joint funding call ‘InCubed2 – Innovation in Public Services with Satellite Earth Observation’ for all UK-based entities developing innovative and commercially viable Earth observation projects. The deadline for pitch proposal submissions is 12 September 2024.

InCubed is a co-funded programme run by ESA’s Φ-lab Invest Office, focusing on initiatives that exploit or enhance the value of Earth observation imaging and data. With the support of its participating Member States, InCubed funds a wide scope of activities, ranging from satellites to ground and downstream applications.

Successful applications receive personalised guidance from world-class experts to develop technical, commercial and financially viable products. Currently, 140 activities are in the pipeline and about 10 satellites are under development. One of these projects was launched in 2023 and three others are planned for launch in 2024.

In collaboration with the UK Space Agency (UKSA), the latest funding cycle will be open exclusively to the UK industry. Following the success of last year’s call, this round will be dedicated to activities with the underlying theme of ‘Innovation in public sectors use of Earth observation data’.

Read the full article on www.esa.int.

ESA Φ-lab synergises with SmartSat CRC for Earth observation advancements

Posted on by mavelar

ESA Φ-lab, a recognised entity at the forefront of Earth observation (EO), and SmartSat CRC, Australia’s leading space research centre, promoted an exchange initiative for researchers and PhD students to develop new onboard processing and synthetic aperture radar (SAR)-related technologies, and flood forecasting datasets.

ESA Φ-lab, a world-class innovation centre with a notorious record in EO, established a three-month exchange initiative with SmartSat Cooperative Research Centre (SmartSat CRC), a broad consortium that leads the Australian space research sector. Along with its commitment to pioneering research in AI4EO, Φ-lab is dedicated to other educational initiatives such as the Young Graduate Traineeships, fostering the development of cutting-edge EO activities.

Following the recently signed agreement between Φ-lab and SmartSat, which rests on the success of the Φsat-1 ESA mission and the integration of HyperScout-2 (an instrument partially supported by Φ-lab) into the South Australian Kanyini satellite, a visiting researcher from Φ-lab and two SmartSat-affiliated PhD students joined the exchange enterprise. This emerged as an opportunity for European and Australian researchers to increase knowledge sharing and develop space solutions, integrating existing AI technologies and innovative research.

One of the participants was Nermine Hendy, a PhD student from the Royal Melbourne Institute of Technology. During her stay at Φ-lab, Nermine worked on a machine-learning approach capable of detecting radio frequency interference in Sentinel-1 SAR raw data and subsequently mitigating it. The approach was designed for onboard implementation to ensure a quicker, more efficient performance. “This experience was incredibly rewarding. I had the unique opportunity to collaborate closely with industrial teams and professional researchers, gaining invaluable insights into real-world applications of satellite technology and contributing to significant projects,” Hendy comments. “The supportive environment at Φ-lab made this internship a truly memorable and transformative period in my academic and professional journey.”

While at Φ-lab, Brandon Victor, a PhD student from the Department of Computer Science and Information Technology at La Trobe University, worked on producing a global flood forecasting dataset. The goal was to exploit existing datasets and models that map a flooding event after its occurrence and turn them into challenge datasets for flooding prediction. Brandon says: “I truly enjoyed my stay. I received a very warm welcome from the staff and made some friends along the way. Φ-lab has a wonderful energy, where everyone is trying to solve big challenges, and being a part of it was remarkable. They are doing research for the public benefit, and I appreciate that in a research lab.”

Both PhD students were supervised by Nicolas Longépé, Earth observation data scientist at Φ-lab: “It was a pleasure to have these students working with us and I look forward to seeing the ideas this enterprise will inspire. As we face climate change and increased natural hazards, EO technologies have an immense potential to improve life on Earth. Bringing together great scientific minds will stimulate an advance in space research and foster a stronger international cooperation between Europe and Australia.”

Roberto Del Prete, a visiting researcher at Φ-lab, worked for the Kanyini mission during his time at SmartSat premises in Adelaide, Australia. As the expert in EO, Roberto created a set of comprehensive documentation to help SmartSat partners understand data products and derivatives from unprocessed – Level 0 – data. Roberto further examined quality control issues within data processing sequences, and together with the Kanyini team developed software for executing onboard AI based on Φsat-2 and other CubeSat standards.

“My three-month stay in Adelaide with SmartSat, working on the Kanyini mission, was an immensely rewarding experience. I am deeply grateful to all SmartSat staff for their unwavering support, guidance, and hospitality. The warmth and friendliness of the Australian culture have made my time there even more special. This opportunity has not only enriched my technical knowledge and expertise but also contributed significantly to my personal growth,” comments Roberto.

This synergistic endeavour yielded further collaborations with SmartSat partners – the University of South Australia, the University of Adelaide, and the Queensland University of Technology – to develop and deploy specific solutions for the Kanyini mission. The Kanyini satellite is scheduled to be launched in July 2024. Φ-lab and SmartSat will continue joining efforts to create a library of interchangeable applications between different satellites. This will allow researchers to tip and cue for facilitated information collection and to demonstrate new swarm capabilities.

Giuseppe Borghi, Head of ESA Φ-lab Division, states that “ESA and Australia have been allies in space for decades. This exchange initiative reflects Φ-lab’s dedication to accelerating the future of EO technologies through unprecedented research, together with SmartSat’s expertise. I look forward to seeing the end products of this fruitful collaboration.”

To know more: Φ-lab, SmartSat CRC

Photo courtesy of Pexels/fauxels

Φ-lab leads the way for new ChatGPT-style tools for Earth observation

Posted on by mavelar

As recently announced, ESA Φ-lab, in conjunction with its technology partners, is leading activities to develop AI foundation models, in a ChatGPT style, aimed at intelligent information retrieval in Earth observation (EO). With the launch of further initiatives exploring large language models, now is a good time to look back at the new and existing work Φ-lab is doing in this field in more detail.

ESA, other space agencies and New Space enterprises operate Earth observation missions for the benefit of science, commerce and society as a whole, but the volume of satellite data available far exceeds the capacity of humans to process and derive actionable insight in a timely manner.

Progress with more traditional AI can however be hampered by the need for a pool of labelled data to train AI models. Foundation models help to circumvent this limitation through generally self-supervised learning from large and varied sources of unlabelled data, in addition to supervised ones that are still necessary. Foundation models also deliver tools that can be adapted to a broad range of tasks, and since their inception in 2018 foundation models have contributed to a huge transformation in machine learning, even leading to chatbots with impressive natural language capabilities and several other emerging properties.

Φ-lab has a proven pedigree in disruptive innovation in Earth observation, with a particular focus on AI4EO and innovative computation paradigms. As covered in an article in March, given the enormous potential of foundation models for rapid, self-supervised learning, Φ-lab is undertaking various initiatives to create foundation models exploiting EO and remote sensing datasets.

The PhilEO project has been running for over a year. Developed by Φ-lab in conjunction with e-GEOS and Leonardo Labs, and exploiting the davinci-1 supercomputer, PhilEO is a geospatial foundation model trained on global Copernicus Sentinel-2 data. The model uses metadata from Sentinel-2 images and is trained to identify geographical features around the Earth, enabling it to learn general features and perform land cover classification, estimation of density and proximity between buildings and road segmentation regression.

In a major milestone, the PhilEO team is now releasing the model itself and associated resources to further research and testing throughout the EO community. PhilEO Bench, an evaluation benchmark that allows the performance of various models to be compared, can already be found on GitHub, and PhilEO Globe, the Sentinel-2 dataset, has been uploaded to Hugging Face. The code for the model will be available on the Hugging Face page in the coming weeks.

Two new activities supported by Φ-lab have also just been launched. A consortium comprising DLR, FZ Jülich, KP Labs and IBM will develop a European foundation model that is expected to significantly progress the state of the art. This project, which is named FAST-EO (Fostering Advancements in foundation models via unsupervised and Self-supervised learning for downstream Tasks in Earth Observation), will develop a multi-modal foundation model. Incorporating both Sentinel-1 SAR and Sentinel-2 optical, worldwide datasets, the model will integrate natural language capabilities and undergo validation through a range of environmentally critical applications such as methane leaks, biomass estimation and landcover change.

A second initiative has commenced in the last months. Foundation Models for Climate and Society is led by the Norwegian Computing Center, along with various national meteorological offices. This project, which is named FM4CS (Foundation Models for Climate and Society), will develop a foundation model that will focus on climate adaptation and extreme-weather-event mitigation. This enterprise will also benefit from the use of LUMI (Large Unified Modern Infrastructure), a petascale, world-class supercomputer.

AI foundation models serve as the engines of digital assistants, whereby the core processing of the foundation model is integrated with natural language models and interactive user interfaces. The general idea of a digital assistant is for all users – from non-technical to EO experts – to be able to perform a query on EO data archives such as “How many different crop types are in this Sentinel-1 image?”, ask more generic questions linked to EO and Earth science such as “How can EO help to monitor urban heat islands?”

To mature the human-interfacing aspect of a digital assistant, Φ-lab has just launched a new project with Pi School to build an EO Virtual Expert (EOVE). The team is exploring a set of large language models (LLMs), which will be trained and fine-tuned on specific and crafted documents related to EO and Earth science. A web platform with a simple graphical user interface and an application programming interface will be created as the gateway for the trained LLM.

The end game for Φ-lab’s various ventures in foundation models and LLMs is to set a path towards an EO digital assistant that can respond to information and knowledge queries posed in natural language and that will produce reliable, validated content.

“Foundation models are bringing a paradigm shift in AI, thanks to the scaling in training data and model size. These intelligent agents can be adapted to several specific applications and are showing impressive emerging properties, unlocking the potential of AI like never seen before. In this field, LLMs are currently disrupting the way humans interact with intelligent agents via natural language,” comments Head of ESA Φ-lab Division Giuseppe Borghi. “The integration of these models with EO and other heterogenous data will ultimately place a dedicated ChatGPT-style tool at the fingertips of EO end users in many sectors.”

To know more: Φ-lab, Norwegian Computing Center, Pi School

Photo courtesy of Pexels/ThisIsEngineering

Global storm surge forecasting: creating early warning models with AI4EO

Posted on by mavelar

ESA Φ-lab, a key point of reference for groundbreaking innovation in Earth observation, was recently invited to present its work on storm surge forecasting to an audience at Google Research. Φ-lab’s initiative follows a state-of-the-art approach, combining deep learning and data fusion of satellite imaging, tide gauge measurements and weather forecasting models. The goal of this enterprise is the optimal prediction of surge-related natural disasters, especially in under-served areas.

According to the United Nations Development Programme, the effects of climate change on coastal flooding will increase up to five times over the century, leaving more than 70 million people in the way of expanding floodplains. Latin America, the Caribbean, the Pacific and Small Island Developing States are expected to be among the most affected areas. Following the UN Sustainable Development Goals framework, ESA supports a rapid and resilient crisis response and Copernicus has its own Emergency Management Service.

Storm surges are one of the most critical climate change consequences. They are ocean dynamics driven by extreme weather, superimposing temporary rises on the mean sea level and causing coastal floods. The short-term prediction of these phenomena is accomplished via tidal gauges – in situ sensors that provide hourly records of sea level changes with high accuracy. They are widely deployed in Europe and in the US, but are sparse in other world regions, especially in developing countries.

As part of its efforts to fight climate change and its impact, ESA Φ-lab, a globally recognised entity in groundbreaking innovation in Earth observation (EO), is conducting research on the use of machine learning to predict storm surges. At the beginning of May, Google Research Flood Forecasting invited Patrick Ebel, an internal research fellow at Φ-lab and lead for the storm surge forecasting project, to give a talk about the recent work and approaches developed in the lab.

To counteract the difficulties presented by traditional prediction methods, this Φ-lab project introduces a global dataset of in situ tidal gauge time series paired with satellite-derived atmospheric and ocean state reanalysis products and global land-sea masks. Neural networks, a machine learning approach designed to recognise patterns, can then assimilate these data to provide forecasts with large spatial coverage. The novelty in this research lies in the generalisation of prediction to locations where tidal gauges are not available, assisting under-served communities with less in situ monitoring infrastructure and aligning with the UN Early Warnings for All initiative goals.

Patrick Ebel presents the work about storm surge forecasting developed at Φ-lab

Google software engineer Oren Gilon was one of the organisers of the session: “We at Google have been very excited to hear about Patrick’s work. Understanding that approaches similar to those that have been applied to riverine floods can be applied to coastal floods changes the way we look at this problem. We hope to find ways to collaborate on this matter in the future.”

The next steps for the project include replacing retrospective reanalysis products with recently developed forecasting models, incorporating data from satellite altimetry, modelling of impact at landfall, and translating storm surges into predictive flood maps. This work will be further discussed at the next IEEE/CVF Computer Vision and Pattern Recognition Conference and MedCyclones Workshop & Training School.

“I would like to thank Google Research for having organised the gathering and for the enthusiastic discussion on the storm surge topic. Our innovative approach to this research brings a fundamental change to the way natural hazards are predicted,” comments Giuseppe Borghi, Head of Φ-lab. “I am keen to see this dialogue between Google and ESA continue, as we work together to address some of our most pressing societal challenges.”

Further details on the research and its initial results can be found on arXiv.

To know more: Φ-lab, Google Research Flood Forecasting

Photo courtesy of Pexels/George Desipris and ESA/Patrick Ebel