# AI & Aliens: New Eyes on Ancient Questions

Richard Cloete, Harvard University and SEAQR Robotics | MLOps Podcast | Episode 288 | 47:59
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ZUBrIlxlNdI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-aliens-new-eyes-on-ancient-questions
Published: 2025-02-04
Tags: multimodal, synthetic-data

## TL;DR
- Richard Cloete uses a network of cameras and other sensors to detect, track, and classify objects in the sky for the Galileo Project.
- Because suitable training data was unavailable, his team built AeroSynth with Blender 3D, Python, and open-source 3D models to create synthetic images.
- SEAQR Robotics is developing uncrewed surface vehicles that could monitor the ocean, support marine research, and provide data for anomaly detection.

## Summary
Richard Cloete describes how he applies machine learning to questions about unusual objects in the sky, interstellar objects, and the ocean. For the Galileo Project, he works with infrared cameras, visible cameras, acoustic sensors, magnetometers, and a hemispherical camera array. Since ordinary image datasets do not show aircraft and birds from sky-facing cameras, his team created AeroSynth, a synthetic data tool built with Blender 3D and Python. Models detect and track objects, while triangulation remains necessary for estimating distance, size, and speed. Cloete also works with the Minor Planet Center on models that classify astronomical objects, including possible interstellar objects, using numerical data. His startup, SEAQR Robotics, is developing autonomous surface vehicles with marine sensors, persistent monitoring, and possible underwater communication links. He is careful about what the current evidence can establish. Interesting observations still need data from multiple instruments before they can be verified.

## Key ideas
### The Galileo Project collects multiple types of sensor data before interpreting unusual events
[02:21](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=141s)
Richard Cloete explains that the Galileo Project combines infrared cameras, an all-sky visible camera, acoustic sensors, a magnetometer, and other instruments. The aim is to monitor as much of the electromagnetic spectrum as possible and detect unusual objects in the sky. The project expects ordinary objects such as birds and aircraft, then looks for observations that need further investigation. Cloete says the scientific approach is to put instruments on the ground, collect data, and let the data speak for itself. One system he works on, the DOL, uses about eight cameras around a hemisphere plus a camera at the zenith. Its video feeds support object detection and tracking.

### Synthetic data fills a gap that ordinary image datasets cannot cover
[04:18](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=258s)
Cloete found that existing datasets did not match the project's setting. A dataset labelled with airplanes might contain aircraft on a runway, on a shirt, or in a cartoon. Bird images could show birds on branches or printed on objects. The team therefore built AeroSynth, using Blender 3D, Python, and open-source 3D models. The tool places airplanes, birds, balloons, blimps, and drones at random positions and orientations within a camera's view. It varies distance, apparent size, and lighting. The resulting images train YOLO models, including YOLOv5 and newer versions. The team also tests tracking methods such as SORT and ByteTrack.

### Tracking trajectories matters because a single object image often says very little
[06:49](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=409s)
The system detects objects in video feeds and tracks them over time. Cloete says the trajectory may be more informative than the object's appearance, since a distant object can look like a small white dot. The team wants to identify movement that does not fit known natural or human-made phenomena. Alerts for unusual objects are a future goal, while detection and tracking are already working during the commissioning phase. The instruments are still being deployed and tested, and the team is building a baseline for what is normal at each site. Cloete stresses that an observation cannot be properly verified without multiple instruments recording the same event. Triangulation would provide distance, which would allow estimates of size, speed, and velocity.

### The current system has found unusual observations, but Cloete does not call them UAPs
[11:01](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=661s)
Cloete says the systems are not yet fully operational and do not continuously process data or send anomaly alerts. Manual review has produced observations that the team cannot explain, although the available evidence is often limited to one video. Other cameras may have been offline, and the acoustic system may not have been working. One example showed a round object moving at constant speed, stopping, remaining still, moving upward, and disappearing. The team could not see wings or propellers and could not find matching audio or views from other cameras. Cloete says a red-tailed hawk remains a plausible explanation. He refuses to make a stronger claim until the project has the right evidence and verification process.

### Edge processing reduces the need to transmit continuous video from remote sites
[13:39](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=819s)
The Galileo Project runs its systems at the edge. Sites have local storage and communication options, including 5G, but they do not continuously send live video. Instead, someone collects data from the previous week or two and transfers it to the Center for Astrophysics and then to a Harvard cluster for analysis. Cloete says this arrangement fits the current infrastructure and data volumes. The system can extract detections and tracks for later review. Site locations remain private because of the stigma around UAPs and concerns about people tampering with cameras, data streams, or instruments.

### Astronomical machine learning can sort the large data stream from the Vera C. Rubin Observatory
[17:57](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=1077s)
Cloete works with the Minor Planet Center on models for data from the Vera C. Rubin Observatory, a ground-based telescope in Chile that will observe much more of the sky and see farther than earlier systems. He wants to classify many object types and flag possible interstellar objects, including objects similar to 'Oumuamua. The current work uses numerical data rather than images. He discusses an existing binary-classification tool that produces a score from 0 to 100, with scores above 65 treated as interesting near-Earth objects. Cloete says false positives and missed near-Earth objects remain problems, so his work aims to improve classification accuracy and confidence.

### SEAQR Robotics applies persistent sensing to the ocean and its surface
[25:56](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=1556s)
SEAQR Robotics is adapting ideas from the Galileo Project for ocean monitoring. The company is developing uncrewed surface vehicles that could observe the sky, sea surface, and water below. Planned sensors include cameras, hydrophones, conductivity sensors, and salinity sensors. Possible users include marine biologists, the Coast Guard, defense teams, atmospheric researchers, climate researchers, and oceanographers. The vehicles could provide data through subscriptions, sell data packages, or let customers control a fleet for a mission. Cloete says persistent coverage would fill gaps left by short research missions, where measurements stop when a vessel returns.

### Wave energy and underwater communication are part of the vehicle design
[31:29](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=1889s)
The first SEAQR prototype is intended to monitor the ocean surface and the first 10 to 20 metres of water, along with the sky above it. Later vehicles could map the sea floor, deploy and retrieve probes, collect samples, and act as communication hubs. Acoustic modems could let underwater vehicles and observatories relay information to a surface vehicle, avoiding the need to surface before transmitting. Cloete is also exploring wave energy because existing systems can run out of power. The vehicles would move within assigned grid cells rather than remain perfectly stationary. Automatic Identification System data and obstacle avoidance would help them operate around shipping traffic.

### Anomaly detection could help astronomers find movement patterns that standard systems discard
[45:42](https://www.youtube.com/watch?v=ZUBrIlxlNdI&t=2742s)
Near the end of the conversation, Cloete suggests applying machine learning directly to telescope images as image quality improves. He also wants algorithms that detect unusual motion, including objects that stop or make right-angle turns. Existing astronomical algorithms are designed around expected movement, so they may lose an object when its behaviour changes. Cloete says unusual observations drive investigation and learning. He sees anomaly detection as useful in space, on Earth, and in other environments where large sensor streams make continuous human review impractical.

## Notable quotes
- Richard Cloete: "As scientists we just want to put instruments on the ground, collect data and let the data speak for itself." (03:09)
- Richard Cloete: "We don't want to rely on humans to be able to process that data. Instead we just build a machine learning model to do it for us." (10:24)
- Richard Cloete: "We can't say that they are a UAP, we can't say that they are a bird or an airplane, just because there's not enough data." (11:59)
- Richard Cloete: "The idea is to take the knowledge that I'd gained through the Galileo Project and build a network of these USVs." (26:59)
- Richard Cloete: "Anomalies is really what drives progress." (47:04)

## Tools & references mentioned
- MLOps Community
- Galileo Project
- Avi Loeb
- SEAQR Robotics
- Seeker Robotics
- AeroSynth
- Blender 3D
- Python
- YOLOv5
- YOLOv8
- YOLOv11
- SORT
- ByteTrack
- Minor Planet Center
- Vera C. Rubin Observatory
- 'Oumuamua
- DIGITS2
- James Webb Space Telescope
- Hubble Space Telescope
- Center for Astrophysics
- Harvard University
- University of Cambridge
- Stanford University
- The Sol Foundation
- Starlink

## Who should watch
- You are building computer vision systems for unusual or poorly labelled environments and need ideas for generating training data.
- Your work involves large sensor streams at remote sites, where local storage, intermittent transfer, and later batch processing are more practical than live cloud analysis.
- You are interested in autonomous ocean monitoring or astronomical object classification and want to hear where machine learning still needs better sensor coverage and verification.

## Related talks

- [Beyond AGI, Can AI Help Save the Planet?](https://mlopstalks.com/talks/beyond-agi-can-ai-help-save-the-planet) (Patrick Beukema, Allen Institute for Artificial Intelligence (AI2), 54:05)
- [The Missing Data Stack for Physical AI](https://mlopstalks.com/talks/the-missing-data-stack-for-physical-ai) (Nikolaus West, Rerun, 52:43)
- [Unbearable Lightness of Data: The Weight of Information](https://mlopstalks.com/talks/unbearable-lightness-of-data-the-weight-of-information) (Rohit Krishnan, Bodo.AI, 54:17)
- [From Spikes to Stories: AI-Augmented Troubleshooting in the Network Wild](https://mlopstalks.com/talks/from-spikes-to-stories-ai-augmented-troubleshooting-in-the-network-wild) (Shraddha Yeole, Cisco ThousandEyes, 11:49)
- [Innovative Gen AI Applications: Beyond Text](https://mlopstalks.com/talks/innovative-gen-ai-applications-beyond-text) (Diana C. Montañes Mondragon & Nick Schenone, QuantumBlack, 54:45)