Richard Cloete uses a network of cameras and other sensors to detect, track, and classify objects in the sky for the Galileo Project.
2
Because suitable training data was unavailable, his team built AeroSynth with Blender 3D, Python, and open-source 3D models to create synthetic images.
3
SEAQR Robotics is developing uncrewed surface vehicles that could monitor the ocean, support marine research, and provide data for anomaly detection.
Summary
Richard Cloete describes how he applies machine learning to questions about unusual objects in the sky, interstellar objects, and the ocean. For the Galileo Project, he works with infrared cameras, visible cameras, acoustic sensors, magnetometers, and a hemispherical camera array. Since ordinary image datasets do not show aircraft and birds from sky-facing cameras, his team created AeroSynth, a synthetic data tool built with Blender 3D and Python. Models detect and track objects, while triangulation remains necessary for estimating distance, size, and speed. Cloete also works with the Minor Planet Center on models that classify astronomical objects, including possible interstellar objects, using numerical data. His startup, SEAQR Robotics, is developing autonomous surface vehicles with marine sensors, persistent monitoring, and possible underwater communication links. He is careful about what the current evidence can establish. Interesting observations still need data from multiple instruments before they can be verified.
The Galileo Project collects multiple types of sensor data before interpreting unusual events
Richard Cloete explains that the Galileo Project combines infrared cameras, an all-sky visible camera, acoustic sensors, a magnetometer, and other instruments. The aim is to monitor as much of the electromagnetic spectrum as possible and detect unusual objects in the sky. The project expects ordinary objects such as birds and aircraft, then looks for observations that need further investigation. Cloete says the scientific approach is to put instruments on the ground, collect data, and let the data speak for itself. One system he works on, the DOL, uses about eight cameras around a hemisphere plus a camera at the zenith. Its video feeds support object detection and tracking.
Synthetic data fills a gap that ordinary image datasets cannot cover
Cloete found that existing datasets did not match the project's setting. A dataset labelled with airplanes might contain aircraft on a runway, on a shirt, or in a cartoon. Bird images could show birds on branches or printed on objects. The team therefore built AeroSynth, using Blender 3D, Python, and open-source 3D models. The tool places airplanes, birds, balloons, blimps, and drones at random positions and orientations within a camera's view. It varies distance, apparent size, and lighting. The resulting images train YOLO models, including YOLOv5 and newer versions. The team also tests tracking methods such as SORT and ByteTrack.
Tracking trajectories matters because a single object image often says very little
The system detects objects in video feeds and tracks them over time. Cloete says the trajectory may be more informative than the object's appearance, since a distant object can look like a small white dot. The team wants to identify movement that does not fit known natural or human-made phenomena. Alerts for unusual objects are a future goal, while detection and tracking are already working during the commissioning phase. The instruments are still being deployed and tested, and the team is building a baseline for what is normal at each site. Cloete stresses that an observation cannot be properly verified without multiple instruments recording the same event. Triangulation would provide distance, which would allow estimates of size, speed, and velocity.
The current system has found unusual observations, but Cloete does not call them UAPs
Cloete says the systems are not yet fully operational and do not continuously process data or send anomaly alerts. Manual review has produced observations that the team cannot explain, although the available evidence is often limited to one video. Other cameras may have been offline, and the acoustic system may not have been working. One example showed a round object moving at constant speed, stopping, remaining still, moving upward, and disappearing. The team could not see wings or propellers and could not find matching audio or views from other cameras. Cloete says a red-tailed hawk remains a plausible explanation. He refuses to make a stronger claim until the project has the right evidence and verification process.
Edge processing reduces the need to transmit continuous video from remote sites
The Galileo Project runs its systems at the edge. Sites have local storage and communication options, including 5G, but they do not continuously send live video. Instead, someone collects data from the previous week or two and transfers it to the Center for Astrophysics and then to a Harvard cluster for analysis. Cloete says this arrangement fits the current infrastructure and data volumes. The system can extract detections and tracks for later review. Site locations remain private because of the stigma around UAPs and concerns about people tampering with cameras, data streams, or instruments.
Astronomical machine learning can sort the large data stream from the Vera C. Rubin Observatory
Cloete works with the Minor Planet Center on models for data from the Vera C. Rubin Observatory, a ground-based telescope in Chile that will observe much more of the sky and see farther than earlier systems. He wants to classify many object types and flag possible interstellar objects, including objects similar to 'Oumuamua. The current work uses numerical data rather than images. He discusses an existing binary-classification tool that produces a score from 0 to 100, with scores above 65 treated as interesting near-Earth objects. Cloete says false positives and missed near-Earth objects remain problems, so his work aims to improve classification accuracy and confidence.
SEAQR Robotics applies persistent sensing to the ocean and its surface
SEAQR Robotics is adapting ideas from the Galileo Project for ocean monitoring. The company is developing uncrewed surface vehicles that could observe the sky, sea surface, and water below. Planned sensors include cameras, hydrophones, conductivity sensors, and salinity sensors. Possible users include marine biologists, the Coast Guard, defense teams, atmospheric researchers, climate researchers, and oceanographers. The vehicles could provide data through subscriptions, sell data packages, or let customers control a fleet for a mission. Cloete says persistent coverage would fill gaps left by short research missions, where measurements stop when a vessel returns.
Wave energy and underwater communication are part of the vehicle design
The first SEAQR prototype is intended to monitor the ocean surface and the first 10 to 20 metres of water, along with the sky above it. Later vehicles could map the sea floor, deploy and retrieve probes, collect samples, and act as communication hubs. Acoustic modems could let underwater vehicles and observatories relay information to a surface vehicle, avoiding the need to surface before transmitting. Cloete is also exploring wave energy because existing systems can run out of power. The vehicles would move within assigned grid cells rather than remain perfectly stationary. Automatic Identification System data and obstacle avoidance would help them operate around shipping traffic.
Anomaly detection could help astronomers find movement patterns that standard systems discard
Near the end of the conversation, Cloete suggests applying machine learning directly to telescope images as image quality improves. He also wants algorithms that detect unusual motion, including objects that stop or make right-angle turns. Existing astronomical algorithms are designed around expected movement, so they may lose an object when its behaviour changes. Cloete says unusual observations drive investigation and learning. He sees anomaly detection as useful in space, on Earth, and in other environments where large sensor streams make continuous human review impractical.
"We don't want to rely on humans to be able to process that data. Instead we just build a machine learning model to do it for us."Richard Cloete10:24
Who should watch
You are building computer vision systems for unusual or poorly labelled environments and need ideas for generating training data.
Your work involves large sensor streams at remote sites, where local storage, intermittent transfer, and later batch processing are more practical than live cloud analysis.
You are interested in autonomous ocean monitoring or astronomical object classification and want to hear where machine learning still needs better sensor coverage and verification.
