New robotic vision system processes event-camera data up to 440 times per second

Key Takeaways

• Penn engineers developed F³, an AI tool that converts event-camera signals into usable robotic vision data.

• F³ processes high-definition data 120 times per second and VGA-resolution data 440 times per second.

• Researchers tested F³ using data from cars, drones and quadruped robots across different environments.

US researchers have developed an artificial intelligence tool that can process robotic vision data at up to 440 times per second, potentially allowing drones, autonomous vehicles, and other robots to respond faster to their surroundings.

Developed by engineers at the University of Pennsylvania, the system, called Fast Feature Field (F³), processes information from event cameras, which detect changes in light rather than capturing conventional images.

In tests, algorithms using F³ operated roughly two to five times faster than previous approaches. The system processed high-definition event-camera data 120 times per second and reached 440 times per second at lower VGA resolution.

The researchers say the technology could help robots recognize objects, estimate distances, and track movement more efficiently, particularly in challenging lighting conditions.

Processing millions of signals

Unlike conventional cameras that capture complete images at fixed intervals, event cameras continuously detect changes in light intensity at individual pixels.

This allows them to capture rapid movements without recording entire frames. However, interpreting their output remains difficult because the cameras generate millions of individual signals rather than recognizable images.

Existing approaches typically require specialized algorithms for different tasks, limiting their wider use in robotics.

More from AI and Robotics

The Penn researchers addressed this problem by developing a neural network that learns to identify useful patterns in event-camera data.

Instead of programming the system separately for object recognition, motion tracking, or distance estimation, the team trained F³ to predict future events based on previous signals.

“To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it,” said Richeek Das, the paper’s first author.

This approach allows the system to extract information about a scene and its movement without requiring a separate processing method for every application.

The researchers also found a way to reduce unnecessary calculations.

Although event cameras can generate tens of millions of signals every second, approximately 95% of possible positions in the data contain no events.

By processing only locations where changes occur, F³ avoids spending computing resources on empty areas. This optimization helped the researchers achieve the processing speeds needed for real-time robotic perception.

Testing across robotic platforms

The team evaluated F³ using event-camera data collected from cars, four-legged robots and flying drones.

Tests included object recognition, distance estimation, and motion tracking across environments ranging from nighttime urban streets to crowded indoor spaces.

According to the researchers, F³-based approaches achieved state-of-the-art performance on multiple perception tasks while operating substantially faster than earlier methods.

The system also allows existingcomputer-visionsoftware to be adapted for event-camera data, potentially reducing the need to develop specialized algorithms for individual applications.

The researchers are now incorporating F³ into autonomousdronesdesigned to navigate cluttered and poorly lit environments.

Eventually, they hope to integrate the technology directly into event-camera hardware, allowing raw signals to be converted into usable visual information without requiring separate processing software.

“Our goal is to make using an event camera as straightforward as using a conventional one,” said Pratik Chaudhari, associate professor at the University of Pennsylvania.

The researchers believe this approach could make event cameras more practical for autonomous systems that must interpret rapidly changing environments.

The findings were published in the journalScience Robotics.

Get the latest in engineering, tech, space & science - delivered daily to your inbox.

By subscribing, you agree to our

and

You may unsubscribe at any time.

0

COMMENT

Subscribe to

Today!

Access to exclusive content, expert insights and a deeper dive into engineering and tech. No ads, no limits.

By

With over a decade-long career in journalism, Neetika Walter has worked with The Economic Times, ANI, and Hindustan Times, covering politics, business, technology, and the clean energy sector. Passionate about contemporary culture, books, poetry, and storytelling, she brings depth and insight to her writing. When she isn’t chasing stories, she’s likely lost in a book or enjoying the company of her dogs.

• 1US ships first home-built centrifuge to boost nuclear fuel supply for next-gen reactors
• 210,000-ton US nuclear submarine to get new turbine generators for powering systems
• 3Humanoid robot achieves 85% success rate in real-life trial with AGIBOT's GO-3 model
• 4Fast nuclear reactors could use 100 times less uranium for clean power
• 5200-lbf jet engine's smaller sibling clears first test after just four months of work

Key Takeaways

• Penn engineers developed F³, an AI tool that converts event-camera signals into usable robotic vision data.

• F³ processes high-definition data 120 times per second and VGA-resolution data 440 times per second.

• Researchers tested F³ using data from cars, drones and quadruped robots across different environments.

US researchers have developed an artificial intelligence tool that can process robotic vision data at up to 440 times per second, potentially allowing drones, autonomous vehicles, and other robots to respond faster to their surroundings.

Developed by engineers at the University of Pennsylvania, the system, called Fast Feature Field (F³), processes information from event cameras, which detect changes in light rather than capturing conventional images.

In tests, algorithms using F³ operated roughly two to five times faster than previous approaches. The system processed high-definition event-camera data 120 times per second and reached 440 times per second at lower VGA resolution.

The researchers say the technology could help robots recognize objects, estimate distances, and track movement more efficiently, particularly in challenging lighting conditions.

Processing millions of signals

Unlike conventional cameras that capture complete images at fixed intervals, event cameras continuously detect changes in light intensity at individual pixels.

This allows them to capture rapid movements without recording entire frames. However, interpreting their output remains difficult because the cameras generate millions of individual signals rather than recognizable images.

Existing approaches typically require specialized algorithms for different tasks, limiting their wider use in robotics.

More from AI and Robotics

The Penn researchers addressed this problem by developing a neural network that learns to identify useful patterns in event-camera data.

Instead of programming the system separately for object recognition, motion tracking, or distance estimation, the team trained F³ to predict future events based on previous signals.

“To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it,” said Richeek Das, the paper’s first author.

This approach allows the system to extract information about a scene and its movement without requiring a separate processing method for every application.

The researchers also found a way to reduce unnecessary calculations.

Although event cameras can generate tens of millions of signals every second, approximately 95% of possible positions in the data contain no events.

By processing only locations where changes occur, F³ avoids spending computing resources on empty areas. This optimization helped the researchers achieve the processing speeds needed for real-time robotic perception.

Testing across robotic platforms

The team evaluated F³ using event-camera data collected from cars, four-legged robots and flying drones.

Tests included object recognition, distance estimation, and motion tracking across environments ranging from nighttime urban streets to crowded indoor spaces.

According to the researchers, F³-based approaches achieved state-of-the-art performance on multiple perception tasks while operating substantially faster than earlier methods.

The system also allows existingcomputer-visionsoftware to be adapted for event-camera data, potentially reducing the need to develop specialized algorithms for individual applications.

The researchers are now incorporating F³ into autonomousdronesdesigned to navigate cluttered and poorly lit environments.

Eventually, they hope to integrate the technology directly into event-camera hardware, allowing raw signals to be converted into usable visual information without requiring separate processing software.

“Our goal is to make using an event camera as straightforward as using a conventional one,” said Pratik Chaudhari, associate professor at the University of Pennsylvania.

The researchers believe this approach could make event cameras more practical for autonomous systems that must interpret rapidly changing environments.

The findings were published in the journalScience Robotics.

Get the latest in engineering, tech, space & science - delivered daily to your inbox.

By subscribing, you agree to our

and

You may unsubscribe at any time.

0

COMMENT

Subscribe to

Today!

Access to exclusive content, expert insights and a deeper dive into engineering and tech. No ads, no limits.

By

With over a decade-long career in journalism, Neetika Walter has worked with The Economic Times, ANI, and Hindustan Times, covering politics, business, technology, and the clean energy sector. Passionate about contemporary culture, books, poetry, and storytelling, she brings depth and insight to her writing. When she isn’t chasing stories, she’s likely lost in a book or enjoying the company of her dogs.

• 1US ships first home-built centrifuge to boost nuclear fuel supply for next-gen reactors
• 210,000-ton US nuclear submarine to get new turbine generators for powering systems
• 3Humanoid robot achieves 85% success rate in real-life trial with AGIBOT's GO-3 model
• 4Fast nuclear reactors could use 100 times less uranium for clean power
• 5200-lbf jet engine's smaller sibling clears first test after just four months of work