The technology powering Gemini Live’s camera and screen sharing is called Project Astra. It’s available as an Android app for trusted testers, and Google today unveiled agentic capabilities for Project Astra, including how it can control your Android phone.
Google demoed the new Project Astra capabilities on a Pixel 9 Pro. In this example, a person is having a live conversation while repairing their bicycle. It starts with the user requesting and Astra finding a PDF manual online. Upon opening, Astra exists as a floating bubble while you see its touch input and a screen glow as it scrolls the file to the desired page.
We next see it opening the YouTube app to search results, scrolling for a video, and then playing. The app then searches the user’s Gmail for information from a conversation. Shown a scene with multiple buckets filled with parts, Astra highlights the correct one on the screen (which something Google teased for Lens years ago at I/O.)
Another impressive capability is being able to make calls that seek out specific information on your behalf. In this case, it’s to a bike shop asking if a part is available, including being able to call back to confirm that the cyclist wants it.
We see Astra is in the middle of a response when it’s interrupted by another person. Astra automatically pauses and ignores that irrelevant conversation. The user then asks to resume the response, with the session continuing this entire time. Meanwhile, when they ask for a bicycle basket that can fit a dog, Astra’s memory capacity cites the dog’s name.
Google calls this “Action Intelligence,” with Gemini leveraging Content Retrieval, Interface Control, Agent Highlighting, Call Assistance, Knowledge Grounding, Context-Aware Dialogue, Personalized Shopping, and Native Audio Dialogue.
With Project Astra, Google wants to build a “universal AI agent that can be truly helpful in everyday life.” This Project Astra app is how Google tests capabilities that will eventually debut in Gemini Live for consumers.
Our ultimate vision is to transform the Gemini app into a universal Al assistant that will perform everyday tasks for us, take care of our mundane admin, surface delightful new recommendations, making us more productive and enriching our
FTC: We use income earning auto affiliate links. More.
Comments