Google Gemini can now read your screen to give smarter, context-aware answers: But there’s a catch

Google is now reportedly testing a new Gemini feature that will allow AI assistants to automatically understand and respond to what users are seeing on their screens, without requiring manual input. The update builds on Gemini’s existing ability to analyse attachments such as images, videos and documents and aims to make interactions with the assistant more contextual and seamless.

The feature internally refers to screen context and enables Gemini to detect when a user’s query relates to on-screen content. Instead of asking users to tap a dedicated “Ask about screen” button, the assistant can now independently capture a screenshot of the active screen and use it to generate relevant responses. It is similar to how Gmail alerts users when they forget to attach a file mentioned in an email.

The feature was spotted in development builds earlier this year, but it is now appearing on select devices as part of a limited rollout. The users who are getting access are greeted with an onboarding prompt explaining how the tool works, along with the option to disable it for privacy reasons.

When the feature is enabled, Gemini can automatically reference what is displayed on the screen while answering questions, such as explaining details in an image, offering help with a video, or providing context-specific assistance within an app. The company has stated that the users retain control, as it can be switched off at any time.

The users who got the access shared their feedback on Reddit and X. They stated that in some cases, it successfully identifies when a prompt relates to on-screen content, while in others, it may miss the context, depending on how the question is phrased. Google is also said to refine the feature further based on user feedback during the testing phase. It must be noted that the company has not announced the public rollout details at the moment. However, we expect that the feature could make its debut in the coming months in a phased manner.