Sony announces the world's first imaging sensors with AI capabilities baked in
The sensor uses stacked design to accommodate AI chip circuits
Only suitable for light AI inference
Sony today has announced the development of two new imaging sensors, the IMX500 and IMX501. What’s new about these sensors is that they pack an AI chip right into the sensor. Sony says the whole idea behind developing a sensor with an AI chip baked in is to circumvent the various issues that can come with cloud-based AI processing.
Sony IMX500 Sensor Structure
The Sony IMX500 is not a regular imaging sensor, but in fact, it utilizes Sony’s stacked sensor design. Sony had developed the ability to sandwich a DRAM layer between the optical layer of the sensor and its circuitry in order to increase the shooting buffer. This is what allowed smartphones to shoot at 960fps super slow-mo back when the feature was originally introduced. Now Sony has managed to pack an AI processing engine into this layer, instead of just having DRAM. The logic chip, as Sony calls it, includes an ISP to process the image signal, a DSP dedicated to AI signal processing and memory for the AI model. This approach has two distinct advantages. The sensor has a resolution of 12.3 megapixels and measures 1/2.3-inch diagonally.
How to Sensor and AI Chip work
Currently, most AI processing related to images happens on the cloud, which, according to Sony, can have privacy issues. Additionally, there’s the issue of connectivity, where if your phone loses its connection to the cloud, the AI processing comes to a halt. Sony’s approach alleviates some of these issues for sure. The AI chip can be programmed by the OEM with any kind of inference engine depending on where the sensor is deployed. Additionally, the sensor-chip assembly can be programmed to either output the image as a whole or forego the image data in favour of inference-based information. The fact that the on-board AI chip is programmable by the OEM truly makes the IMX500’s usage extremely flexible. It is interesting, however, that devices by Apple like the iPhone 11 Pro and those by Huawei like the Huawei P40 Pro utilise on-device AI processing and do not rely on the cloud.
Potential Use Cases
The Sony IMX500 sounds like a sensor that could be used in a large variety of ways. For smartphones, it could take the load off of the main SoC by running inference right on the sensor, while the higher-level AI tasks can still be conducted using the cloud. The sensor can also be used in security cameras, where it could be programmed to, for example, identify whether the person entering the building is wearing a mask or not. Given times of the COVID, this could actually be a very useful thing. The possibilities, however, are limitless.
Sony IMX500 and IMX501 Availability
Sony has mentioned that the IMX500 has already been shipped to partner OEMs while the IMX501 will start shipping in June. We don’t expect to see this sensor in the smartphone space for now given that the industry is currently focused on the megapixel war once again.
Follow Us
Swapnil Mathur
Swapnil was Digit's resident camera nerd, (un)official product photographer and the Reviews Editor. Swapnil has moved-on to newer challenges. For any communication related to his stories, please mail us using the email id given here. View Full Profile
