Will Computer Vision Kill the IoT Sensor?

we're back in the current insight studio for a panel discussion on the topic of will computer vision kill iot sensors we love a dramatic title to get people talking and let me introduce our panel we've got from nvidia charbel own from access communications andrea sorry and from shredder nicholas koitken welcome to the panel so is there something computer vision can do that iot sensors can't do and you make these cameras Andrea so maybe I'll start with you sure thanks so I think definitely they're complementary to start with the beauty of cameras is that they provide first of all video images visual understanding what's going on so you have visual understanding about the data that is collected and next to that the cameras are becoming more intelligent they have capabilities for deep learning analytics so there is the tendency to try to do more with less use the camera to do more tasks for multiple stakeholders for multiple use cases and Nicholas what kind of use cases are you seeing people adopt your streetlights with cameras on what are people turning to computer vision for the first use case is security like we're seeing so we have some products that are smart poles where we can integrate different applications and so depending on the requirement of the cities the citizens the space in which we are we can we can then customize i would say to the requirements of the of of in the needs of the cities security being one of them but there are many other ones sometimes surprising ones so the obvious ones i would say are the ones that are then linked to mobility parking management but for instance we had recently a project in germany where we were asked to count the number of mosquitoes which normally is rather in computer vision a thing you don't want to do because you would like to see beyond I would say the mosquitoes what is happening but in this case it's it was a project where the quality of the environment or they looked for indicators for the quality of the environment and actually the number of mosquitoes and the size of mosquitoes apparently brings I would say a indicator on how well and sustainable, I would say, the environment is there. So here, an application which is probably unusual, but replaces maybe, to a certain extent, an IoT sensor, which would be an environmental sensor. So yes, with camera, we can do a lot of things. It might also be hard to develop a sensor that can detect mosquitoes without being so small. And Sharbella, if there's any surprising uses you've heard of computer vision? To me, a camera is an IoT sensor, and a camera could be part of an independent platform because an IoT sensor is a sensor that measures air quality, a sensor that measures temperature, a camera that sees. So all these are IoT sensors. I think the camera today, we say a picture is worth a million worth, right? Today, if a human eye can see something, a human brain... can perceive and analyze, camera plus AI are able to do that at a scale, allowing human to focus on decision-making as opposed to mundane tasks of observing frames, right? Especially when we're talking about 30 frames, 40 frames, 60 frames per second. Now, can the camera do some extreme stuff? Yes. It can count mosquitoes. We have partners looking at tracking honey behavior, bees behavior for honey production. what flowers they go to. We have partners looking at the pig farm stocks to analyze livestock health and productivity and food they eat and activities they do. This is the environment. We have salmon fish monitoring partner, but also we have the traditional operational efficiency, safety, and health. Transport is a big area. All airports are looking at it. So it's used for everything a camera can see. Now, the difference is, in the past, a legacy IoT approach would be to record those videos on a recorder and access them if, when needed, based on a court order or an event. The difference now is you have the ability to convert everything a camera sees in real time into in sight. So now a camera is being used to transfer dumb stream. video streams into data and insight. What can you do with this? Anything you want. You can understand, react to behavior, but also you can move into predictive analytics. So when Shredder are doing a street light, yes, they can detect a lot of things, counting people, free car park. They can put an air quality sensor, correlate between traffic light and air quality. This is all possible because technology. Now, you need a- We need a partner like them to enable this to happen. We need a partner like them to provide the right sensor, camera in that case, and other sensors to do the application or the workload. What is the application? I think what is your imagination, genuinely speaking? What are some of the applications where perhaps the camera isn't as good as a sensor? Parking, for example, you can do with magnetic sensors under the parking yard. Which is better for something like that? I think it really depends on... the local conditions in which you are and what are the requirements. There are questions, for instance, on reliability of the information that you get. Some sensors might be at some point more reliable than, I would say, what a camera can see or what image processing can do. But in other cases, a camera is better. So a parking sensor, as you mentioned, if it is a camera. some weather condition doesn't work but the same for the camera when you have some mist or some when the visibility is not very good or when you have trees the camera might be limited so it's very important actually when you deploy a smart city project to take all these conditions into consideration and then to have probably a mix of both the most interesting i think what i see now is due to edge computing we are capable of reacting very very fast so for lighting for instance detect a human being or even an animal and immediately send a message to another object like another luminaire and we don't need to go through the cloud for that and I think this actually gives us the opportunity to make all this information from different sensor work together and to have what we would call probably data fusion so that means I confirm what I see for instance on the camera with a microphone a few meters down the road and I say yes not only the camera tells me at 95 % it's a car but with a microphone and maybe a signature of the microphone I could say no it's 100 % a car and actually it's a car that emits too much noise or and so I think this is what is interesting is that while the camera brings a lot of information a lot of other IOT devices can complement this information to really enrich. the information which then allows to get more insights and not just data. I was going to ask actually about accuracy because an underground sensor is fairly easy. It's either on or off, but computer vision has to interpret what it's looking at and can make mistakes. So when we're planning cities and using this kind of data to make informed choices about our lives, how can we reduce the amount of mistakes? I tend to disagree with you. So I will tell you a real life. example remote energy farms they use sensors motion sensors and this they have well in the case I worked on and I looked around it's a common percentage accuracy 75 percent accuracy at best yes they detect motion but the motion of what is it a plant shaking running a rabbit a dog or a human breaking into the location so yes it can give you a motion yes or no but just an anonymous motion the camera now here is what I want to change your opinion about the camera can go up to 98% accuracy I don't think anybody have reached that and you can go up to 99 if you have enough data to keep training because the way AI works and this is what we do forcing cameras and street light with the power of AI, and that's what we do, right? So it's about training and influencing. Before the camera can do what it does, right, we've got to train it, train the AI model, which means we're going to give the AI imagery of a certain thing and say, this is a cat, this is a cat, this is a cat, each in a different environment, different color, different shape, different color background, one in nature, all variation, and we're going to train them. model, and then we're going to ask it again, is this a cat? We keep doing this in iteration and cycle until we hit the 99%. 97% and above is good enough, but you can keep going. Now, you move into inferencing. You port this model, and you put it on the streetlight, right? And running on the camera. Now, it could be all these could be cloud connected or back and data center connected. If the response time is not critical, if he wants to analyze how many people are passing, that's not time critical. He can stream it line if the cost is... So portability of the model is important, edge to cloud. But if he's detecting people in distress, he want to know in a nanosecond. He wants the police or the emergency service. So the application, the use case, the workload defines, in my opinion, all these variations. but all of them are possible and Axis are doing all kind of cameras and they're doing edge because for their use case edge is the right case i have other scenarios where they choose the cloud i don't think one is right one is wrong i think the use case defines what you do or not but don't underestimate a camera however it's not the only answer and i totally agree with with schroder because what you're trying to do define if you need to fuse data, because at the end, camera is data. It's a visual data where other sensors are different type of data. So you need, in some cases, all of the data. I suppose with a camera, at least, you get the visual so you can get a human to check it. I'm thinking about those smart doorbells you get. Sometimes it gives you an alert and says, I think someone's at the door, and you check it, and there wasn't anyone there. But then at least you've verified the information. The flight passed by. My first alarm system. early days i wanted a remote camera before things became what it is now and luckily i i had so many alerts on my iphone because there was the light changes on the wall at the early days right now now the system you have is so accurate it's not going to be fooled by because the camera detection was taught that's a light don't disturb me with the light disturb me when you see a human or fire or smoke you see where i'm going with that it's what you the AI to do. AI doesn't do magic, AI is the outcome of what we do, what he wants to do. And do you have a view on privacy when it comes to edge versus the cloud? Is it better to process things in the camera if you can, if it's suitable to? Let's simplify the chain of custody and simplify the distribution of sensitive data a lot. You can even use the camera for sending not pictures, only data, only in the context of the picture. is required so definitely this is an enabler but if I may I can I would like to come back on your previous question because I think Chevrolet inspired me I mean the beauty of video is really that you can verify on the recording not only the accuracy but also if you have missing information if you have missed an alarm and that enables really to raise the quality of your solution to the max also the thing is that Until now, we basically use analytics to detect what we know, what we expect to happen. But with the latest technology, we can also discover things that are unknown, that we were not expecting, that the deep learning analytics were not trained for. So, to make an example, we can detect unsafe conditions. If we look at the pedestrian crossing, we know the amount of incidents through reporting, fatalities. right? But we don't know how many near -missed incidents we have. By recording video, by understanding the paths, the patterns of cars, the number of braking wheels sound that we collect from that pedestrian zebra, we can understand how dangerous it is actually, not based on the fatalities on the reporting of the incidents, but on the amount of potential risk situation. And let's talk about privacy. Sometimes people are concerned about cameras, even if they're not streaming data, they don't necessarily want high resolution cameras. There are some providers that are producing IOC cameras that have very low resolution sensors, good enough to see if there's a car in a parking space, but not 4K, good enough to see which car or the registration plate. Which solution should we be looking towards, do you think? I think the resolution is not the answer to that. I would say it's really also edge computing, for instance. can allow really to give this safety or the satisfaction I would say to the customer to say okay we will not stream the data we will locally on the on the on the computer inside for instance of the luminaire we analyze I would say what is happening and we just send the data which is out and there is and you can guarantee indeed that there's going to be no I would say no no privacy question that is linked to that another way of doing is maybe not using cameras and then as i said use different sensors and bring this information i would say of the different sensors together to try so radar for instance and then microphones and others or lidars and then to try i would say to compute um an image if you want you know with with this information with but which does not indeed identify people people personally then there is another aspect it's the way how you I would say the position in which the camera is, because there are some positions, especially when you think about Luminaire with a horizontal, I would say, top -down view, you might actually never see the faces of people, right? You might just see the top of the head of people or of cars. So there is a lot of ways of, I would say, explaining to the customer how actually we can guarantee privacy and that. But if you just allow me to come back to the IoT aspect and the data fusion of it. I think this is where it's very important that a lot of manufacturers work together to bring open and interoperable systems. Because at the end of the day, what the customer wants is all this data to come together at some point in platforms and that we can help them really to make sense of everything that is identified, I would say, in the public space or indoor spaces. And so this is also an aspect which I think is important. So, you can do… different ways of contouring, I would say, the question of privacy and ensuring to your customer that indeed he will be able to guarantee to his citizens that the privacy of citizens, for instance, are not hurt. And did you have a view that perhaps higher resolution is sometimes better? It's maybe a shame if we deploy lots of low resolution cameras? I will tackle that question by... looking at what problem are you trying to solve? What is it that you're trying to do? And what is the parameters you're working within from legislation, guidelines, et cetera, privacy, and so on? And the answer would be sometimes yes, sometimes no. So think about it this way. AI is making genuinely the impossible possible. It's solving problems that we couldn't solve before because this is reality. This is what customers are telling us. us this is not something we were able to do. So when you want to deploy a system you need to think about different aspects. You want to use the camera as a imagery device and transport the data over 5G whatever you want to a remote processing center data center cloud to process you have to answer the question is that secure what's the cost of the bandwidth what's the latency what's the privacy challenges I face am I allowed to do that if yes that could be what's the ROI what's the cost benefit of this on the other hand and I agree with in their decision and in this edge is but inheritance is privacy built why because you're not retaining the data you're processing the data so when I look at the frame I convert it into metadata when I see you in a frame I'm not I'm not saying it's you, I'm saying it's a male wearing orange and black. You see what I mean? If that's what you want, that's what I mean. Or the backpack, whatever you want. Or he's standing, or sleeping, or he's in panic, or he's fighting. Whatever you wanted to do, I don't know who you are, I got what I want. Because the objective is not to know who you are, it's to understand the behavior so I can respond to it and solve a problem X and Y. So, to me, always data privacy is a concern. higher resolution lower resolution is a question what do you want to run on the system if you're buying a system to do one function which is counting vehicles yeah you can look at the shadows of the vehicle that's good enough which is fine i don't have an issue with that right but if you're looking at a system that he wants to analyze the type of the vehicle the age group as a random example that low resolution camera is not going to give you that value again an ROI question you see you see what I'm going with that what do you want so there is no one size fit all and there is no one answer and besides besides the the knowing uh how many people I have I want to count the cars and then I want to see how certain if there is a rain and I want to so what is interesting with cameras is that actually you can build a lot of use cases out of one single object put I would say in a repurposed infrastructure like like streetlight it's genuinely one picture worth a million word it has a million value because camera shows you a lot what do you want to see we show one picture to the three of us and everyone will be looking at different object what is it that you want to solve defines everything else the architecture the edge the cloud the resolution high resolution then the parameters around which is legislation guideline what is not allowed, not allowed, defines edge, not edge, encrypted. We have systems today that can run all of this with ghosting the whole frame. All I know is that a person, I don't know, anything about resolution here is not the answer. I suppose the resolution is a limiting factor, deliberately so. If you rolled out across a whole city, every streetlight had a low resolution sensor on it, that couldn't be physically exploited for something else. Whereas, in theory, if you had 4k cameras on every lamppost we went into another pandemic it's a very easy thing then for a local government to say we're going to use those ai powered cameras now to monitor whether people are leaving the house and perhaps people wouldn't want that you cannot look at technology alone you got to look at technology you're going to look at the people using the technology what they're allowed to do not to do and the process is put in place to manage this now if you're doing edge there is no Imagery of people to look at because there's no imagery of people full stop. There is a metadata of anonymous people It's what you set up the system to do Of course if you set up a system to identify him and that's not allowed you're already starting from the wrong place You're not supposed to do that So yeah in the absolute yeah, but these data are encrypted Assuming you're gonna transport the data to the cloud because non -critical operation where it's allowed these has to be encrypted what do you use them for has to be defined by a set of processes and guideline I don't believe in a single source of trust that's the same challenge with banking right there's a distributed trust with encryption with our medical data right how do you know that your doctor is not using the data for the wrong purpose or how do you know that the police is not tracking you we should have a system in place to remove the trust from the individual and put it in a distributed trust. That's my opinion. I think technology has to develop in terms of improving resolution and pixel quality, but also offering all kinds of capabilities to make sure that the data owner is in control about the data. And as you said, it's a matter of trust. I mean, it's really a matter of giving tools to data owners to be trustworthy, rather than... on the resolution and the pixel description. I think the good thing of deep learning and analytics in general is that can be also used in favor of privacy. We can use the deep learning to enable new way of masking faces, masking dynamics, or ensuring that we send the pictures to law enforcement and data to traffic authorities, for instance. benefit and the enablers rather than the resolution itself what is the public reaction towards IOT sensors IT cameras generally quite positive then it's it's always very subjective but over the last 10 years of the acceptance of video surveillance in cities has changed I think more and more people and authorities sees this the camera is an auxiliar resource to improve the city services rather than for controlling people. I remember when Facebook recently brought out some video camera glasses and they told me at the time that the reason they did that was to prepare people for camera glasses as we move towards an AR future. Do you think perhaps half the battle has already been done because we're so used to having cameras around in cities especially that people are less bothered now by IoT sensors that maybe don't feel threatened by them? it's funny you mentioned that because I was going to answer your question jokingly saying yes people accept cameras particularly when they are when they are hidden and so and so when you have them in the glasses obviously you don't have a big I would say a sign on your head saying I have I'm wearing I'm wearing so I think the example of that is yeah when it's when it's hidden obviously people don't know about it and so it's not a question of choice right but there is also a lot of places where basically the cameras bring even if they don't work actually bring feeling of safety and you know when we allow we're working a lot with lights also at night because light brings feeling of safety sometimes we don't need to have certain light levels but we still have them because few people feel more safe so I think it's every one of us has a bit of a duality here with cameras on one side you know we don't want to be spied on but on the other side you know, they also bring a lot of advantages. Might it be on security, or might it be just on helping us solving traffic problems, right? Informing us in advance of what is going to happen, and so on and so forth. So, and when you look, as you said, I mean, the glasses or even autonomous cars, at some point, you know, the environment needs to be captured so that we are able to increase automation in general in the public space. it's a bit difficult to escape the fact of not having cameras or as we call it image sensors in the public space indeed. I wanted to ask you a question about street lights in particular as that's your business. For the future of fully autonomous cars we're going to need a lot of data about our cities which your street lights can capture but once we have fully autonomous cars we might not need street lights anymore on the motorways because there won't be people looking at the road. Absolutely. one of the the motors that we have is also to save resources and we work a lot and we work very hardly on reducing energy consumption especially now with the with the crisis in which we in which we are so yes of course there's going to be less light on motorways that said everywhere where you have people that means on streets where you have bars where you have what we call the night economy yeah you're going to continue to have light because this where people need to where people need to meet where people need to sometimes in in some some countries you know the light provided outside allows young people to study or to because they don't have lights inside of their homes and so forth so public lighting is something that is definitely not going to disappear the i think the market might shrink or the applications might be less needed but good so because you know we're going to use less energy on on that but other aspects where where people are involved where I think at night we still will have a public lighting I believe yeah. I wanted to end by just asking what are some of the things you're excited about that computer vision can bring what what's next in this space Andrea? Definitely going beyond public safety so expanding toward environmental monitoring, improving detection of wildfire detection, improving the understanding about the environmental like rain precipitation, snow on the ground, visibilities, all these kind of things really enhance the use of the camera for multiple scopes. There is one thing that the camera can cannot do with video analytics which is as we say you can see here talk but when it comes to smell to understand the air quality we need to combine two sensors so that is an example where two different sensors need to be combined and if we want to correlate traffic conditions presence of people with the quality of air we need to combine two sensors. I believe there is a fusion of technology happening so he touched on it is when so to train AI you need data you can train autonomous vehicle through the real world physical world data which is going to take you forever or you could do what NVIDIA is doing where we built a virtual world to train autonomous vehicle and that's the engine that's powering all the autonomous vehicle that use nvidia platform which you you know about so the same concept is happening now with camera so what i'm excited about is the fusion of the physical world and the digital world which is what's going to power our next 30 years of evolution right the metaverse and nvidia has omniverse which is our toolkit to enable and develop them now when you see camera coming from the physical world feeding the digital world and in a continuous loop the possibilities are unlimited that is what I'm excited about personally well we certainly got a lot to look forward to thank you so much to our entire panel thanks for being here thank you for inviting us and don't forget you can see more of our panels on our website check out current insights.com