According to the child safety organization Kids and Cars, in 2021, 23 children died of heat stroke in the United States after being left unattended in vehicles. Depending on conditions, studies have shown that the inside of a vehicle can reach 125 degrees Fahrenheit in minutes, even with outside temperatures as low as 60 degrees.
In light of this, Toyota Connected, a software and innovation center, recently introduced its ‘Cabin Awareness’ concept technology. In practice, the technology uses millimeter wave high-resolution 4D imaging radar to detect vehicle occupants, including pets, who have been left in the vehicle and potentially at risk.
The concept technology has the unique ability to detect micro-movements such as movement, breathing and heart rate of the occupants. The platform can also classify vehicle occupants based on size, position and posture.
We spoke to ‘Cabin Awareness’ managing engineer Simon Roberts to learn more about the concept.
Just Auto (YES): Can you tell me about your position and what it entails?
Simon Roberts (SR): I am the managing engineer of the cabin awareness team. This team was formed from a hackathon at Toyota a few years ago and it has grown a bit organically. The title of the challenge was: Safety security and social good.
My team – we won the hackathon and the result was that Toyota said let’s see if this idea has anything and let’s see if we can expand this idea in 36 hours. There is only so much you can accomplish in that time.
So we got a little flexibility from the leadership team who said to spend a few extra hours here and there, and a few of us started working on that. Over the next few months, it started to become real that this idea had potential.
Then fast forward to today where we really started to solidify the concept and the concept is what we demonstrated in our promotional video. So now I also lead a team of about half a dozen engineers, designers and electrical engineers on the Toyota side.
Every day is different, and that’s one of the things I love about Toyota Connected in general. We are encouraged to find solutions to problems and that leads you down all kinds of paths; it’s a fun role.
Can you explain the technology concept and how it works?
It doesn’t have to be exceptionally hot for this to be a problem. I’m in Texas where it gets really hot – over 100 degrees Fahrenheit on some days. But it doesn’t have to be that kind of temperature, it can be as low as 60 degrees, so one is only sitting in the sun for a few hours.
I’m sure you got into a car on a sunny day and thought ‘oh my god, that’s so hot’. For children, they do not regulate their body temperature as well as adults. It’s the same for pets; dogs pant because they can’t sweat. So this can be a real problem if they are in a hot car.
We evaluated all kinds of technologies. There are other systems on the market today, such as weight sensors in the seats. I’m sure you’ve had the experience of throwing a backpack or something in the back seat and then getting the thing thing thing because they can’t tell the difference between your backpack and a kid.
We have door logic today, back door logic, which is kind of a good proxy, a check, for something that because you open that door there might be someone behind it, but it doesn’t cover everything. We wanted to expand that.
We also looked at cameras. From a Toyota perspective, we’ve looked at that, but first and foremost, there’s a privacy issue. Personally, I don’t really want a camera pointed at me and my family in the car, and I know the Toyota isn’t going to do anything with that, but it’s also a psychological thing.
Plus, there’s the computational cost, it’s intensive, you can’t necessarily put a camera in the rearview mirror, and someone can have a rear-facing child seat, which means the camera can’t see the baby. Same problem if you cover the baby with a cloth because it’s sunny outside and they’re sleeping – it’s just problem after problem.
Then there is the ultrasonic technology. There are vehicles with ultrasonic in them that can’t distinguish between people and objects and that actually led us to this millimeter wave radar. I was challenged by this particular manager to find as many use cases for this sensor as possible, because this is a big problem for the industry, and you basically want to try and find as much use as possible. With this millimeter wave radar we can really do that.
Real world testing is currently underway, can you discuss what that entails?
We have partnered with a robotic axi or autonomous mobility as a service (MaaS) company called May Mobility and we have several vehicles from them. Sienna’s minivans are equipped with the Cabin Awareness concept system. The technology gives the driver of an autonomous vehicle an interface on an iPad app we created.
We have our sensor in the vehicle and it is essentially connected to an iPad app. For each passenger seat in the vehicle in the second and third rows for those vehicles, we have an option for each seat that says what is the status? Is it uninhabited or inhabited by a child or an adult? Is there something in the footwell or someone in the footwell in front of you? Are they wearing the seat belt?
There are pictures for them to answer all those questions, and we’ve given them an interface for the driver so they can say, is this right? Yes or no? Then we log that data for every trip they take, and we get that feedback.
So if they do 1,000 drives in a month and 900 of them are good, then we have 90% accuracy. But if they’re not good let’s say behind the driver’s seat, every time you put a baby in that seat it always comes out as an adult, they’d say no, that’s wrong. Then they would put in what that is, that it’s actually an adult and that’s the nature of the information collected.
The technology does not collect any personal information or anything like that. It’s also super fast.
We record the radar data and it is uploaded to our system at the end of the day. Then our engineers, working with the VR engineers, can analyze those results and say, okay, we see a trend in that seat, we always classify a child as an adult. With that data, we can train the neural network to fix that.
What ideas are there about how the system would give a warning that someone was in trouble in the vehicle?
We have the Toyota app and of course also an app with the Lexus brand. The vision with the app is in the hot car scenario. Let’s say I’m driving, and I have my kid in the back. The car will know that; it will know that there is only me and my child in the car, and where we are in the car.
When you turn off the ignition, you get that little message on the dashboard screen that says check the rear seats. The idea is, let’s change that message, let’s be contextual about that message. If we know that no one has ever been in the backseat, and you haven’t even thrown your backpack in there, it would be great not to have that message. So we can be more contextual about that, maybe pop something up on the infotainment screen and say, don’t forget your kid is back there.
Let’s say you get out of the vehicle; we want to give you some time to do things like refuel your car. If you still haven’t gotten the kid out after that time and you may have locked the door and it’s obvious you’re not with the vehicle, we’d like to flash the lights, beep the horn and try and get your attention.
They are just ideas, because at that time there are many variables at work. What is the temperature of the vehicle? What must we do? Do we have to warn them in five minutes, ten minutes? We try to figure those things out.
After that, if you have the Toyota app or the Lexus app, and if you’ve configured notifications, you may be able to get a push notification saying there’s another child in the car. If you’re in the middle of a busy meeting or somewhere you can’t check your phone, we may also have an escalation path. Suppose you have added the emergency contacts to your profile, they can get a notification that something is wrong.
If you have a smart home, smart home lights, Google Home, some smart home devices, we can communicate with that too if you’ve paired them up and we can throw messages on that and wake up your whole smart home, your television, that kind of things. We might be able to really get your attention if you still haven’t done it after a while and things might get critical.
We have the Toyota Safety Connect system on all vehicles; they’re equipped with connected vehicle technology, so we can use that and that’s a call center, at least in the US and North America. There are call centers that, if you have an accident today, will activate the SOS system and connect you to a Toyota agent who can send emergency services for you. So the thought is that we might be able to get in touch with them as well. This is the escalation path we envision.
Is there anything else you would like to share with our readers?
I’ve been working on this for two and a half years; it’s a daily driver for me, this is near and dear. You read the stories and it’s heartbreaking.
I was talking to my doctor’s office yesterday and I was chatting with the technician. He just had a baby and said in response to leaving a child in the car, “I would never do that.” That’s a common attitude. I don’t blame him, because that’s a common reaction: If you’re a responsible adult, why leave your baby behind?
However, it is more complex than that. Data from the experts in that industry says it happens because of a change in routine, for example; you just gave birth and normally mommy takes the baby to nursery but this morning she has something else going on. Suddenly Dad has to do it and it’s out of his routine. Maybe the baby was awake every two hours during the night, he didn’t sleep much; it’s all these compound things. That’s a true story.
Toyota is trying to do its part and hopefully lead and guide the industry in that. I want to share it 100%. I think it’s also critical that everyone in this industry works together and shares that story of how this can happen. Even if it looks like you’re dropping it with rain, it can still happen.
I think sharing that with the readers isn’t to shame anyone, it’s more about raising awareness.