Neural Network Data Fusion for Self-Driving Cars

Fusion Image (R, G, B, IR) Network

Use 4D images (R, G, B, IR) to achieve the best performance for systems that include both RGB and IR cameras. The detection score of this DNN is the detection score of the combination of the 2 DNNs mentioned in Section B above, while using only half the computational resources. This combination/fusion will return the best results for a single system.

However, some factors must be considered when choosing a fusion network approach. By using 2 different DNNs (as shown in Section B), the system obtains the redundancy factor required for the safety of autonomous vehicles (preventing failure in case of occlusion, lens dirt, failure of one of the sensors, etc.). Also, the improved RGB + IR network can be tuned a bit more, so removing one of the cameras (RGB or IR) will not affect the DNN of the remaining channels.

A) RGB DNN trains IR DNN. B) RGB DNN and IR DNN are mutually trained. C) 4D Image DNN

Conclusion

RGB cameras will likely remain the primary sensors in self-driving cars. In some cases, RGB cameras become unreliable, requiring the use of additional sensors. When choosing a companion sensor, the following points should be considered:

• Sensor performance under environmental conditions (lighting, weather, etc.) that affect RGB camera performance
• sensor price
• Easy to train AI algorithms and full data fusion capability
• LWIR cameras perfectly satisfy all of the above considerations. The system leverages state-of-the-art artificial intelligence capabilities to quickly and reliably achieve complete data fusion to make fully autonomous vehicles a reality.

RGB visible light image (left), LWIR image (middle), and fused RGB-IR image (right)