Simulation of Advanced Driver Assistance Systems

McLaren is well known for their heritage in motorsport, especially for their success in Formula One. The impact created by the cutting-edge technologies developed for F1 reaches far beyond the world of motorsport, though. One example for McLaren’s successfull approach of applying insights from motorsport to the development of street cars is their vehicle dynamics simulator that I had the pleasure to work on during my 6-month internship in the UK:

© McLaren Applied Technologies, image taken from here

With the help of the simulator, car manufacturers can drastically cut the development times of new models by replacing expensive prototype iterations and pre-production experiments with digital test-runs. To demonstrate that the simulator cannot only be used for developing and optimizing vehicle dynamics but also for accelerating the development of driver assistance systems, I was tasked with building proof-of-concept applications for lane keeping and pedestrian emergency braking in the simulator.

A lane keeping system can be simplified into two parts: lane detection and an automatic control system. To detect the current and neighbouring lanes, many sensors can be used to gather input data. I decided to go with mono camera input, so a virtual front-facing camera was attached to the simulated car. I developed a Python-based pipeline that analyzes the incoming images using the computer vision framework OpenCV and computes left and right splines marking the lane’s edges. When overlayed onto the live camera stream, the detected lane looks like this:

Aside from providing stable and precise lane estimations at different vehicle speeds, code efficiency was one one of the main challenges. Obviously, the algorithm needs to run in real-time to attach an automatic control system to the back of it, and computing power is expensive. With various optimization iterations, I managed to get the algorithm running at 30FPS on a standard laptop processor without GPU acceleration. In a production environment, a system like this would never use Python but rather a faster and more efficient language like C++, but this was only a proof of concept.

With the lane detection up and running, I developed an automatic control algorithm that takes the current lane detection as well as car speed and steering angle into account and calculates a steering wheel momentum to maintain the planned trajectory derived from the lane estimation. The system outputs a momentum rather than a fixed steering angle so a physical driver can overrule the assistance system or use it during normal driving to prevent accidentally leaving the lane. Hooked up to the simulator, I was able push a button on the dashboard, let go of the steering wheel and had myself driven around by the algorithm I developed:

If you’re wondering why I’m wearing helmet in this picture, let me say this: My algorithm was good, but not perfect, and while a crash in the simulator obviously is a lot less painful than a real crash, it’s till far from comfortable.

The second system I developed targets urban driving environments: Detecting pedestrians in the car’s driving corridor and initiating an emergency braking maneuver. To accomplish this, I utilized a neural network architecture for real-time object detection called YOLOv2 (you only look once). I hooked up the trained network with the car’s camera stream to obtain bounding boxes of humans within the current frame. Using geometrical projections and estimations of the car’s current trajectory, I developed an automatic control algorithm for calculating the amount of brake pressure to apply. When braking for pedestrians worked sufficiently well, I proceeded to add functionality to further analyse objects within their detected bounding boxes within one forward pass of the neural network using OpenCV, which allowed for systems like a red traffic light detector that can also initiate automatic braking if the driver reacts too late.

All that was left to do now was getting into the simulator and trying to run over virtual pedestrians, which made for some good laughs. Both driver assistance systems worked well in the real-time simulator environment and demonstrated that McLaren’s vehicle dynamics simulator is a powerful tool for gaining a competitve advantage in the rapidly emerging field of driver assistance system development.