Menu
Teamfoto

Augmented Reality for Food & Beer Orders in Stadiums

Imagine not having to get up during a football game but rather being able to get a cool beer delivered right to your seat. At a hackathon hosted by Switzerland's top football club, we aimed to turn this dream into reality using Augmented Reality technology.

Each year, the YB Bern football club hosts the YB Hackathon. Located in their VIP lounge with an impressive view of the stadium, teams are given 24 hours to design and build innovative software products which they then pitch to a jury.

In the 2019 edition of the hackathon, Jonathan Külz, his father Joscha Greuel, my brother Jan Carius and I took on the challenge set by ELCA: Using augmented reality to enhance the spectators’ experience in a stadium during the club’s football games.

After brainstorming different ideas and rating them by factors such as impact on the stadium experience, ease of adoption for the stadium operators, involvement of interesting technologies we wanted to learn more about, and the possibility to create a minimum viable product within 24 hours, we decided to develop a mobile app enabling the delivery of food and drink to stadium visitors.

To realize our vision, we developed a Kotlin app for Android devices that utilizes the ARCore framework and Google Cloud Anchors to create collaborative Augmented Reality experiences. The concept of the application we developed is the following: To order a beer or a hotdog, stadium visitors can open the app and quickly perform a 3D scan of their environment by moving their smartphone camera around. In the background, a SLAM algorithm constructs a feature map of the environment. Once the detected features are sufficient, a menu of possible orders is displayed to the user. Upon selection of an item, the feature map and the order are uploaded to the cloud.

With the order placed, servers (or other stadium visitors, if operators decide to go with a community-based approach) can intuitively see pending orders. As soon as they open the app and scan their environment in a similar fashion as the stadium visitors did earlier, the feature map of their smartphone is compared to uploaded orders. When a match is found, the corresponding order is visualized by a 3D model of a beer or a hotdog “floating” above the person who placed the order, allowing the servers to deliver the items.

This concept allows stadium operators to provide the service of in-game food and drink deliveries without the need to install order systems or require visitors to manually describe their location within the stadium. Visitors can order from anywhere in the stadium and servers can walk around with a selection of drinks and food, delivering them to guests on the fly. The approach is applicable to a wide array of scenarios, a few examples being trade shows, concerts, or fairs.

In just 24 hours, we managed to develop a fully functioning prototype to showcase both our idea for augmenting visitors’ stadium experiences and its technical realization. We made sure to leave enough time to prepare an engaging pitch with a live demonstration of our system and even got to sleep for a few hours. To top off the awesome two days we had, we were rewarded with the first price by the jury and went home with a big cheque and a trophy:

See also: Netzwoche.ch Sebastian Helbig on LinkedIn YB Hackathon Tweet

Lars Carius

Tech entrepreneur with a passion for computer vision, augmented reality, machine learning, and simulation engineering with a Master's degree in "Robotics, Cognition, Intelligence" from the Technical University of Munich.