Process Automation through Augmented Reality

It all started in 2019 with our team preparing for the YB Hackathon. All we knew at the time was that the challenge involved Augmented Reality, so we played around with Google’s ARCore framework and quickly prototyped an app that allowed users to place chickens on flat surfaces (the chicken was the first 3D model with the correct format that found online):

We ended up winning the hackathon with our Augmented Reality system for food & beer orders in stadiums and realized the potential the technology has for industrial processes. After lots of interviews with experts and potential clients, we decided to tackle the challenge head-on and founded the company Chicken Technologies, its name being a homage to our first encounter with Augmented Reality:

So what did we actually do? After talking to representatives of countless different industries (I’m talking multiple hundred interviews here), we singled out the construction and facility management sector as the field that can profit most from the introduction of AR into operative processes for two main reasons: Firstly, the environments in construction and facility management settings are often unstructured, partly unknown, and fast-changing. This prohibits companies from applying hardware-based beacon solutions or one-off 3D scans to communicate location-based information. Secondly, many stakeholders are involved in the construction and management of a building, and often communication barriers lead to costly misunderstandings. On top of this, communication often has to happen asynchronously: An employee of one trade might have to leave a message to a member of another trade and company but might not even know who this person is and at what time they will resume the work.

One of our early pitchdecks contained this illustration:

As the diagram shows, our approach was using collaborative Augmented Reality as a central tool for communicating location-based information: A construction material delivery driver would drop the shipment at an arbitrary position on the lot, perform a quick environment scan with their smartphone and click on the shipment’s position to create a visual marker. The local 3D scan is then uploaded to our cloud backend, so any construction worker or manager can now select the shipment and get intuitive visual guidance to its location on the premises.

This system solves two problems: Firstly, it takes out any human errors due to miscommunication or inability to communicate at all. Secondly, it alleviates the need for stringent storage and warehousing systems on construction sites which are often difficult and costly to realize in inherently fast-changing local circumstances.

This is a screenshot from an early version of our construction site manager app showing an AR-tagged delivery:

In our first pilot project, we quickly realized that our approach has a major flaw: As the environment scans only cover a local area, to display the AR marker, one already needs to be pretty close to the object in question. However, Augmented Reality also provides a solution for this problem: In addition to placing a marker at the delivery’s location, workers can also add some navigation hints to guide others to the dropoff location:

When the user is walking with their smartphone pointed forward, ARCore is quick enough to perform the environment scans in real-time without additional device movements required, so the user only has to tap the floor every few seconds to add an arrow, no additional time is taken.

With this solution, we also covered another use case: If any information is necessary for the retrieving party along the route, like an access code to a locked gate, it can be conveniently placed and retrieved by attaching it to the AR markers. Once additional information is available, it will automatically pop up and be displayed to the person following the route.

The navigation system we built works in any environment, no previous 3D scans or any other knowledge about the respective location needs to be available. This makes our system widely applicable, even in environments that are hard and expensive to solve with classic GPS or beacon-based navigation systems. The natural step is therefore to deploy it to indoor scenarios as well, as anyone with a smartphone can now offer navigation features to visitors of their indoors space in a matter of minutes:

This feature led us to our next use case: facility management. In facility management, usually, one party reports a fault or problem and then waits for another party to fix it. The communication is mostly asynchronous, meaning reporting and fixing parties often won’t be in the respective location at the same time, and in large buildings, it can be hard to describe where exactly the fault is located. With our tagging and navigation feature, both problems can be overcome: The reporting user simply places an AR tag with some additional textual description at the fault location and, if required, adds some navigation markers to guide the facility management personnel to the location in question. The facility manager now simply follows the route whenever they are on premise and will find all required information without losing valuable time by communicating back and forth with the reporting party to find out what to do:

The last use case we explored in the construction sector is the management of work instructions in handicraft businesses. Managers could walk through the construction sites and leave instructions for their workers, which can later scan the rooms, find out what to do, and perform the required tasks, again sparing both parties countless loops of communication:

Aside from the construction and facility management market, we realized a number of solutions for customers of other markets, reinforcing our opinion that Augmented Reality can drastically boost performance in many business settings.

Thanks for reading, I hope you enjoyed seeing some of the work we did at Chicken Technologies to improve industrial processes using collaborative Augmented Reality! If you or your company are thinking about using Augmented Reality as well, drop me a message and let’s discuss opportunities for you to profit from this fascinating technology!