Making an intelligent Robot NPC using Hugging Face 🤗 and Unity

Making Games with AI Course, Chapter 1

This tutorial is part of the Making Games with AI Course. A free upcoming course on creating AI-powered games for Unity and Unreal.

---

The integration of cutting-edge AI models in video games is opening a whole range of new exciting gameplays.

One of them is the ability for Non-Player Characters (NPCs) to understand and respond to the player’s voice or text inputs. And this is what we’re going to do today.

In this tutorial, you’re going to create this smart robot that can understand your orders and perform them.

You can play with my version on your browser here 👉https://huggingface.co/spaces/ThomasSimonini/SmartRobot

To be able to do it, we’re going to use an AI model that understands any text input and finds the closer (if any) action on its list. It’s called a Sentence Similarity model.

What’s interesting with that system, contrary to classical game development, is that you don’t need to hard-code every interaction. Instead, you use this Sentence Similarity model that selects what’s robot possible action is the most appropriate given user input.

To make this project, we’re going to use:

• Unity Game Engine (2022.3.13 and +).

• The Jammo Robot asset is made by Mix and Jam.

• Hugging Face Unity API 🤗, an easy-to-use integration for the Hugging Face Inference API, allows developers to access and use Hugging Face AI models within their Unity projects. Don’t worry if you never used it before we’ll explain how to use it step by step.

The requirements:

• You need to know to use Unity. If it’s not the case, you can check this excellent free course 👉 Create with code

At the end of this tutorial, you’ll build your own version of this smart robot, add more actions and share it with others online for free.

Then, you’ll be able to create other games using this AI. Imagine a stealth game, for example, where you guide your character to sneak around a museum and pull off daring heists?

(One of the prototype I’m working on using Sentence Similarity)

You can find the complete Jammo the Robot Unity Project 👉 here

So let’s get started!

The power of Sentence Similarity 🤖

Before diving into the implementation, we need to understand how the game works and what sentence similarity is.

How does the game work?

With this game, we want to give more liberty to the player. Instead of giving an order to a robot by just clicking a button, we want them to interact with it through text.

The robot has a list of actions and uses a sentence similarity model that selects the closest action (if any) given the player’s order.

For instance, if I write, “Hey grab me the red box”, the robot wasn’t programmed to know what’s “Hey grab me the red box”, is. But the sentence similarity model made the connection between this order and the “bring me red cube” action.

Therefore, thanks to this technique, we can build believable character AI without having the tedious process of mapping by hand every possible player input interaction to robot response. By letting the sentence similarity model do the job.

What is Sentence Similarity?

Sentence Similarity are language models able, given a source sentence and sentences, to calculate how much similar sentences are to the source sentence.

For instance, if our source sentence is “Hey there!” it’s very close to the “Hello!” sentence.

You can try for yourself here in your browser 👉 here

By using a Sentence Similarity model, we’re able to let our robot “decide” given input text what’s the most appropriate action to take. The models are already trained so we can use it directly.

And to find a Sentence Similarity model, we’re going to use Hugging Face.

---

What is Hugging Face 🤗?

Hugging Face is a platform where users can upload or download pre-trained AI models in more than 35 different tasks from text generation with Llama v2, speech-to-text with Whisper, text-to-speech (AI voices) with Coqui to 3D generation.

For most of these models, Hugging Face provides a free API called Inference API that allows you use them via simple API calls.

So how I can select a Sentence Similarity Model on Hugging Face 🤗?

All the AI models are listed here https://huggingface.co/models. On the left of this page, you have the differents tasks.

In our case you want to select Sentence Similarity.

Now we have a list of models that are Sentence Similarity models (approximately 3,000). We’ll see just after how to test them.

Now that we understand what Sentence Similarity and Hugging Face is, let’s build our smart robot 🤖!

---

Let’s build our smart robot 🤖

Step 1: Getting Started with Hugging Face 🤗

Hugging Face contains a lot of amazing AI models and an API (Accelerated Inference API) to directly plug them into your projects.

But first, you need to create a free account.

When your account is created, go to Settings by clicking on your profile picture (top right-hand corner).

In Settings, select Access Tokens

Create a new Access Token, give it the name you want, and select write Role:

Now that the token is created, copy the API Token, it’s the key you’ll need to be able to use the API.

⚠️ For security reasons, DO NOT SHARE THIS KEY to others, it’s a private key.

Step 2: Select your sentence similarity model 🤖

Now that we have the API key, we can try some models to choose one.

When you click on of the models, you can try the model directly on the website. Let’s make a test, go to the first 5 models and test them.

Here I tested this model: all-MiniLM-L6-v2

We see that “fetch the red block” is close to the robot action “bring me red cube”, so the model is working correctly.

Now that we choose this one, we need to get the API URL, to do that click on Deploy > Inference API.

This opens a window where you can copy the API_URL:

You’re now ready to use this model in Unity.

Step 3: Install the Unity API Plugin

You can find the complete Jammo the Robot Unity Project 👉 here

1. Open the Unity Project

2. Go to Window -> Package Manager

3. Click + and select Add Package from git URL

4. Enter https://github.com/huggingface/unity-api.git

5. Once installed, the Unity API wizard should pop up. If not, go to Window -> Hugging Face API Wizard

6. Enter your API key. Your API key can be created in your Hugging Face account settings.

7. Test the API key by clicking Test API key in the API Wizard.

8. Paste your model API_URL on the Sentence Similarity Task

9. Check ✅ wait for model

Step 4: Build the Robot Behavior 🤖

Now that we’ve connected our Unity Project to the Hugging Face API, we need to define the behavior of our robot.

The idea is that our robot has different possible actions and the choice of the actions will depend on the API output.

We need first to define the Finite State Machine, a simple AI where each State defines a certain behavior.

Then, we’ll make the utility function that will select the State hence the series of actions to perform.

The State Machine 🧠🤖

In a state machine, each state represents a behavior, for instance, moving to a column, saying hello, etc. Based on the state the agent is it will perform a series of actions.

In our case, we have 7 states:

The first thing we need to do is create an enum called State that contains each of the possible States:

Because we need to constantly check the state, we define the state machine into the Update() method using a switch system where each case is a state.

For each state case, we define the behavior of our agents, for instance in our state Hello, the robot must move towards the player, face him correctly and then launch its Hello animation, then go back to an Idle State.

We have now defined the behavior for each different State. The magic here will come from the fact that’s the language model that will define what State is the closest to the Player input. And in the utility function, we call this state.

Let’s define the Utility Function 📈

Our action list looks like this:

• The sentence is what will be fed to the API

• The verb is the State

• Noun (if any) is the object to interact with (Pillar, Cube, etc)

This utility function will select the Verb and Noun associated with the sentence having the highest similarity score with the player input text.

But first, to get rid of a lot of strange input text, we need to have a similarity score threshold. I put for my model 0.20, but don’t hesitate to test with different thresold to see what’s the best.

For example, if I say “Look at all the rabbits”, none of our possible actions are relevant. Hence instead of choosing the action with the highest score, we’ll call the State Puzzled which will animate the robot with a perplexed animation.

If the score is higher, then we’ll get the verb corresponding to a State and the noun (goalObject) if any.

We set the state corresponding to the verb. That will activate the behavior corresponding to it.

And that’s it, now we’re ready to interact with our robot!

Step 5: Let’s interact with our Robot 🤖

In this step, you just need to click on the play button in the editor. And you can prompt some orders and see the results.

You can also try online 👉 https://huggingface.co/spaces/ThomasSimonini/SmartRobot

It may have some bugs, especially the model loading bug, if you see the robot staying in idle mode, wait 45 seconds because it may be the model loading for the first time. And check the console for any error messages.

Step 6: Add new actions

How can I add more actions?

Let’s take an example:

• Copy YellowPillar game object and move it

• Change the name to GreenPillar

• Create a new material and set it to green

• Place the material on GreenPillar

Now that we’ve placed the new game object, we need to add this possibility into the sentences, click on Jammo_Player.

In the list of actions click on the plus button and fill in this new action item:

• Add Go to the green column

• GoTo

• GreenColumn

And that’s it!

Step 7: Share your demo with the world 🔥

Congratulations on completing your demo! Now, let's make it accessible to everyone to play it by publishing it online.

Hugging Face offers a convenient and free solution called Spaces.

We've prepared a quick 5-minute tutorial to guide you through the process: https://huggingface.co/blog/unity-in-spaces

Follow the steps in the tutorial to showcase your creation and let others experience it 🚀

I published mine on Spaces, you can try 👉here!

---

That’s all for today, with a few steps we’ve just built a robot that’s able to perform actions based on your orders that’s amazing!

What you can do next is dive deeper into the code in Scripts/Final and look at the final scene in Scenes/Final. I commented on every part of it so it should be relatively straightforward.

If you liked my article, please click the ❤️. And don’t hesitate to follow me on substack and Twitter. I write about AI in Games.

This tutorial is part of the Making Games with AI Course. A free upcoming course on creating AI-powered games for Unity and Unreal. Don’t forget to sign up 👉 here

If you want to know more about language models and check out the amazing HuggingFace course on it (it’s free) 👉 https://huggingface.co/course/chapter1

In the next tutorial, you’ll build a game demo call "Say Something 🗣️," where an AI asks you to write something nice, mean, or neutral.

🤖 The AI understands you and rate what you said, and if ✅ gives some points and additional time ⏱️.

See you next time,

Keep learning, stay awesome 🤗