Dissecting a Sentry Gun

Before we start, I’ll like to say a few things. Firstly, I’m assuming this is TF2 at 2007: No hats, no unlocks. Reason why is that it makes my job easier, and I’ll explain why later in the article. Secondly, I am making a guess on a video game mechanic. Grab your mountain of salt from a Scout getting walled off by a Sentry Gun and read this post with it.

The Sentry Gun. Love it, hate it, we all know what it does. It tracks down enemy team members and shoots them. However, does anyone ever thought of how it actually functions?

I’m going to break down how the Sentry Gun behaves to a few characteristics, and make a guess as to how it would function IRL based on real-life engineering.

1. It tracks down enemy mercenaries

First things first, it is highly convenient that both teams have such a consistent colour scheme, RED is red and BLU is blue. Because not only does this make my job easier, it also gives me a hint as to what the tracking mechanism is.

Since the mercs on both teams are identical to their counterparts except for the colours of their clothes, I’m going to assume that the Sentry Gun’s mechanism is vision-based. And conveniently, we do have real-life technology that picks up and highlights targets from a visual input.

How it works is that a program will receive an image as an input, scan the image for a pattern that resembles a reference image (or a series of references) given to the program, and marks the part that resembles said reference as a match. The sensitivity of the program will need to be calibrated: Too sensitive and the Sentry will fire away at anything that is the same colour as the enemy team, and if it’s not sensitive enough it wouldn’t even shoot a Heavy standing right in front of it because there’s a speck of dust on his right cheek. So now that we briefly explained how this works, let’s see how many template images do we need.

Assuming that using different weapons (primary, secondary, melee) will change a class’s posture significantly, each loadout slot will need a template image. That is 3 each for Scout, Soldier, Pyro, Demoman, Heavy, Medic, Sniper and Spy (while in-game there are 4 slots for Spy, there’s no way a Sentry would’ve picked up a Spy with his Invis Watch active for rather obvious reasons). Engineer has 4 loadout slots (the pose he assumes when using either PDA is similar enough to one another) plus him hauling his toolbox, making it 5 poses. Add another pose for Heavy deploying his Minigun, one for Sniper aiming down his scope and 1 humiliated pose for all, that makes 40 different poses for all nine classes.

But then, the mercs could be running, standing or just moving around in general. So, assuming all the different movements can be simplified to either airborne, upright (standing or walking), crouching or swimming, that makes it 40 x 4 = 160. And since the Sentry Gun also shoots enemy buildings, 4 different types of Sentries, 3 types of Dispensers, and 2 Teleporters (their appearances don’t change much when upgraded, but it looks different depending on whether it’s active or not) bumps the number up to 169 different reference images needed.

Good point, so 8 directions (front, left, back, right and and in between any two adjacent directions) times 3 different elevation levels (same level, higher up or lower than) means that there are 24 different angles from which the Sentry Gun could be looking at the merc, assuming the difference in between any two adjacent viewpoints are small enough to not trigger a false negative. 24 times 169, that gives me … 4056. Dell will need 4056 reference images for the Sentry to recognize all possible targets. And this is the reason why I excluded all the unlocks and cosmetics from this venture: it’ll take too long for me to account for all of them since most of them, if not all, can change the overall appearance of the class by a significant amount.

So in this case, the Sentry Gun will take the input, mark out the target if it’s present based on those reference images, and sends a signal to the motors for it to aim at said target.

2. It tracks the closest target if there are multiple targets

There is a real-life tech that can map out distances of objects too: the Kinect. This is how the Kinect actually functions. To give a brief summary, an infrared projector projects an array of dots in front of it. A sensor some distance away from the projector marks out the position of each dot. Comparing the image obtained and a reference image of how the dots will be positioned if there is nothing in the room will give a 3D map of what the Kinect is looking at. Now, take this tech, take the output and match this to the output from part 1, and we’ll have a system that can find the closest target to the Sentry and aim at it.

3. It will shoot the target when the gun is directly on the target

This part is slightly easier. Previously, we talked about how the Sentry can find the closest target to it. Add a function that triggers the guns when the target is in the center of the image (or a certain offset from the center since the camera is definitely not inside the barrel), and that should be it.

4. It does this in real time

What real time means is that if the Sentry takes a snapshot of its environment every second for processing, between the time of it taking the first picture and when it takes the second picture, it needs to process the image, find out the closest target, and move the motors of the Sentry to put its crosshair on said target, and shoot when the target is on it. In other words, it has one second to do all this. Should be possible… I guess.

5. It can aim and shoot at targets behind cover, as long as parts of them are visible

If a Heavy peeks out from behind cover, exposing only his face, there really isn’t anything to mark him as RED or BLU. In fact, even the Engineer himself standing by his Sentry won’t be able to tell too. Besides, since most of his body is behind cover there is no way that Sentry will even be triggered if it uses the mechanisms I previously described. Hmm, my previous guesses seem to be completely wrong.

6. It can aim and shoot targets behind it, and there ain’t no cameras or anything that could have a camera there

Huh. Definitely not radar, since it can still differentiate between RED and BLU. Maybe…. maybe a live satellite feed of the surroundings + GPS?

7. Mate, it can shoot people inside buildings too. And last I check X-ray satellite imaging doesn’t exist. And X-ray images are not coloured

Bloody hell.

8. Not to mention, it can also shoot at Spies dumb enough to disguise within line of sight too.

…

9. Also, I’m pretty damn sure the Sentry Gun can work in complete darkness as well

… [CENSORED]

You know what? I give up. Maybe I’ll be able to figure it out if someone gives me a few bars of Australium, but until then, I have no clue how in the world does this thing work. And don’t get me started on Torbjorn’s sentry. Somehow it knows who to shoot even when the characters look the bloody same no matter what team they’re on.