A Literal Aimbot – A Thought Experiment
No, no, I’m not talking about actual cheats. Users of those are the scum of video game communities anywhere and everywhere. This article is about computer vision and its implementation. Consider it a thought experiment.
So, you guys know how VAC or any other anti-cheats detects hacks and cheats, and bans whoever decides to use a bot to shoot for them in a game? I found a loophole that should, in theory, allow me to use a bot to aim and shoot for me, while bypassing anti-cheats completely*.
*depending on how the anti-cheat functions and how you implement it.
That method to bypass VAC is rather simple. If you use a camera to look at the screen, send the image feed to, say, a Raspberry Pi, and have that Raspberry Pi output signals that simulates a mouse input back to the computer so that it can aim and shoot for you, in theory that should not trigger any sort of anti-cheat that checks for discrepancies in your game files or even those that check all other files on your computer, since your “aimbot” is essentially a actual robot that is completely independent from your computer and doesn’t tap into the server connection directly.
The reason why I propose using the RPi (or any other microcontroller you choose to use, but I like using RPi for computer vision purposes) to output simulated mouse input signals rather than have it control servos to move an actual, physical mouse is because of speed. From getting the visual feed, processing the data, transmitting the commands to the servo to the servo actually moving the mouse, there are simply too many things that slows things down. Sure, it’ll be an extremely impressive display, but I don’t think you’ll hit anything at all unless by pure luck, or everyone in the enemy team went AFK right in front of you for a good 10 seconds.
As for how the code works, it’ll be as follows. I’ll be using TF2 as the game for the bot to operate in, and you’ll see why later.
Step 1: Identify the people
First, take a look at this screenshot. There are two Snipers in the screen.
There are algorithms that can be trained to identify human silhouettes. In fact, there are even pre-trained algorithms available in OpenCV, a popular open-source computer vision package, for this purpose. The silhouettes of people in TF2 may be extremely exaggerated, but they still have the same basic structure as actual real-life humans, with one head, one torso, two legs, and two arms, as well as moving in a bipedal manner. Thus, they should work still with pre-trained models available for detecting regular humans. If not, it is still possible to train fresh models to recognise the mercs.
Step 2: Identify the enemies
To know if the identified mercs are friend or foe, the bot must first know what team it’s on. This can be rather easily resolved by using the bottom left corner of the screen.
The area bound by the purple square at the bottom left corner will always be in the bot’s team colour. The bot can just check there to see what team it’s on.
After that, it can check the detected figures from Step 1 to see if it is RED or BLU, then determine if the person is friend or foe based on team colours. That is why I use TF2 for this exercise, since the team colours simplify things a lot.
Step 3: Identifying where to shoot
Once Step 2 is done, the bot needs to know where to shoot. Assuming we are using a hitscan class, we can pick where to shoot based on the class. For classes capable of headshotting people, such as Sniper or Ambassador Spies, the bot will aim for the head. For other hitscan classes, the bot can aim for the torso instead. Considering that we identified the person in Step 1, we should be able to identify the location of the head and the torso.
After that is done, just have the RPi move the crosshair to the centre of the nearest target. Once the crosshair is in position, just press M1.
Granted, I am grossly oversimplifying the implementation of this bot. Aside from that, based on my own experiences the pre-trained models are not extremely accurate, and training them is something that requires time, expertise and experience, which I honestly don’t have at the moment of writing. Still, at the very least it will be a rather interesting programming project.
By the way, there’s another reason why this article is titled “A Literal Aimbot”, because all it does is aim (and shoot). I cannot for the life of me figure out how to get the bot to know where it needs to go and how to get there. Without direct access to the server info and the game itself I don’t think that’s possible. Dammit.
So in theory, you could create something like this, but the getting information from the game would probably still get you VAC-banned.
By the way, just to emphasise this again, please do NOT actually use or make cheats. The whole point of this article is to discuss the implementation of computer vision through gaming. Seriously, don’t cheat.