I think the right environment would help, low level lighting, perhaps some sound effects and music to set the scene.
One could do a jump scare as a sound effect, eg unexpected scream.  Those sound board apps for RPGs seem to be quite good these days.

Can a game be scary?

Have I genuinely been scared playing a game?

The answer to these questions is no, but @robsaunders you ask an interesting question which has got me thinking.

You mention the films, books etc you like and some games that gives you tension while playing. You have to ask yourself what makes the films / books scary for you, after all it’s only fiction.

This is is where I think it’s not been done in gaming as set of mechanics for a game, as our brains know that it’s only fiction and we are only playing a game and I think it’s that, that stops us from having that scary feeling.

Now, as others have pointed out RPG games are your best bet by having an excellent DM who creates the tension with low lights, ambient music / sound effects and if you can have a projector project images around the room. These ideas could work for board games and miniatures games but I really think not though.

So I think the bottom line in this is definitely no, based on game mechanics alone, however visual aid with sound effects will give a better experience. But again it’s all about tricking our minds.

My own personal experience of this is no, they can’t.  I don’t personally find books or Boardgames to be scary because of the level of detachment between the player/reader and the medium.  I think TV, movies and videogames have a different (better?) level of immersion that enables fear to be generated that RPGs and Boardgames simply can’t achieve.  I do think that they can create tension but not actual fear.   For example the Space Hulk mechanic is very good at creating tension with its “roll 6 or die” mechanic.  You know, as the Space Marine player that if a genestealer gets in combat with you you’re toast.  At best the sergeants have a sort of 50/50 chance in melee but everyone else is likely to be cut to shreds. Your only hope is that you can roll a 6 on a D6 before the gribbly reaches your square; the closer your adversary gets the more tense it feels before you roll the dice.  However, as tense as it can be, it’s not fear.  You don’t feel fear for the life of the Marine,  or that you will witness some kind of grizly, violent death – you just take the model off the board.

Most games workshop games are scary… I mean have you seen the prices..!!!!   my eyes bleed..

Creepy like  mansions of madness i would say….

I’d argue that an RPG certainly can, it just entirely depends on context.

A video game or movie is more thrilling or exhilarating, no doubt. There’s live feedback and response, there is sound design, jumpy visuals, etc. It doesn’t require much engagement from you, because you are kind of ‘trapped’ by the medium itself. You’re staring at a screen, wearing headphones.
You don’t have to give the game or movie permission to scare you, because you can’t help it. If it blasts some loud scream through your headset, it’s going to startle you.

A role-playing game is more collaborative. You have to go into it willing to suspend your own disbelief. If you aren’t willing to co-operate, there’s no way it can be scary. No more than a movie can be scary if you don’t look at the screen.
Things suggested here like music, low lights, and the sort, are all aids to immersion, because they help remove real world distractions and let you put more focus into the game itself, which in turn makes it easier for you to ‘turn your mind off’.

Horror entirely depends on this engagement. This ‘contract’ between viewer and source. RPG’s by nature tend to be more loose in this regard, because you are sitting around a table, eating snacks, with your best friends, etc. The most emotional scene can be interrupted in an instant by someone getting up to grab a drink or making an off-colour joke. If you view other products in the same environment, they’re equally unintimidating.

I produce, distribute, and license genre (mainly horror) movies for a living. I promise you, that of the hundreds I watch every year, not a single one of them is even remotely scary when we’re sitting around in the office playing it on the TV. At BEST you’ll get the collective “Ooooooh” when something gross happens, but that’s it. Sitting around the office table, eating lunch and riffing about the movie is the same kind of environment most people are in when playing RPG’s.

That being said, I’d say my RPGs are about 75% ‘horror’ themed. Have I ever genuinely scared my players? No. Of course not. None of them actually feared for their lives or anything, just like nobody in a theatre is actually really ‘scared’.
Have we gotten goosebumps, or the chills, or jumped at noises? Hell yea. I’ve played World of Darkness around the campfire in the woods, far from civilization, and I promise you that once players really get hooked, and allow themselves to feed off of the environment and the other players, it can be close to what you’d call “scary”.

Is that partly because being out in the woods at night is eerie? Of course. So is being in a dark room. So is facing a bright screen in a dark room, wearing noise-cancelling headphones and wondering if something is moving around behind you right that very second.
Having played in the woods, in old trailers, in abandoned apartments, and in other strange places, I can tell you that playing RPGs there is certainly a lot scarier than hanging out there and talking, so in my books I’d say that counts.

Your mileage may vary, as with everything, and of course some people will genuinely lack the ability to engage with their imagination in a way that would allow them to feel real emotions in something like a game, but they can definitely be just as scary as any other form of entertainment in certain conditions.