You wouldn’t be able to install Windows there.
Even if you somehow manage, it won’t be a thing which manufacturer meant it to be anymore.
It most surely will mean losing compatibility with game controller, other different connections like simple connecting it to TV.
Windows require its own drivers, and manufacturer simply doesn’t create it, you would be needed to create them yourself.
In short: you’ll break a device.
Answering your direct question, as far as know, Steam Machine is built with Linux and Windows emulator (vine if I remember correctly), so you’ll be able to play Windows games on the device.
Presumably, if you’re able to launch and play and use, than theoretically you’ll be able to install mods for any games.
But it’s only my personal theory, there might be other things which would make hard or impossible to use mods on Steam Machine.
My offer - get it and try yourself. If it won’t work - refund it