As I understand you take in some stream video akin to /dev/video0 and want to enlarge it so you can look at your monitor while playing? Whenever it is something with video ffmpeg can probably solve it. FFmpeg flags like -vf
also work on the video player, ffplay.
https://trac.ffmpeg.org/wiki/Scaling
At the bottom they talk about choosing scaling algorithms, try some out to see what combination of quality and speed you get.
It kinda sounds like some kind of driver issue to me. The fact that it doesn’t detect modes other than a single super basic one sound like not having a proper graphics driver. However, I have no clue why it would work again on reboot and maybe even more importantly, why the other monitor does have a proper mode detected. I wouldn’t expect that if the driver was messed (you would expect all monitors to be assigned some basic mode).