It says to edit the config and cmdline files on the SD card.
Well, that's very informative...
What happens when you press <alt+F4>?
That should close the dialog.
You're apparently using the Pi without a screen attached. If you want to get a higher resolution you have to change the window size.
You can do that using the Pi configuration utility (from the GUI menu) or 'sudo raspi-config' from the command line.
Alternatively you can also set also set the framebuffer width and height in config.txt.