On your VT320 Terminal, what is the output from:
The default terminal is the VT100, some VT320 features may not work unless the system knows you are using a VT320.
I have a VT320 terminal which I also connect to my Pi's using a Serial to USB converter. I found it was very usefull to be able to plug into the Pi's USB port and get access to a command line during the early days of Pi WiFi which was not very reliable, my Pi's run headless.
It took a bit of setting up to get it to work, and even more setting up to get the numeric keypad to work in Vim, but that's another story.