How To Get Mousepad To Read Standard Text Files

[scruss]

OP does not want ANY particular protocol or character set. Just an editor that recognizes 7 bit ASCII created at the dawn of printing machines and used by 99% of all desktop computers to this very day.

Good to know. Mousepad, Leafpad and any other Linux editor you could name already do that. The fact that you were getting the character set warning is because you didn't have 7-bit clean ASCII in your files.

UTF-8 is 100% 7-bit ASCII compatible; that was the whole point of it. Ken Thompson and Rob Pike did a good job of designing it back in 1992. I'll bet your files have "smart quotes" in them: 0x93 and 0x94 bytes for quotation marks. Windows WordPad was particularly rancid for putting them in without permission. Maybe run one of these text files through xxd to hexdump the character codes: you might be surprised. Yes, I know you didn't type them if they're there.

If your files have got polluted with CP-1252 characters, there's not much you can do to get it to display on a Raspberry Pi. I don't know of any locales that use 1252. You're going to have to make friends with the iconv command to convert everything to UTF-8. Thankfully Windows knows about it these days and should display it cleanly.

(Back in the late 90s I ran the computational division of a dictionary company. Windows character weirdnesses bit us all the time. It's been a while since I've seen this problem in the wild.)

On the other hand take Windows Wordpad. It handles both nixie and wintel line endings transparently and smartly with no protocol hiccups. With about 1/10th the resources too.

Then again, WordPad is 100% Windows system libraries. All that Rich Text and GUI handling is part of Windows, so of course it doesn't appear bloated.

Actually Windows IS the rest of the computing world.

I dunno; I see an awful lot of Macbooks out there. And Microsoft is getting with the programme too with WSL. We'll eventually see the back of that extra EOL character that mechanical teletypes needed to read and discard while the print head flew back. But maybe not quite in my lifetime.

[Heater]

Meanwhile, the most widely deployed operating system in the world today is most likely Minix:
https://www.cs.vu.nl/~ast/intel/
https://www.youtube.com/watch?v=pq9ksus1jWI

Just sayin'

[emma1997]

could you possibly, may be, zip up one of the text files that is causing your problems and post it here? I'm curious to know what all the editors I have here do with it out of the box.

Here's a file downloaded and trimmed down to one line. Mousepad burps and the offending character is clear. Leafpad accepts the file and don't show the character. Thanks for taking time and I'd be very interested to see what you find.

[emma1997]

The fact that you were getting the character set warning is because you didn't have 7-bit clean ASCII in your files.

Exactly. And BTW the problem files are not actually 'mine'. As mentioned before IIRC I've had no need last few decades to deviate from 7 bit in any files created. Most everything that causes trouble was downloaded from websites like this one using either Firefox or Chromium. Either saved as text or, more commonly, cut and pasted.

I would be much happier if all the tools make believe there are only 128 characters in the set.

BTW no luck changing locale. Maybe not doing it right but turned into another sticky wicket of wastes hours. The message hovering over was not help either because altering source files is not an option.

With the two editors online I can fix the root problem but unfortunately busted my neat 4g partition to 8g and now taking me twice as long to backup and play with images. Better than 8x longer (32g) but still a huge waste of time.

Also should mention that I have no particular love affair with ISO8859-14. It's only the default choice when clicking 'other' and have since found most of the others work too.

PS In a nutshell this is not really a Unix thing. Just poor judgement on the part of Mousepad creators. A setting option to enable or disable the check would be far more reasonable.

[jahboater]

The offending character is 0x96 which is strange:
Take a look at:

https://en.wikipedia.org/wiki/ISO/IEC_8 ... age_layout

You can see why mousepad might complain - its not a printable character in any character set ?

Vi and Nano display it in reverse video as:

(0x96 & 0x1F) | 0x40

the character V
which I don't like either.

[emma1997]

Maybe it's more Mousepad complains and Leafpad just ignores.

Anyway I see the same mass confusion when deviating from teletype and find it annoying. If it was just once in a while but it's not unusual for me to open a hundred or more files in one day. Only open few seconds but REALLY annoying.

I feel like the George Segal movie character where he was a computer scientist who hated being told what to do by machines. Then they put one in his head! lol

[Heater]

emma1997,

Thanks for the example.

Apparently this is not any kind of ASCII file:

Code: Select all

$ file ng2.txt
ng2.txt: Non-ISO extended-ASCII text, with no line terminators

'cat' silently drops the troublesome char, or displays it as blank.

None of my editors display it properly, whatever it is supposed to be.

Interestingly even if I open it as ISO 8859-14 in Microsoft's Visual Studio Code it displays it as just box.

[emma1997]

What I intended to send was a 72 byte file 'compresses' to 103 bytes that contained the following line:

Code: Select all

df -H --output=source,size,used,avail – Display the amount of disk space

The character before " D" is the 96dec troublemaker as noticed by jahboater. Mousepad halts because of it and Leafpad makes believe it ain't there. Cutting and pasting to this site shows up as a hollow rectangle.


I'll download the file here and look at it again then try it on a 'real' (lol) PC and see what happens.

[Heater]

Anyway I see the same mass confusion when deviating from teletype and find it annoying. If it was just once in a while but it's not unusual for me to open a hundred or more files in one day. Only open few seconds but REALLY annoying.

I have a lot of sympathy for that view. Having been initiated into computing via a teletype myself.

But this is 2020 and we live in the web. Despite the admirable efforts of Unicode and the wonderful utf-8 there is corruption of text going on, on a mass scale, all over the net. Especially when one is cutting and pasting stuff from web pages.

Somehow we have to live with it. It's not something that text editors can be expected to fix without some human intervention.

I happen to think Unicode has gone insane. If only because you can do this:








Č̶̡̢̛̱͇̹̳̯̭̞̙̮̝̣̖̥͈̙͓̲̞̲̼͎͔̫̻͓̣͔̹̺̈́̈́̄͊̇̈́̊́̍̈́̀̇͛̂̍̊̔̅́͑́͂̇̓͌̓͋̑̾̚̕͘͝a̵̡̨̧̢̧̨̢̛̗̯̪̠͙͖͍̫͎̯̹͓͓̮̫̼͔̻̥̜͚͈̠̭͖͇͈̠̯̱̬̩͚͙̗̜̳͙͍̮͓̺̣̼͉͇͎̥̥̞̬͍͈̲̙̣̯̳̩̻̼͖̞̭̙̟͇̲̯̝͔̥̪͉̰̣̥̩̙̣̜̔͆̀̈́̂̉̈́̈́̋̏͂͆̇́̊̌̃͊̈́͛̓͑̀̔̑͐͌̂͐̏͂̈́͑̎̎͊͒̔͌̿͌͂̋́̏́̎͗͐͒̍͆̓͆̾̈́̾͐͛̒̃̔͛͋́̏͘͘͜͠͝͝͝n̴̡̨̡̢̧̢̧̳̱̱̩̰̬̜̹̮͇̥̩̲̯͎͕̝̣̺̲̮̤͕̟̠͈̬̙̮͕͚͓͍̯͔̗̭̪̠̗͖̤̳̠͚̗̦̞̪̣̹̭̼̼̺̣̫̻͎̟͎̗̋̒̐̈͌͌̇̐̃̐̇̀̇̅̈́͗͐͌̿̔̏͌̎͂͑̔̚̕͜͜͜͝͠ͅͅͅ ̵̧̢̡̛̛̛͔̦̺͎͖̱̞͙̲̙̭̏̑̉͒̊̑͋̊͌͆̑̿͗͌̎̾̃̽̒̋̇̏̅̔̑̔́̈́͋̈́͑̃̾̊̎̄͑̏̀̀̀̈́̒́̎̈̄́̀̄̉͆̌̃̔̋́̀̀̅̓̽̓̓͑͑͑͐̈̄͌͆̌̋̐̉͐̚̕͘̚͠͝͝͝͝͝ͅy̴̨̧̢̨̧̨̧̨̧̛̛͓̘̳̣̣͈̮͓̮̫̻͖̼̘̭̝͍͙̠͔̩̪̹̦̩͍̺̪̯͎̳̱͔͉̼͓̝͈͇̩͔̯̳̣̳͍̯̫͕̲̙̖̫̥͍͚̗̞̥͔̘̩̜̣̼̱̖͚͚̱̜̜̼̺̎́̀̍̇̾̑̂̈́͋̑̾͌͌͐̈̐͛̈́̑̍̋̆̽̍̾̃͗̆͆̈͊̎͒̒̈́͌̾͊̏̀̔͋̑̄̈́̇̀̉̅̔̅͐̋́̾̏͊̕͘̕͘͜͝͠͝ͅͅͅǫ̵̨̢̢̨̡̡̨̫̱̖̯̗̣̮͔̘̳̘̪͖̞̙̬̜̬̬̟̬͉̱͎̪̻̗̫̝̙͈͓̳͓͉̻̞̻̤̳̜̼̗̥̠̠͚̼̹͎͚̬͔̙̤̠̪͔͇͙̲̖̜̳̥͓̘͔̖̣̥͎̩͙̱̼̝̰̂̎͒̿̑͑͂̓́̈́͗̏̓̀̓̑̄̎͘͘̕͜͜͜͠͝ͅͅͅͅͅư̴̧̨̡̨̧̮͓̝̯̬͕̩̺̲̘̙̣̬̟̺̘͎͚̗̝͍͈͖̣͈̮͍̬̺̟̤͍̗̬̦̤̤̭̘̞̝̗͚͖̭̙͖͈͎̺͙͔͕̰̭̬̪̭̮̩̳̔͊́͑̄̏͋̔̐͋̐͋́̌́̍̽̃̀͂̾̀̀̍͊̅͐̃̈́͒̏́̀͂́̈́́͋̅̋̔͊̓̆͒̏̔̈̄͊͆̊̌̒̊̍̓́̈́͒̎̈̀͌̆͑͌͋̈̍̾̅̕͘͘̚̕̚̕͘͜͜͠͠͠͝͝͝ ̷̨̧̨̡̡̧̛̛̛̮̘̝̝̼̖̫̜͖̤̭̭̥͙̘͓̪̦͓̬͔̹̬̭͔̹̣̟̪͔̞̝̙͔͍͎͈͖̣̗̱̮̟͓̝͔͖̟̻̟͈̗̮͚̣̗͚̠̰̯̳̗̯̯̤̾̏́̉͆̔̍̍̾̓̓̈̅̉͂̒͒́̔͌̅́̽̂́͑̆̑͆̎̊̍̆̆̈́͂̌͒͆͒̔́̿̀̎̾̈̇͒͆́̅͋͑̐͐̏̈́͂̏̓̊͂̋̑̓̿̃͑̔̓̾̽̚͘̚̕̕͘̚͘͜͜͜͠͠͝͝͝ͅr̴̢̧̡̢̨̼͖̼̤̖̹̝̣̯̭͍̜̝̳̘͇̲̲͓͕̫̭͔͇̰͖͉͉̤̜͎̫͕͚͈̗̮̟͚͓̹̩̙̟͉̻̤̼̲̠̩̥̩̠̻̻̘͎͕͕̟̤̻͓̮͇̼̮̟̺̲̭̝͉̓̍̀͆̀̎̇͛̽̔̐̏̍̓͆̀͜͜ͅͅͅë̵̡̢̧̡̧̞̙̘͚͙̩̫̭͈̗̟̝̜̤͙͖̻͙̩͉̻̖͍̪̝͇͉͖͚̺̺̺̣̗̙͔͖̰̥͙̜͔̯̝͎̘̳̯͓̰̤͍̹́̎̀͂͒̀͊̀̈́̊̍̏̏̑̎̌̀̊̈́͂͌͂̿̎̔́̚͘̚͘͜ͅa̴̧̡̡̧̢̧̡̛͇̖̝̫͖͚̯̟̮̘̘̳̜͔̤̹̦̝̮̦̠̩̥̖͈̖̝͇͙̲̭̣̯̘̻̠̱̟̬͇͕̖̬͚̼͎̪̻̙̹̗̝̭̺̹̞̤͉̲̥̣̳͓̯͎̣̲̰̙͖̜̯̞̲͍̯̭͓̩̜͕͍̜̩̜̐̿͐͋̍̊̔̇́̑͜͠ͅḑ̶̧̨̧̡̡̧̢̡̧̮̣̗̟̙͈͈͍̥͇͔̼̱̠̼͇͇̺̻̼̤̳̟̤̫̬̱̘̙̣̲̺̟̺̱̖͉̙̘͚̰̰͕̲̹̘̜͎̗̗͚̠̦̯̪̺̜̭̗̞̜͓̭̫̹̻͙̣̖͇̠̤̜̳̺̌́͊̊̈́̆̄̋͆͊̎̃̋̄͌̉͂̈́̆̿͘̕̕͜͝͝ͅ ̴̨̧̧̢̨̨̛̛̩͓͉̣̥͎͕̥͇̯̼̺͍̱̺̣̤̮̲̥̺̺͓͚͙̪̮͉̘̳̣̦̖̖͍͉͉͕̰͙̙̩̬̲̮̞͖͉̦͖̤̘̩͙̳̬̳͙̼͔̦̰̝͚͉̭̻̮̗̞̤̺̱̗̌͑͒͆̏̋̈̈͌̀͒͊͋̒̇̂̌̃͐̃̃̊͋́͛͑̉̒̈́̏̂̄͋̓́̏̋̀͛͆̓́̈̂͐̄͜͜͠͝ͅͅṯ̷̨̨̢̢̨̣͓̣̣̗̪̳͕͙͍͇̬̖̲̟̱͇̳̟̻̮͎̪͉̯͚̣̣͇̱̪̟͔̹͕͍̙̱̼̤͂̈͒̓́͂̓̀̇̃̅̒̈́͆̐́̄͗̀̄̌̐̂̆̈́͛̂̔̂̈͆̔̑̊̅̓̽̈́̌̾̎̀̇͗̎̓͂̌̀̌̅̌̓̓̓͑̀͒͌̀̊̾͂͌̓̊̈́͆̑̓̒̈́͘̚͘̚͘͜͠͝ḩ̶̝̩͎̘̺̤̖̣̃̈̀̎̂̏̌͆̍͑͐̋̇͆͗̈́̈́̿͂̈́͌͒̓͂̾̕͘͝͝͝i̷̛̛̛̛̟̭̯̥͕̲̤͍̥̞͙͈̲̐͑̎̾̀̓͐̊͋͋͐̒͊̀͊̑͂͂̈́͆̽̾̀́̐̌̏́̃̐͌̑͊̓̾͊̋̅̃́̂́͗̉̾̎̈́̍̐͋̓͆̆̄̀̽͗̃̈́̾̈́́̂́̒̌̊̈́̽̽̌́̀͆̑̍̊̏̚̕͘̚͝͝͝͝͝ͅs̸̢̤̯͕͕͚͇̩̼̳̣͕͓͖͕͈̦̙̣̝̮̤̞̆͒̎̂̈̂̎̂͒̉̀̀̀̅̎͑͋̋̅̈̂͊̓̃͒̿͒̌̌́̒͋͌̅̿̆͘͜͜͝ͅ?̶̢̧̢̧̧̛̛̝͈̩̥̘͍̝̞̻͕͙͔̻̹͉̞͎̠̞̰̮̲̝̻̘̯͉̠͔̗͚̟̗̫̼̘̘̻̪̫͉̠̺̰̥͔̼̩̞͓͔͖̘̻̫̫͍͍̗̫̟̘̮͆̀̾͊̎̈́͐͆͐͛͊̑͋̎̉͂̋̔́́͘̕͜͜͝͝͝ͅͅͅ










What does mousepad make of that if you cut an paste it?


By the way, I'm working on a "real" PC here. Win 10.

[emma1997]

Somehow we have to live with it. It's not something that text editors can be expected to fix without some human intervention.

Not true IME. Besides Leafpad, Notepad, Wordpad, SEE, Nano, and probably dozens of other editors handle without need to interrupt startup. So it can be done. Just dumb programming IMO.

That zip checks ok on my end. I'm sure you know it has to be unzipped, right? So you seem to have even more ASCII trouble than me. Now I don't feel so bad and glad I started this whole thing. In any case

Analog Weekend started half hour ago so 73.

[jahboater]

None of my editors display it properly, whatever it is supposed to be.

0x96 is not supposed to be anything - its not displayable in any character set:

https://en.wikipedia.org/wiki/ISO/IEC_8 ... age_layout

That's presumably why file describes it as non ISO.

Cat doesn't touch it of course, its the terminal device driver that drops it.

Editors just print anything they like, mine prints a '?' in inverse video.

[jahboater]

That zip checks ok on my end. I'm sure you know it has to be unzipped, right?

He does. The "file" command knows what bzip2 is!

[Heater]

Somehow we have to live with it. It's not something that text editors can be expected to fix without some human intervention.

Not true IME. Besides Leafpad, Notepad, Wordpad, SEE, Nano, and probably dozens of other editors handle without need to interrupt startup. So it can be done. Just dumb programming IMO.

That zip checks ok on my end. I'm sure you know it has to be unzipped, right? So you seem to have even more ASCII trouble than me. Now I don't feel so bad and glad I started this whole thing. In any case

Analog Weekend started half hour ago so 73.

CQ

I don't think so. There are many different code pages/encoding. If your editor opens a file written to one encoding with the wrong encoding it may well appear to be handling it without problems but you may well end up seeing the wrong thing. That is why we have locales and encoding selection menus.

Who said I has a problem unziping it? I got the text out OK.

Here on my Win 10 with LSW machine:

Nano - Fail, displays a box with a question mark in it.
Vim - Fail, displays the troublesome character as "<96>"
MS Visual Studio Code - Fail, displays a diamond with a question mark in it.
Notepad - OK displays "-"
Wordpad- OK displays "-"

Telling VS Code it is some encodings produces:

Code: Select all

Mac Roman:              "ñ"
DOS (CP437):            "û"
Arabic (Windows 1256):  "–"            BINGO! Is this what it was supposed to be!
Arabic (ISO 88596):     An empty square box.
...

A gave up, there are so many. But it seems to only come out correctly in VS Code when selecting a non-standard encoding, Windows 1251 etc.

Of course Notepad and Word pad get this example right, because, you know, Windows. Likely if my Windows was set to some other language I would be seeing the wrong thing.

So no, it's not "dumb programming".

[pica200]

The text editors can't know what encoding is used. At best they can try and guess but it never works out really. The way Notepad displays it is just as wrong as most other text editors because it is unprintable. This way you believe it's working but in reality it's broken. This is exactly why the encoding selection is in Mousepad to make you think about what you are opening. It's annoying as said but there is no better way.

[bensimmo]

Windows-1252 is a superset of ISO-8859-1 that includes the characters from ISO-8859-15 and popular punctuation such as curved quotation marks. It is common that web page tools for Windows use Windows-1252 but label the web page as using ISO-8859-1, this has been addressed in HTML 5, which mandates that pages labeled as ISO-8859-1 must be interpreted as Windows-1252.

And the en-dash is that character.
Still used in webpages, if you follow to the Wikipedia. I think it's old windows webpage creators to blame.

Edit change m to m and add
https://www.w3schools.com/charsets/tryi ... &ent=ndash

[Heater]

Turns out that the byte value 96 hexadecimal is printable in old Microsoft codepages. It is a dash "–". For example see the table on Wikipedia's Windows-1252 page. https://en.wikipedia.org/wiki/Windows-1252

That Wikipedia page describes this encoding as used by legacy Windows components. As such it's not something one should find in web pages.

Wikipedia also goes on to say:

As of March 2020, 0.5% of all web sites declared use of Windows-1252,[1][2] but at the same time 2.3% used ISO 8859-1 (0.7% of top-1000 websites),[1] which by HTML5 standards should be considered the same encoding,[3] so that 2.8% of web sites effectively use Windows-1252.

Which is not quite true. The ISO 8859-1 standard does not define a whole bunch of codes from 7fh to 9fh. They are therefore unprintable in the ISO standard.

[pica200]

Windows-1252 is a superset of ISO-8859-1 that includes the characters from ISO-8859-15 and popular punctuation such as curved quotation marks. It is common that web page tools for Windows use Windows-1252 but label the web page as using ISO-8859-1, this has been addressed in HTML 5, which mandates that pages labeled as ISO-8859-1 must be interpreted as Windows-1252.

And the en-dash is that character.
Still used in webpages, if you follow to the Wikipedia. I think it's old windows webpage creators to blame.

Edit change m to m and add
https://www.w3schools.com/charsets/tryi ... &ent=ndash

What the hell? That's just stupid. They are not the same. What were they thinking?


edit:
In the attached example above i'm not sure if "-" makes sense to separate the comment but dunno.

[bensimmo]

Heater you sideways text can you read this example works ok on Android/chrome
but on Pi/chromium it is all bunched up with loads of whitespace top and bottom.
Sorry for the poor photo.


Android

Android

Screenshot_20200314_101012_com.android.chrome.jpg (130.39 KiB) Viewed 603 times

Raspbian

Raspbian

PhotoPictureResizer_20200314_101402_copy_684x912.jpg (188.98 KiB) Viewed 603 times

[pica200]

Maybe the default font in Raspbian just doesn't support this insanity?

[Heater]

bensimmo,

There is no sideways text in my insane text example. It's just normal text with a ton of Unicode diacritics. You know, all those dots, ticks, and squiggles various languages add to their characters. It's called "zalgo text".

You can make your own here: https://zalgo.org/

I don't expect any good old text terminals to be supporting this nonsense any time soon. Please God.

[bensimmo]

I did not know that such sillyness existed, I did not try reading the sideways stuff either.

My kids are going to have such fun with that.

Luckily not on the Pi

[Heater]

bensimmo,

Great. Glad I brightened up somebodies day

Perhaps emma1997 would consider changing the thread's title to the more correct "How To Get Mousepad To Read Nonstandard Text Files" ?

[scruss]

I don't expect any good old text terminals to be supporting this nonsense any time soon. Please God.

zalgo text of “thankfully, Heater was wrong about that …” displayed in the Raspberry Pi terminal

Screenshot from 2020-03-14 14-21-55.png (20.86 KiB) Viewed 519 times

I wouldn't describe being able to display other languages as ‘nonsense’. More people speak tonal languages than don't.

emma1997: Got a solution for you. It's not super-well tested, but it showed and edited your file without interruption on a Raspberry Pi 4 running Raspbian Buster:

we have a Windows 1252 en dash, folks!

Screenshot from 2020-03-14 15-24-02.png (21.88 KiB) Viewed 519 times

What you need to do:

Code: Select all

# please substitute your two-letter country code
# instead of "CA" in this example

# install a nicer editor
sudo apt install gedit gedit-plugins

# you will only need to do this section once
mkdir ~/.config/localedir
cp /usr/share/i18n/charmaps/CP1252.gz ~/.config/localedir
cd ~/.config/localedir
gunzip CP1252.gz
# this next command is quite noisy
localedef -cv -i en_CA -f ~/.config/localedir/CP1252 ~/.config/localedir/en_CA.CP1252

# you will need to do this for every session;
# perhaps put them in your ~/.profile ?
export LOCPATH=/home/pi/.config/localedir
export LANG=en_CA.CP1252
export LC_ALL=en_CA.CP1252

# now edit that pesky file ...
gedit ~/Desktop/ng2.txt

I can't guarantee that this won't mess up the rest of your system's editing functions, but since you're doing nothing as root, everything should be recoverable.

(for the search engines: How to create a Windows Codepage 1252 locale on Linux on a Raspberry Pi. derived from HOW TO: Enable the Integration Service on Linux with "MS Windows Latin 1 (ANSI), superset of Latin1" codepage )

[Heater]

scruss,

That is a brave attempt by the of the Pi terminal to display my zalgotext example. But really, it's a fail. Does not look anything like it should.

I wouldn't describe being able to display other languages as ‘nonsense’. More people speak tonal languages than don't.

Perhaps not.

But it leads to the madness that is zalgotext and the complication for everybody to support it. Still not done properly in string handling in any programming language.

And it has lead to the insanity of emoji. For example

I mean, really, does this have any place in a serious engineering standard?

[pica200]

At least it has 1 good side: kaomojis

This guy will look terrible in some fonts.
( ͡° ͜ʖ ͡°)━☆ﾟ.*･｡ﾟ