User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Bash Script help

Sun May 31, 2015 6:08 pm

Hi when i do this

Code: Select all

wget -O youtube.txt youtube.com  and i do cat /home/pi/youtube.txt| awk 'NR==1' | tail -c 119

I get for example 1234raspberrypi
The problem is when i go by a friend and use their network and do the same command

Code: Select all

wget -O youtube.txt youtube.com  and i do cat /home/pi/youtube.txt| awk 'NR==1' | tail -c 119

I get 4raspberrypi instead of getting the full 1234raspberrypi
Does anyone know whats the problem?
:mrgreen:
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
MarkHaysHarris777
Posts: 1820
Joined: Mon Mar 23, 2015 7:39 am
Location: Rochester, MN
Contact: Website

Re: Bash Script help

Sun May 31, 2015 6:40 pm

it is what it is... what problem do you think you have?
marcus
:ugeek:

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Sun May 31, 2015 7:12 pm

The problem is I want to be able to get

Code: Select all

1234raspberrypi
every time on any network :mrgreen:
SO any suggestions? What is happening and why doesn't it output the same string?
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
jojopi
Posts: 3079
Joined: Tue Oct 11, 2011 8:38 pm

Re: Bash Script help

Mon Jun 01, 2015 2:17 am

If you always want to get "1234raspberrypi", then use:

Code: Select all

echo "1234raspberrypi"
You have had at least three threads about this issue, but the problem is that you cannot explain, even to humans, what you are trying to do. The file you refer to does not actually contain the strings "1234" or "raspberrypi", so which part are you really trying to extract?

In order for the computer to automate whatever extraction you are trying to do, it will need precise and unambiguous rules to follow. A good first step will be to formulate those rules in English. Translating them to work in Bash or Perl or Python is the part we can definitely help with.

Alternatively, if you can explain why you are trying to extract a string from a file, perhaps we can suggest a completely different solution to the whole problem.

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 3:39 am

The first post shows and explains exactly what am trying to do, at home am basically downloading an html page such as youtube.com and read a line in that html file. The problem is i am getting a different result when i do the command to read a specific line in the html file, when i go by a friend and use their internet to do the task.
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
jojopi
Posts: 3079
Joined: Tue Oct 11, 2011 8:38 pm

Re: Bash Script help

Mon Jun 01, 2015 4:30 am

expandables wrote:The first post shows and explains exactly what am trying to do
Sorry, but it absolutely does not.

The code in your first post takes the last 119 characters of the first line of a web page source file. You say that you expect the result to be "1234raspberrypi", but that is not even 119 characters long. When I run your code I get this useless incomplete chunk of Javascript and HTML:

Code: Select all

if (a.length > 1) {ytcfg.d()[a[0]] = a[1];} else {for (var k in a[0]) {ytcfg.d()[k] = a[0][k];}}}};</script>  <script>
I still have no idea which part of the page you intended to extract, but that is probably not it.

If you want to find something in an HTML file, especially one that is generated on demand like youtube, you are going to have to look for the context that surrounds it. Just selecting a particular fixed range of characters will not be reliable.

User avatar
B.Goode
Posts: 8225
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Bash Script help

Mon Jun 01, 2015 7:47 am

expandables wrote:basically downloading an html page such as youtube.com and read a line in that html file. The problem is i am getting a different result when i do the command to read a specific line in the html file, when i go by a friend and use their internet to do the task.
So, Important Lesson Number One is that you cannot rely on a massive service like YouTube to return the same result to every user, every time, everywhere...

Just display the first line of the file yourself to check...

Code: Select all

head -1 youtube.txt
When I do this I do not see anything related to 'raspberry' or to the hostname of my RPi.

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 4:08 pm

In my post i was showing an example. On the real website

Code: Select all

http://www.giniko.com/watch.php?id=209
which i am downloading the html file it has the same exact string every time. I first use the "find" command to find the string am looking for, example find /home/pi/giniko.txt -type f -exec egrep -w 'file' {} \;
which finds a string that says this

Code: Select all

file: "http://nimlive1.giniko.com/atlastv/atlastv.stream/playlist.m3u8?wmsAuthSign=c2VydmVyX3RpbWU9Ni8xLzIwMTUgNDoxMjoyNSBQTSZoYXNoX3ZhbHVlPVJxeW5aeXM5cWhWS2NtZDlZSk5zQlE9PSZ2YWxpZG1pbnV0ZXM9NjA="
then i use the cat command to find the 119 characters but it is not getting the full characters when i do it on a different network.
On my network i get

Code: Select all

c2VydmVyX3RpbWU9Ni8xLzIwMTUgNDoxMjoyNSBQTSZoYXNoX3ZhbHVlPVJxeW5aeXM5cWhWS2NtZDlZSk5zQlE9PSZ2YWxpZG1pbnV0ZXM9NjA="
On my neighbor network i get

Code: Select all

ign=c2VydmVyX3RpbWU9Ni8xLzIwMTUgNDoxMjoyNSBQTSZoYXNoX3ZhbHVlPVJxeW5aeXM5cWhWS2NtZDlZSk5zQlE9PSZ2YWxpZG1pbnV0ZXM9NjA="
I just didn't wanted to post the real website so i use youtube as an example.
I use VLC to play that string, so that is why i need it to be exactly the same everytime.

Code: Select all

matchbox-window-manager & xinit /usr/bin/vlc http://nimlive1.giniko.com/1tvarm/1tvarm/playlist.m3u8?wmsAuthSign=$line --no-qt-error-dialogs
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

Joe Schmoe
Posts: 4277
Joined: Sun Jan 15, 2012 1:11 pm

Re: Bash Script help

Mon Jun 01, 2015 4:36 pm

When you are posting something that is "just an example", I think it is important to make this fact clear. Otherwise, people will quite naturally assume that the thing you posted is in fact your real problem, and to react/respond accordingly. And, in fact, this problem, the way you posted it, seems nonsensical and, frankly, make you look like a beginner.

If we had known that it was an abstracted example, we'd have responded differently.
And some folks need to stop being fanboys and see the forest behind the trees.

(One of the best lines I've seen on this board lately)

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 4:40 pm

OK sir can you tell me why i am getting two different strings?
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
B.Goode
Posts: 8225
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Bash Script help

Mon Jun 01, 2015 4:50 pm

I would need to be persuaded that this is not an attempt to sidestep the access controls that giniko.com choose to implement. (Because if so that would be off-limits for these forums.)

lazarus78
Posts: 236
Joined: Thu Jul 25, 2013 5:16 pm

Re: Bash Script help

Mon Jun 01, 2015 4:55 pm

Well your result isn't even 119 characters, but the latter result is.

The string you seem to be looking for, could you not just do some wizardry and cut out what you need? Or add in anything.

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 5:08 pm

I added something like this but i dont know what happens if i go on another network. Will the string be different as well?

Code: Select all

if grep -q ign= "/home/pi/.Roms/cable3/rtmp/giniko.txt"; then
matchbox-window-manager & xinit /usr/bin/vlc http://nimlive1.giniko.com/1tvarm/1tvarm/playlist.m3u8?wmsAuthS$line --no-qt-error-dialogs
else
matchbox-window-manager & xinit /usr/bin/vlc http://nimlive1.giniko.com/1tvarm/1tvarm/playlist.m3u8?wmsAuthSign=$line --no-qt-error-dialogs
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
rpdom
Posts: 14979
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Bash Script help

Mon Jun 01, 2015 5:36 pm

expandables wrote:In my post i was showing an example. On the real website

Code: Select all

http://www.giniko.com/watch.php?id=209
which i am downloading the html file it has the same exact string every time. I first use the "find" command to find the string am looking for, example find /home/pi/giniko.txt -type f -exec egrep -w 'file' {} \;
Is the file always going to be "/home/pi/giniko.txt"? Because if it is (or another known name), then that is a pointless use of "find". find searches for files or directories with certain attributes. It then can return a list of those names or do certain actions on them.

In this case you have it searching for one specific file "/home/pi/giniko.txt", so that's what it finds. You then tell it to run egrep on that file. Why not just run egrep on that file directly?

Code: Select all

egrep -w 'file' /home/pi/giniko.txt
will produce the same result as

Code: Select all

find /home/pi/giniko.txt -type f -exec egrep -w 'file' {} \;
It's easier to read and understand too :)

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 5:38 pm

Code: Select all

wget -O ginko.txt http://www.giniko.com/watch.php?id=209

Code: Select all

find /home/pi/giniko.txt  -type f -exec egrep -w "file" {} \; > /home/pi/giniko2.txt 

  
cat /home/pi/.Roms/giniko2.txt| awk 'NR==1' | tail -c 119 > /home/pi/giniko3.txt 

Code: Select all

#!/bin/bash

line=$(head -n 1 /home/pi/giniko3.txt)
if grep -q ign= "/home/pi/giniko3.txt"; then
matchbox-window-manager & xinit /usr/bin/vlc http://nimlive1.giniko.com/1tvarm/1tvarm/playlist.m3u8?wmsAuthS$line --no-qt-error-dialogs
else
matchbox-window-manager & xinit /usr/bin/vlc http://nimlive1.giniko.com/1tvarm/1tvarm/playlist.m3u8?wmsAuthSign=$line --no-qt-error-dialogs
fi
exit 0
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

Joe Schmoe
Posts: 4277
Joined: Sun Jan 15, 2012 1:11 pm

Re: Bash Script help

Mon Jun 01, 2015 5:56 pm

Note, BTW, that there is nothing wrong with posting an abstracted example, and I certainly didn't mean to imply that there was, but the point is you need to identify it as such. Otherwise, people will, as I've said, assume that it is your real problem...

In this case, posting the abstracted example, and clearly identifying it as such, would have avoided the stupid "Oh, we can't help you try to get around blah, blah, blah, " stuff that you saw in this thread. I.e., you abstracted it for a reason, and a good one at that.
And some folks need to stop being fanboys and see the forest behind the trees.

(One of the best lines I've seen on this board lately)

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Mon Jun 01, 2015 6:02 pm

Ok i posted my full script. Can anyone please tell me why do i get different strings on a different network?
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
jojopi
Posts: 3079
Joined: Tue Oct 11, 2011 8:38 pm

Re: Bash Script help

Mon Jun 01, 2015 8:51 pm

expandables wrote:On the real website which i am downloading the html file it has the same exact string every time.
The string is not the same every time, or you could just use echo "thestring", as I said earlier. This is important, because to match a varying string successfully, you will need to know which parts are fixed and which are changing. Not only is the string not always the same, it is not even always the same length, which is why the cut -c119 method failed.

In your previous abstracted strings, it was always impossible to tell how the string could be distinguished from the rest of the line. What you should have said is that you had a line of the form:

Code: Select all

    file: "http://site/page?arg=xxxxxxxxxxxxxxxxxxxxxx"
(And that the arg is base64 encoded, so it sometimes ends with another equals.) Now it is clear which parts are fixed and which are changing.

In this case the easiest way to extract the URL is to cut between the double quotes:

Code: Select all

grep "file:" THEFILE.txt |cut -d\" -f2
If you want just the base64 argument, remove everything up to the end of the fixed part:

Code: Select all

grep "file:" THEFILE.txt |cut -d\" -f2 |sed -e's/.*arg=//'
(Here I used the fixed part from my abstracted example, rather that the real one.)

User avatar
expandables
Posts: 654
Joined: Fri Jun 27, 2014 7:34 pm
Location: Neverland with Michael Jackson

Re: Bash Script help

Tue Jun 02, 2015 12:39 am

Hey thanks i really appreciate the help that works great! :mrgreen: Maybe i need to learn sed it seems to be ideal for my needs.
By thinking like an engineer you can create a raspberry pi.
Michael Jackson enthusiast.
I got the PI model B, B+ and PI 2 model B.
When will I get the A? I don't know.

User avatar
rpdom
Posts: 14979
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Bash Script help

Tue Jun 02, 2015 5:26 am

expandables wrote:Hey thanks i really appreciate the help that works great! :mrgreen: Maybe i need to learn sed it seems to be ideal for my needs.
It can take some time to get your head around sed, especially when you get to complex pattern matching, but it is very powerful. For example, you could change:

Code: Select all

grep "file:" giniko.txt |cut -d\" -f2 |sed -e's/.*AuthSign=//'
to the slightly more complex looking, but just one sed command rather that grep, cut and sed:

Code: Select all

sed -n "s/^.*file: .*AuthSign=\(.*\)\"/\1/p" ginko.txt

Return to “Beginners”