jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

php request to display datas - help with javascript

Wed Mar 08, 2017 12:35 pm

Hello
I'm a teacher in a primary school et we're part of the weatherstation project with raspberry pi
we have a website with our measurements : http://meteovictorhugo.ddns.net:1800/demo
everything is working.
Now, we'd like to have measurements from other stations in the world tu study them and work in sciences, geography...
We can have them here : https://apex.oracle.com/pls/apex/raspbe ... easurement
but it's not really readable for pupils

First, we would like to extract datas to put them in a table and in a second time be able to choose which station we want to display

It's a bit difficult for us

All I succeed to do is this :

Code: Select all

$json = file_get_contents('https://apex.oracle.com/pls/apex/raspberrypi/weather$
$json = utf8_encode($json);
$obj = var_dump(json_decode($json));
 
echo $obj;
which gives us, http://meteovictorhugo.ddns.net:1800/demo/world.php so as to say nothing ...

If someone has an idea, we will be very grateful and you'll help pupils (and their teacher) to learn things

Thank you

jerome
Last edited by jeromec on Sun Mar 19, 2017 7:35 pm, edited 1 time in total.
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Wed Mar 08, 2017 6:10 pm

I find it much easier to do these kind of things in node.js. Like so:

Code: Select all

'use strict';

// Use the https module.
let https = require('https');

function getWeatherReport(url) {

    // Make https request to weather data server
    https.get(url, (res) => {
        const statusCode = res.statusCode;
        const contentType = res.headers['content-type'];

        // Check for errors
        let error;
        if (statusCode !== 200) {
            error = new Error(`Request Failed.\n` +
                            `Status Code: ${statusCode}`);
        } else if (!/^application\/json/.test(contentType)) {
            error = new Error(`Invalid content-type.\n` +
                            `Expected application/json but received ${contentType}`);
        }
        if (error) {
            console.log(error.message);
            // consume response data to free up memory
            res.resume();
            return;
        }

        res.setEncoding('utf8');
        let rawData = '';

        // Append chunks of incomming data to a single string
        res.on('data', (chunk) => rawData += chunk);

        // Here the response is eneded we have all the data.
        res.on('end', () => {

            // Parsing may fail if data corrupted, so wrap in a try block
            try {
                // Parse the returned data and do whatever with the entries.
                let report = JSON.parse(rawData);
                console.log(report);

                // Scan all items in report and print some of it.
                for (let i = 0; i < report.items.length; i++) {
                    console.log('Name:', report.items[i].weather_stn_name);
                    console.log('Ambient temperature:', report.items[i].ambient_temp);
                }

                // Get the next page of the report. (No this is not dangerouly recursive).
                if (report.next){
                    console.log('Next page:', report.next.$ref);
                    let url = report.next.$ref;
                    getWeatherReport(url);
                }
            } catch (e) {
                console.log(e.message);
            }
        });
    }).on('error', (e) => {
        console.log(`Got error: ${e.message}`);
    });
}

let url = 'https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement';

getWeatherReport(url);

Which gets me one page of the weather data and prints some of it. Like so:

Code: Select all

$ node weather.js
{ '$ref': 'https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement?page=1' }
Name: Middletown HS Weather Station
Ambient temperature: 23.51
At the point in the code where data is printed out you could easily write it to a file in HTML format with tables and all. That could then be served up by your Apache/PHP.

You will find tons of node.js tutorials on the net. And video introductions on Youtube.

Interesting project.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Wed Mar 08, 2017 9:50 pm

Thank you for your answer
I have a little success with php using this

Code: Select all

<!DOCTYPE html>
<html>
<body>



<?php

$json = file_get_contents('https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement');
$json = utf8_encode($json);
$obj = json_decode($json);

foreach ($obj->items as $value) {
        $id = isset($value->weather_stn_id) ? $value->weather_stn_id : 'N/A';
        $name = isset($value->weather_stn_name) ? $value->weather_stn_name : 'N/A';
        $lon = isset($value->weather_stn_long) ? $value->weather_stn_long : 'N/A';
        $lat = isset($value->weather_stn_lat) ? $value->weather_stn_lat : 'N/A';
        echo $name . ' : ' . $id . ' : ' . $lat . ' : ' . $lon . '<br/>';
}

?>
</body>
</html>
which gives me a page with weathersation name, id , latitude and longitude
http://meteovictorhugo.ddns.net:1800/demo/world10.php

It's a frist step, but it's not enough I would like the user can choose the weatherstation data he wants to display

I don't know node.js at all, but I will give it a try

thank you again
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Wed Mar 08, 2017 11:06 pm

If you are using Javascript already in your web pages than node.js can be very convenient to use on the server end as well. Same language everywhere!

I notice that the weather data source returns a url referencing the next page of data available. But there only seems to be one page of data so far. The referenced next page has an empty array in it.

It's just that at some point you might have to be ready to follow the references and fetch many pages. The node example I posted already does that.

Do you have a database in your set up? MySql perhaps. You could save the data collected from node to the database. Then have the PHP fetch that data from the database to present a list of stations and allow selection of station etc.

I'd be inclined to do the whole thing in node and forget Apache and PHP. Node makes a web server very easily. But that might be a bit much to handle all at once. Especially if you already have a lot of PHP in use.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Fri Mar 10, 2017 8:03 pm

Hello
thank you for your script and your answer
node js seems a good solution and saving query to database also.

I added some stuff to your script to populate a base, but I have a problem I have to stop the script with ctrl+c because I don't know where to put dbconn.end(), as you can see I'm really a newbie.

Is it possible for you to have a look and give me a hint.

Code: Select all

'use strict';

// Use the https module.
let https = require('https');

var mysql  = require('mysql');
var dbconn = mysql.createConnection({
  host     : 'localhost',
  user     : 'root',
  password : 'xx',
  database : 'xx'
});


dbconn.connect(function(err){
  if(err){
    console.log('Database connection error');
  }else{
    console.log('Database connection successful');
  }
});




function getWeatherReport(url) {

    // Make https request to weather data server
    https.get(url, (res) => {
        const statusCode = res.statusCode;
        const contentType = res.headers['content-type'];

        // Check for errors
        let error;
        if (statusCode !== 200) {
            error = new Error(`Request Failed.\n` +
                            `Status Code: ${statusCode}`);
        } else if (!/^application\/json/.test(contentType)) {
            error = new Error(`Invalid content-type.\n` +
                            `Expected application/json but received ${contentType}`);
        }
        if (error) {
            console.log(error.message);
            // consume response data to free up memory
            res.resume();
            return;
        }

        res.setEncoding('utf8');
        let rawData = '';

        // Append chunks of incomming data to a single string
        res.on('data', (chunk) => rawData += chunk);

        // Here the response is eneded we have all the data.
        res.on('end', () => {

            // Parsing may fail if data corrupted, so wrap in a try block
            try {
                // Parse the returned data and do whatever with the entries.
                let report = JSON.parse(rawData);
                console.log(report);

                // Scan all items in report and print some of it.
                for (let i = 0; i < report.items.length; i++) {
                    console.log('Name:', report.items[i].weather_stn_name);
                    console.log('Ambient temperature:', report.items[i].ambient_temp);
		    console.log('ID:', report.items[i].weather_stn_id);
		    var record= { LATITUDE : report.items[i].weather_stn_lat, LONGITUDE : report.items[i].weather_stn_long, STATION_NAME: report.items[i].weather_stn_name, STATION_ID: report.items[i].weather_stn_id, TEMPERATURE: report.items[i].ambient_temp, DATE: report.items[i].reading_timestamp};
		    dbconn.query('INSERT INTO WEATHER_MEASUREMENT SET ?', record, function(err,res){
                      if(err) throw err;

                      console.log('Last record insert id:', res.insertId);
                });
                }

                // Get the next page of the report. (No this is not dangerouly recursive).
                if (report.next){
                    console.log('Next page:', report.next.$ref);
                    let url = report.next.$ref;
                    getWeatherReport(url);
                }
            } catch (e) {
                console.log(e.message);
            }
        });
    }).on('error', (e) => {
        console.log(`Got error: ${e.message}`);
    });
}

let url = 'https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement';

getWeatherReport(url);

thank you

jerome
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Fri Mar 10, 2017 8:37 pm

jeromec,

Cool, that looks like great progress already.

It's a long time since I used mysql, and never with node, but after a quick glance I think something like this might do the trick:

Code: Select all

                // Get the next page of the report if there is one. (No this is not dangerouly recursive).
                if (report.next) {
                    console.log('Next page:', report.next.$ref);
                    let url = report.next.$ref;
                    getWeatherReport(url);
                } else {
                    // Got everything, may as well close the db connection.
                    dbconn.end();
                }
Basically, if the page of data just fetched does not link to another one, then it must be the end and we can close the db connection.

If there are no connections open or events to wait for then node should quit the program.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Fri Mar 10, 2017 10:13 pm

Thank you, it works perfectly !
http://meteovictorhugo.ddns.net:1800/demo/

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 11:48 am

I've made a post on the weatherstation forum to show the results of my research to display datas from other stations
viewtopic.php?f=112&t=177401

Thank you Heater for helping me with the first step!
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 3:24 pm

Great stuff.

My next step would have been to replace the Apache/PHP by adding HTTP server capability to the node.js program. It's very easy if you use the Express module.

But as you have it working in PHP might as well stick with that....for now :)

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 5:51 pm

Thank you.
I will have a look at the express module.
But I have to learn.

The thing that bothers me is the slowness of my php pages with queries.
I don't know where it comes from : queries, php the fact that my server is on the raspberry...

Regards

Jérôme
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 6:10 pm

Not sure.

Chrome's dev tools tell me there is a 6 second waiting time. The actual data transfer takes no time at all.

How have you set this up?

Is the page request, running PHP, which then runs the node.js to fetch the data and populate the database, and then queries the database and returns the page?

The plan was that the node.js runs independently fetching the data every hour or so. The data is then in the db ready to be fetched by PHP on a page request.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 8:23 pm

Thanks for your answer.

There is a cron on the the node js script which populates database every hour, then the php page which query the database.

maybe it's because of my query :

Code: Select all

$select='select * from (select ID as Id, DATE as Date, LONGITUDE as longitude, LATITUDE as latitude,STATION_NAME as Station,TEMPERATURE as Temperature FROM WEATHER_MEASUREMENT WHERE DATE(DATE)>=CURDATE() ORDER BY DATE DESC) as T1 GROUP BY Station';
I think it's too complex, I've tried easier way but havent' succeed.
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Wed Mar 15, 2017 9:08 pm

Sounds good.

I'm no MySql expert, have not used it since last century, but your query looks fine to me.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Thu Mar 16, 2017 10:25 pm

I've found the guilty one : google reverse geocoding api

with : http://meteovictorhugo.ddns.net:1800/demo/monde.php

without : http://meteovictorhugo.ddns.net:1800/demo/mondebis.php

it's a pity I find it quite useful ...
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas from a webpage

Fri Mar 17, 2017 5:14 am

That's much better. Almost instant!

Where are you doing that reverse geocoding? On the server?

You should be able to do that in the background from your node.js program. Just have it do the geocoding lookup when it gets the weather station info and save what it finds in your DB.

See the node.js "request" module.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Fri Mar 17, 2017 8:32 pm

Right, inserting it to database seems the best solution

I haven't succed with request

I've tried with node geocoder library http://nchaulet.github.io/node-geocoder/

I partially succed with this script, I can log the results from the reverse geocoding, but I don't understand how to choose the data I want and insert it in my database. I've tried to take example from your script but I'm stuck.

Code: Select all

'use strict';

// Use the https module.
let https = require('https');

var mysql  = require('mysql');

var dbconn = mysql.createConnection({
  host     : 'localhost',
  user     : 'root',
  password : '',
  database : 'world'
});

var NodeGeocoder = require('node-geocoder');

var options = {
  provider: 'google',
  language: 'fr',
  
  // Optional depending on the providers
  httpAdapter: 'https', // Default
  apiKey: '', // for Mapquest, OpenCage, Google Premier
  formatter: null         // 'gpx', 'string', ...
  
};

var geocoder = NodeGeocoder(options);

dbconn.connect(function(err){
  if(err){
    console.log('Database connection error');
  }else{
    console.log('Database connection successful');
  }
});




function getWeatherReport(url) {

    // Make https request to weather data server
    https.get(url, (res) => {
        const statusCode = res.statusCode;
        const contentType = res.headers['content-type'];

        // Check for errors
        let error;
        if (statusCode !== 200) {
            error = new Error(`Request Failed.\n` +
                            `Status Code: ${statusCode}`);
        } else if (!/^application\/json/.test(contentType)) {
            error = new Error(`Invalid content-type.\n` +
                            `Expected application/json but received ${contentType}`);
        }
        if (error) {
            console.log(error.message);
            // consume response data to free up memory
            res.resume();
            return;
        }

        res.setEncoding('utf8');
        let rawData = '';

        // Append chunks of incomming data to a single string
        res.on('data', (chunk) => rawData += chunk);

        // Here the response is eneded we have all the data.
        res.on('end', () => {

            // Parsing may fail if data corrupted, so wrap in a try block
            try {
                // Parse the returned data and do whatever with the entries.
                let report = JSON.parse(rawData);
                console.log(report);

                // Scan all items in report and print some of it.
                for (let i = 0; i < report.items.length; i++) {
                    console.log('Name:', report.items[i].weather_stn_name);
                    console.log('Ambient temperature:', report.items[i].ambient_temp);
					console.log('ID:', report.items[i].weather_stn_id);
		            geocoder.reverse({lat:report.items[i].weather_stn_lat, lon:report.items[i].weather_stn_long}, function(err, res2) {
					console.log(res2)
					
					});
			var record= { LATITUDE : report.items[i].weather_stn_lat, LONGITUDE : report.items[i].weather_stn_long, STATION_NAME: report.items[i].weather_stn_name, STATION_ID: report.items[i].weather_stn_id, TEMPERATURE: report.items[i].ambient_temp, DATE: report.items[i].reading_timestamp};
		    dbconn.query('INSERT IGNORE INTO weather SET ?', record, function(err,res){
                      if(err) throw err;

                      console.log('Last record insert id:', res.insertId);
                });
                }

                // Get the next page of the report. (No this is not dangerouly recursive).
                if (report.next){
                    console.log('Next page:', report.next.$ref);
                    let url = report.next.$ref;
                    getWeatherReport(url);
                } else {
		    dbconn.end();
            }} catch (e) {
                console.log(e.message);
            }
        });
    }).on('error', (e) => {
        console.log(`Got error: ${e.message}`);
    });
}

let url = 'https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement';

getWeatherReport(url);

you have time can you have a look and give me an hint,

in all cases thank you for your help so far !

jerome
http://meteovictorhugo.ddns.net:1800/demo/

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas from a webpage

Sun Mar 19, 2017 7:34 pm

hello, it's me again
and once again I had small success and once again I'm stuck

I've succeed in using node-geocoder, to get city and country from my lat and longitude
and I'm stuck because these 2 variable are in an if statement, and I can't access them outside this if and write them in the dtabase.

I put my script again, I've been searching a lot, maybe the answer is obvious and surely I'm tired

Code: Select all

'use strict';

// Use the https module.
let https = require('https');

var mysql  = require('mysql');



var dbconn = mysql.createConnection({
  host     : 'localhost',
  user     : 'root',
  password : 'victorhugo',
  database : 'world'
});

var NodeGeocoder = require('node-geocoder');

var options = {
  provider: 'google',
  language: 'fr',
  formatterPattern:'%c,%P',
  
  // Optional depending on the providers
  httpAdapter: 'https', // Default
  apiKey: 'AIzaSyBQaSaxHzoj_rRQF8K8tVrtRWTsML4suTY', // for Mapquest, OpenCage, Google Premier
  formatter: 'string'        // 'gpx', 'string', ...
  
};

var geocoder = NodeGeocoder(options);

dbconn.connect(function(err){
  if(err){
    console.log('Database connection error');
  }else{
    console.log('Database connection successful');
  }
});




function getWeatherReport(url) {
    // Make https request to weather data server
    https.get(url, (res) => {
        const statusCode = res.statusCode;
        const contentType = res.headers['content-type'];

        // Check for errors
        let error;
        if (statusCode !== 200) {
            error = new Error(`Request Failed.\n` +
                            `Status Code: ${statusCode}`);
        } else if (!/^application\/json/.test(contentType)) {
            error = new Error(`Invalid content-type.\n` +
                            `Expected application/json but received ${contentType}`);
        }
        if (error) {
            console.log(error.message);
            // consume response data to free up memory
            res.resume();
            return;
        }

        res.setEncoding('utf8');
        let rawData = '';

        // Append chunks of incomming data to a single string
        res.on('data', (chunk) => rawData += chunk);

        // Here the response is eneded we have all the data.
        res.on('end', () => {

            // Parsing may fail if data corrupted, so wrap in a try block
            try {
                // Parse the returned data and do whatever with the entries.
                let report = JSON.parse(rawData);
                console.log(report);

                // Scan all items in report and print some of it.
		var city='';                
		var country='';
		for (let i = 0; i < report.items.length; i++) {
                    console.log('Name:', report.items[i].weather_stn_name);
                    console.log('Ambient temperature:', report.items[i].ambient_temp);
		    console.log('ID:', report.items[i].weather_stn_id);
		    geocoder.reverse({lat:report.items[i].weather_stn_lat, lon:report.items[i].weather_stn_long}, function(err, results) {
					
					  if (err) {
						city = "";
						country = "";
					
					} else {
						var resultsstring = results.toString();
					    var adresse = resultsstring.split(",");
					    city = adresse[0];
					    country = adresse[1];
					}
					// here it works

                                        console.log(city);
					console.log(country);
					});

			//here it does'nt work !!

			var record= {CITY : city, LATITUDE : report.items[i].weather_stn_lat, LONGITUDE : report.items[i].weather_stn_long, STATION_NAME: report.items[i].weather_stn_name, STATION_ID: report.items[i].weather_stn_id, TEMPERATURE: report.items[i].ambient_temp, DATE: report.items[i].reading_timestamp};
		    dbconn.query('INSERT IGNORE INTO WEATHER_MEASUREMENT SET ?', record, function(err,res){
                      if(err) throw err;

                      console.log('Last record insert id:', res.insertId);
                });
                }

                // Get the next page of the report. (No this is not dangerouly recursive).
                if (report.next){
                    console.log('Next page:', report.next.$ref);
                    let url = report.next.$ref;
                    getWeatherReport(url);
                } else {
		    dbconn.end();
            }} catch (e) {
                console.log(e.message);
            }
        });
    }).on('error', (e) => {
        console.log(`Got error: ${e.message}`);
    });
}

let url = 'https://apex.oracle.com/pls/apex/raspberrypi/weatherstation/getalllastmeasurement';

getWeatherReport(url);

thanks
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas - help with javascript

Sun Mar 19, 2017 10:06 pm

Your code is getting a bit big to get a full grasp of but...

You have to be aware of the "scope" or validity of your variables.

For example. Where you commented that "it works" you are inside a function that was given as a parameter to geocoder.reverse. So "city" is in scope I guess.

BUT, "city" will have no meaningful value until a response is received from the reverse look up and the given function is called. Which may be seconds or minutes later. Or never.

So, your next statement "var record= ..." which is executed inline immediately has no valid "city" to work with.

Looks like you have to move the "dbconn.query" stuff into that same call back function so that it is executed when the data is valid.

I think that:

var city='';
var country='';

Should also be moved into the call back function.

The main point here is that JS does not run around a loop sequentially doing what you write and waiting for things to happen. As it might be be in C or Python or whatever. It is event driven. And those events, like the response to the reverse look up can happen at any time. Then the call back function is called. Then you can do the next thing.

Hope this makes some kind of sense.

jeromec
Posts: 116
Joined: Sun Apr 14, 2013 5:14 pm

Re: php request to display datas - help with javascript

Tue Mar 21, 2017 12:09 pm

Thanks, it works and it's a lot faster than before : http://meteovictorhugo.ddns.net:1800/de ... terbis.php

I think I will have to find another solutions later because of the call limits by google api

Some longitude and latitude are not geocoded, but I can do it manually.

thanks again !

jerome
http://meteovictorhugo.ddns.net:1800/demo/

Heater
Posts: 7906
Joined: Tue Jul 17, 2012 3:02 pm

Re: php request to display datas - help with javascript

Tue Mar 21, 2017 1:42 pm

Great. That's nice and fast, even on my phone.

Well done!

Return to “Other languages”

Who is online

Users browsing this forum: No registered users and 8 guests