I am not. still using the previous version.CharlesGodwin wrote: ↑Tue Sep 18, 2018 12:32 amAre you using the latest version. I made some changes in the last 2 days
Re: Using a Pi to Log Data in a Magnum Energy System
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
sagreen83 wrote: ↑Tue Sep 18, 2018 12:30 amtestRS485-20180917_1920.zipCharlesGodwin wrote: ↑Sat Sep 15, 2018 11:09 am@Scott. glad its working. I'd really like a copy of the txt file generated when you run the testrs485 script as I'm trying to fine tune the packet processing and every site seems to have different packet "features" so it's a learning process. Please let me know what equipment you have in the network too.
Thanks
Charles
@Charles
See attached file created from testrs485
Looks like there are some unknown's in the file. such as:
Bytes read:21:UNKNOWN=>0005140A6419280601EB9B820005000000000000D0
Bytes read:22:UNKNOWN=>400001F8000577000100331E312A73000002025800FE
Again, my configuration is
Magnum MS4448PAE
MagWeb interface
BMK
AGS (w/Honda EU7000is generator)
ME-ARC
Outback FlexMax 80 charge controller
12 VmaxTank 155slr Batteries (420ah)
6 300 watt 72 cell solar panels.
Thanks,
Scott...
Got the file. I noticed it hiding in the forum.
Perhaps if you used a current version of the program you would not need to report problems that have already been fixed.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
@scott
I reran your data using the latest parsing code and all unknown packets are identified except the first packet, which will almost always be a throw away as reading begins in the middle of an unknown conversation.
although I identify a type D0 remote packet now, I have no idea what it is as Magnum Energy has not published specs for the type. Or I haven't seen the document yet. If anyone has it, let me know.
I reran your data using the latest parsing code and all unknown packets are identified except the first packet, which will almost always be a throw away as reading begins in the middle of an unknown conversation.
although I identify a type D0 remote packet now, I have no idea what it is as Magnum Energy has not published specs for the type. Or I haven't seen the document yet. If anyone has it, let me know.
- Attachments
-
- testRS485-20180918_1711.txt.gz
- (1.35 KiB) Downloaded 104 times
Re: Using a Pi to Log Data in a Magnum Energy System
Making progress. We've recreated the 3 graphs from the normal MagWeb site... its a work in progress, but I'm excited to get some more functionality than we have with magweb!
Re: Using a Pi to Log Data in a Magnum Energy System
Original Magweb Graphs
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Change to distribution
I have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution/tags
Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution/tags
Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.
Last edited by CharlesGodwin on Sat Sep 22, 2018 4:50 pm, edited 1 time in total.
Re: Change to distribution
@CharlesGoodwin,CharlesGodwin wrote: ↑Fri Sep 21, 2018 5:08 pmI have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution
Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.
Awesome work! Thanks for all that you have done to make this happen!
Scott...
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
I don't want this topic to branch into a discourse on statistics and graphing but.....
I just came across https://grafana.com/ that looks interesting.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
I am ready to publish release 1.0.0 but I am asking for input before I do so.
If anyone has installed and tested this software and has NOT contacted me I'd like to know of any problems or requests.
I plan to publish this week. The only changes since my last test release are:
Future Plans:
If you have been using the software without modification you don't need to read any further.
I have decided to review the architecture of the product before it goes to release 1.0.
I will be making these changes:
If anyone has installed and tested this software and has NOT contacted me I'd like to know of any problems or requests.
I plan to publish this week. The only changes since my last test release are:
- Allow override of text for Date field
- Allow override, using a filter, of JSON field names
- Provide access, via GitLab, to my entire Eclipse project used to develop this software
Future Plans:
- Support to generate a specific MySQL schema for a site (using filter or no filter)
- Support to log MySQL INSERT statements to a file
- Support for logging directly to a MySQL database. (This will take a little longer)
- Any requests?
If you have been using the software without modification you don't need to read any further.
I have decided to review the architecture of the product before it goes to release 1.0.
I will be making these changes:
- Enhancing time stamp logic to allow setting the servers timezone as some statistical software like UTC/GMT timezone data.
- Adding timezone as a immutable field in the data
- Stripping all enhanced formatting from server. Everything to do with CSV, SQL and formatting will be offloaded to the client access code
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
__The deed is done!__
The software has been significantly simplified since earlier versions. It is now just a REST website. Going to the server's URL (i.e. http://localhost:19450) will return a JSON file of all the data points in the Magnum Energy network.
All of the value added processing is now handled by client side utilities with examples implemented in command line PHP.. The plugins interface has been simplified to make adding data providers easier but since it is now just a curl interface the ME server can be combined with separate programs to generate a log record without doing any Java work.
Anyone who worked with the pre-release versions should review the current software and reference manual as much has changed.
I hope this can be useful to some of you.
Thank you to David, Lee and Yam for help during this project.
Always take the newest release from here: https://gitlab.com/Magnum_Energy/distribution/tags
Charles Godwin
The software has been significantly simplified since earlier versions. It is now just a REST website. Going to the server's URL (i.e. http://localhost:19450) will return a JSON file of all the data points in the Magnum Energy network.
All of the value added processing is now handled by client side utilities with examples implemented in command line PHP.. The plugins interface has been simplified to make adding data providers easier but since it is now just a curl interface the ME server can be combined with separate programs to generate a log record without doing any Java work.
Anyone who worked with the pre-release versions should review the current software and reference manual as much has changed.
I hope this can be useful to some of you.
Thank you to David, Lee and Yam for help during this project.
Always take the newest release from here: https://gitlab.com/Magnum_Energy/distribution/tags
Charles Godwin
Last edited by CharlesGodwin on Thu Oct 18, 2018 5:46 pm, edited 5 times in total.
Re: Using a Pi to Log Data in a Magnum Energy System
Charles,
I just tried to hit the new link, and im getting a 404 error. Seems the link doesnt exist yet.
Scott...
I just tried to hit the new link, and im getting a 404 error. Seems the link doesnt exist yet.
Scott...
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
Standby. It's an access control problem. i'm new to Gitlab so I'm learning.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
Fixed. Try it now.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
I have release a revised version of the server software. There was a bug that caused it to fail when processing PT-100 packets.
It is available here: Take the newest version
https://gitlab.com/Magnum_Energy/distribution/tags
If you have an RSS news reader, add this to get updates of any new release.
https://gitlab.com/Magnum_Energy/distri ... ormat=atom
It is available here: Take the newest version
https://gitlab.com/Magnum_Energy/distribution/tags
If you have an RSS news reader, add this to get updates of any new release.
https://gitlab.com/Magnum_Energy/distri ... ormat=atom
Re: Using a Pi to Log Data in a Magnum Energy System
Charles,CharlesGodwin wrote: ↑Sun Sep 16, 2018 8:29 pm
Plugins
Adding extra "device" data is straight forward but not yet documented. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is dynamically loaded at run-time using the plugins= option. Your can have multiple classes by coma delimiting their class names in the option. This class is responsible for returning an ArrayList of objects that implement ca.godwin.magnum.DeviceRecord. How you collect the data in these objects is implementation specific. For example my MorningStar controller has a ModBus interface so I communicate using their published interface, extract the data I want, populate the object so the two methods provide data and I'm done. I also have a class that returns a weather record with current temperature, pressure and humidity. The values are appended after the magnum fields in the order that the plugins are defined and order of each DeviceRecord object in the ArrayList.
I'm getting close to having a REST server ready that will read data from my charge controller. Any chance that I could get a copy of the java that you used for your Morningstar plugin so that I can use it as a start for my charge controller reader?
Thank you!
Scott...
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
@scott
Yes, I can send you the Morningstar java. Just send me an email and I forward it to you. I do not want to publish it.
But....
I think that code will be overkill as it deals in painful detail with extracting data from a ModBus interface. If you're looking for an example of inserting some extra JSON data into the JSON generated by my reader I recommend you look at the code in JSONReader.java in the GitLab repository (https://gitlab.com/Magnum_Energy/distri ... win/magnum) as this code is a simple process of: read a JSON record, stuff it in the stream, repeat. Changing the file reader to a REST call would be straightforward.
If you want the Morningstar stuff, just email. You can find my email address in the GitLab project GETTING_STARTED.md doc.
Yes, I can send you the Morningstar java. Just send me an email and I forward it to you. I do not want to publish it.
But....
I think that code will be overkill as it deals in painful detail with extracting data from a ModBus interface. If you're looking for an example of inserting some extra JSON data into the JSON generated by my reader I recommend you look at the code in JSONReader.java in the GitLab repository (https://gitlab.com/Magnum_Energy/distri ... win/magnum) as this code is a simple process of: read a JSON record, stuff it in the stream, repeat. Changing the file reader to a REST call would be straightforward.
If you want the Morningstar stuff, just email. You can find my email address in the GitLab project GETTING_STARTED.md doc.
Re: Using a Pi to Log Data in a Magnum Energy System
Charles,
My downstream system needs the date to be in the format of...
2001-07-04T12:08:56.235-0700
So, per the directions for SimpleDateFormat Im setting the line in magnum_server.options as follows.
dateformat=yyyy-MM-dd'T'HH:mm:ss.SSSZ
When I make this change the server is returning nothing on port 19450. Neither the 'T' or the .SSSZ parts of the format string work.
Thoughts?
Scott...
My downstream system needs the date to be in the format of...
2001-07-04T12:08:56.235-0700
So, per the directions for SimpleDateFormat Im setting the line in magnum_server.options as follows.
dateformat=yyyy-MM-dd'T'HH:mm:ss.SSSZ
When I make this change the server is returning nothing on port 19450. Neither the 'T' or the .SSSZ parts of the format string work.
Thoughts?
Scott...
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
Thoughts
1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream
I'll test tomorrow when I'm near my system
1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream
I'll test tomorrow when I'm near my system
Re: Using a Pi to Log Data in a Magnum Energy System
1 : The server works fine with the default dateformat (without override)CharlesGodwin wrote: ↑Sun Nov 18, 2018 2:03 amThoughts
1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream
I'll test tomorrow when I'm near my system
2 : No messages show up in the log file indicating a problem
3 : Haven't thought about reformating the downstream system. I was hoping to be able to deal with this server side.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
I have tested and the problem is the 'Z'. It works, in a test setup, without the Z.
This code, same as that in the server, works.
If you put the Z back in it fails. I know the solution but it won't be ready for a day or two. I need to change LocalDateTime to ZonedDateTime.
A work around is to use the time zone string provided in the field named 'timezone' in the JSON output.
I'll notify this forum when it's fixed.
This code, same as that in the server, works.
Code: Select all
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
String now = LocalDateTime.now().format(formatter);
System.out.println(now);
A work around is to use the time zone string provided in the field named 'timezone' in the JSON output.
I'll notify this forum when it's fixed.
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
@scott
I may have a fix. Please try it and if it is working, I will officially release it.
for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.
I may have a fix. Please try it and if it is working, I will officially release it.
for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.
Re: Using a Pi to Log Data in a Magnum Energy System
Trying to test now... Oddly I'm getting the following error in the server log, and I have no idea why yet... I've not seen that error before and I've rebuilt and installed 100's of times.CharlesGodwin wrote: ↑Sun Nov 18, 2018 7:46 pm@scott
I may have a fix. Please try it and if it is working, I will officially release it.
for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.
Starting server Sun Nov 18 16:56:03 CST 2018 1.0.6 options:
Error: Could not find or load main class MagnumServer
Scott...
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
You said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?
Re: Using a Pi to Log Data in a Magnum Energy System
actually building from source. Using your ANT process. Thats been working fine for 1.0.5 (Using this process to compile my plugin into the main jar files)CharlesGodwin wrote: ↑Sun Nov 18, 2018 11:02 pmYou said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?
Scott..
-
- Posts: 87
- Joined: Wed Aug 22, 2012 9:57 pm
- Location: Ottawa Canada
Re: Using a Pi to Log Data in a Magnum Energy System
I have not tested that way for a while. I use eclipse which is slightly different. I'll look at it later. I'm not near my PC.