sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Sep 18, 2018 12:33 am

CharlesGodwin wrote:
Tue Sep 18, 2018 12:32 am
Are you using the latest version. I made some changes in the last 2 days
I am not. still using the previous version.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Sep 18, 2018 3:44 pm

sagreen83 wrote:
Tue Sep 18, 2018 12:30 am
CharlesGodwin wrote:
Sat Sep 15, 2018 11:09 am
@Scott. glad its working. I'd really like a copy of the txt file generated when you run the testrs485 script as I'm trying to fine tune the packet processing and every site seems to have different packet "features" so it's a learning process. Please let me know what equipment you have in the network too.

Thanks
Charles
testRS485-20180917_1920.zip

@Charles

See attached file created from testrs485

Looks like there are some unknown's in the file. such as:


Bytes read:21:UNKNOWN=>0005140A6419280601EB9B820005000000000000D0
Bytes read:22:UNKNOWN=>400001F8000577000100331E312A73000002025800FE

Again, my configuration is

Magnum MS4448PAE
MagWeb interface
BMK
AGS (w/Honda EU7000is generator)
ME-ARC
Outback FlexMax 80 charge controller
12 VmaxTank 155slr Batteries (420ah)
6 300 watt 72 cell solar panels.

Thanks,
Scott...
There's no attached file, perhaps you should email ti to me. My address isin the documents I have posted
Got the file. I noticed it hiding in the forum.

Perhaps if you used a current version of the program you would not need to report problems that have already been fixed.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Sep 18, 2018 9:19 pm

@scott
I reran your data using the latest parsing code and all unknown packets are identified except the first packet, which will almost always be a throw away as reading begins in the middle of an unknown conversation.

although I identify a type D0 remote packet now, I have no idea what it is as Magnum Energy has not published specs for the type. Or I haven't seen the document yet. If anyone has it, let me know.
Attachments
testRS485-20180918_1711.txt.gz
(1.35 KiB) Downloaded 34 times

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Sep 19, 2018 2:07 am

Making progress. We've recreated the 3 graphs from the normal MagWeb site... its a work in progress, but I'm excited to get some more functionality than we have with magweb!
solar monitor graph.png
solar monitor graph.png (85.48 KiB) Viewed 2110 times

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Sep 19, 2018 2:21 am

Original Magweb Graphs
original magweb.png
original magweb.png (26.75 KiB) Viewed 2110 times

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Change to distribution

Fri Sep 21, 2018 5:08 pm

I have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution/tags

Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.
Last edited by CharlesGodwin on Sat Sep 22, 2018 4:50 pm, edited 1 time in total.

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Change to distribution

Fri Sep 21, 2018 8:55 pm

CharlesGodwin wrote:
Fri Sep 21, 2018 5:08 pm
I have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution

Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.
@CharlesGoodwin,

Awesome work! Thanks for all that you have done to make this happen!

Scott...

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Fri Sep 21, 2018 9:22 pm

sagreen83 wrote:
Wed Sep 19, 2018 2:07 am
Making progress. We've recreated the 3 graphs from the normal MagWeb site... its a work in progress, but I'm excited to get some more functionality than we have with magweb!

solar monitor graph.png
I don't want this topic to branch into a discourse on statistics and graphing but.....

I just came across https://grafana.com/ that looks interesting.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Sep 23, 2018 3:40 pm

I am ready to publish release 1.0.0 but I am asking for input before I do so.

If anyone has installed and tested this software and has NOT contacted me I'd like to know of any problems or requests.

I plan to publish this week. The only changes since my last test release are:
  • Allow override of text for Date field
  • Allow override, using a filter, of JSON field names
  • Provide access, via GitLab, to my entire Eclipse project used to develop this software
Note: The exact link to access the the release software may change but I'll clarify that when I publish.

Future Plans:
  • Support to generate a specific MySQL schema for a site (using filter or no filter)
  • Support to log MySQL INSERT statements to a file
  • Support for logging directly to a MySQL database. (This will take a little longer)
  • Any requests?
Read this part if you have customized any of the projects code

If you have been using the software without modification you don't need to read any further.

I have decided to review the architecture of the product before it goes to release 1.0.
I will be making these changes:
  • Enhancing time stamp logic to allow setting the servers timezone as some statistical software like UTC/GMT timezone data.
  • Adding timezone as a immutable field in the data
  • Stripping all enhanced formatting from server. Everything to do with CSV, SQL and formatting will be offloaded to the client access code
If you have been customizing my code, let me know so I can help you understand the changes.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 01, 2018 12:01 pm

__The deed is done!__

The software has been significantly simplified since earlier versions. It is now just a REST website. Going to the server's URL (i.e. http://localhost:19450) will return a JSON file of all the data points in the Magnum Energy network.

All of the value added processing is now handled by client side utilities with examples implemented in command line PHP.. The plugins interface has been simplified to make adding data providers easier but since it is now just a curl interface the ME server can be combined with separate programs to generate a log record without doing any Java work.

Anyone who worked with the pre-release versions should review the current software and reference manual as much has changed.

I hope this can be useful to some of you.

Thank you to David, Lee and Yam for help during this project.

Always take the newest release from here: https://gitlab.com/Magnum_Energy/distribution/tags

Charles Godwin
Last edited by CharlesGodwin on Thu Oct 18, 2018 5:46 pm, edited 5 times in total.

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 01, 2018 6:52 pm

Charles,

I just tried to hit the new link, and im getting a 404 error. Seems the link doesnt exist yet.

Scott...

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 01, 2018 7:11 pm

Standby. It's an access control problem. i'm new to Gitlab so I'm learning.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 01, 2018 7:14 pm

Fixed. Try it now.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Oct 02, 2018 8:53 pm

I have release a revised version of the server software. There was a bug that caused it to fail when processing PT-100 packets.

It is available here: Take the newest version

https://gitlab.com/Magnum_Energy/distribution/tags

If you have an RSS news reader, add this to get updates of any new release.
https://gitlab.com/Magnum_Energy/distri ... ormat=atom

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 22, 2018 12:20 pm

CharlesGodwin wrote:
Sun Sep 16, 2018 8:29 pm

Plugins
Adding extra "device" data is straight forward but not yet documented. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is dynamically loaded at run-time using the plugins= option. Your can have multiple classes by coma delimiting their class names in the option. This class is responsible for returning an ArrayList of objects that implement ca.godwin.magnum.DeviceRecord. How you collect the data in these objects is implementation specific. For example my MorningStar controller has a ModBus interface so I communicate using their published interface, extract the data I want, populate the object so the two methods provide data and I'm done. I also have a class that returns a weather record with current temperature, pressure and humidity. The values are appended after the magnum fields in the order that the plugins are defined and order of each DeviceRecord object in the ArrayList.
Charles,

I'm getting close to having a REST server ready that will read data from my charge controller. Any chance that I could get a copy of the java that you used for your Morningstar plugin so that I can use it as a start for my charge controller reader?

Thank you!
Scott...

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Oct 22, 2018 1:37 pm

@scott

Yes, I can send you the Morningstar java. Just send me an email and I forward it to you. I do not want to publish it.

But....
I think that code will be overkill as it deals in painful detail with extracting data from a ModBus interface. If you're looking for an example of inserting some extra JSON data into the JSON generated by my reader I recommend you look at the code in JSONReader.java in the GitLab repository (https://gitlab.com/Magnum_Energy/distri ... win/magnum) as this code is a simple process of: read a JSON record, stuff it in the stream, repeat. Changing the file reader to a REST call would be straightforward.

If you want the Morningstar stuff, just email. You can find my email address in the GitLab project GETTING_STARTED.md doc.

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 1:58 am

Charles,

My downstream system needs the date to be in the format of...

2001-07-04T12:08:56.235-0700

So, per the directions for SimpleDateFormat Im setting the line in magnum_server.options as follows.

dateformat=yyyy-MM-dd'T'HH:mm:ss.SSSZ

When I make this change the server is returning nothing on port 19450. Neither the 'T' or the .SSSZ parts of the format string work.

Thoughts?

Scott...

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 2:03 am

Thoughts

1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream

I'll test tomorrow when I'm near my system

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 2:08 am

CharlesGodwin wrote:
Sun Nov 18, 2018 2:03 am
Thoughts

1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream

I'll test tomorrow when I'm near my system
1 : The server works fine with the default dateformat (without override)
2 : No messages show up in the log file indicating a problem
3 : Haven't thought about reformating the downstream system. I was hoping to be able to deal with this server side.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 5:05 pm

I have tested and the problem is the 'Z'. It works, in a test setup, without the Z.

This code, same as that in the server, works.

Code: Select all

DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
String now = LocalDateTime.now().format(formatter);
System.out.println(now);
If you put the Z back in it fails. I know the solution but it won't be ready for a day or two. I need to change LocalDateTime to ZonedDateTime.

A work around is to use the time zone string provided in the field named 'timezone' in the JSON output.

I'll notify this forum when it's fixed.

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 7:46 pm

@scott

I may have a fix. Please try it and if it is working, I will officially release it.

for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 10:57 pm

CharlesGodwin wrote:
Sun Nov 18, 2018 7:46 pm
@scott

I may have a fix. Please try it and if it is working, I will officially release it.

for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.
Trying to test now... Oddly I'm getting the following error in the server log, and I have no idea why yet... I've not seen that error before and I've rebuilt and installed 100's of times.

Starting server Sun Nov 18 16:56:03 CST 2018 1.0.6 options:
Error: Could not find or load main class MagnumServer

Scott...

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 11:02 pm

You said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?

sagreen83
Posts: 22
Joined: Wed Sep 05, 2018 2:54 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 11:05 pm

CharlesGodwin wrote:
Sun Nov 18, 2018 11:02 pm
You said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?
actually building from source. Using your ANT process. Thats been working fine for 1.0.5 (Using this process to compile my plugin into the main jar files)

Scott..

CharlesGodwin
Posts: 60
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Nov 18, 2018 11:10 pm

I have not tested that way for a while. I use eclipse which is slightly different. I'll look at it later. I'm not near my PC.

Return to “Automation, sensing and robotics”