Using a Pi to Log Data in a Magnum Energy System

[sagreen83]

Are you using the latest version. I made some changes in the last 2 days

I am not. still using the previous version.

[CharlesGodwin]

@Scott. glad its working. I'd really like a copy of the txt file generated when you run the testrs485 script as I'm trying to fine tune the packet processing and every site seems to have different packet "features" so it's a learning process. Please let me know what equipment you have in the network too.

Thanks
Charles

testRS485-20180917_1920.zip

@Charles

See attached file created from testrs485

Looks like there are some unknown's in the file. such as:

Bytes read:21:UNKNOWN=>0005140A6419280601EB9B820005000000000000D0
Bytes read:22:UNKNOWN=>400001F8000577000100331E312A73000002025800FE

Again, my configuration is

Magnum MS4448PAE
MagWeb interface
BMK
AGS (w/Honda EU7000is generator)
ME-ARC
Outback FlexMax 80 charge controller
12 VmaxTank 155slr Batteries (420ah)
6 300 watt 72 cell solar panels.

Thanks,
Scott...

There's no attached file, perhaps you should email ti to me. My address isin the documents I have posted
Got the file. I noticed it hiding in the forum.

Perhaps if you used a current version of the program you would not need to report problems that have already been fixed.

[CharlesGodwin]

@scott
I reran your data using the latest parsing code and all unknown packets are identified except the first packet, which will almost always be a throw away as reading begins in the middle of an unknown conversation.

although I identify a type D0 remote packet now, I have no idea what it is as Magnum Energy has not published specs for the type. Or I haven't seen the document yet. If anyone has it, let me know.

[sagreen83]

Making progress. We've recreated the 3 graphs from the normal MagWeb site... its a work in progress, but I'm excited to get some more functionality than we have with magweb!

solar monitor graph.png (85.48 KiB) Viewed 7779 times

[sagreen83]

Original Magweb Graphs

original magweb.png (26.75 KiB) Viewed 7779 times

[CharlesGodwin]

I have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution/tags

Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.

[sagreen83]

I have moved all distribution to GitLab for easier support in the future. All previous links to my distribution folders have been disabled.
You can access the runtime software at this link. https://gitlab.com/Magnum_Energy/distribution

Currently I am not distributing the source but it too will be added to GitLab as soon as I get it setup.

@CharlesGoodwin,

Awesome work! Thanks for all that you have done to make this happen!

Scott...

[CharlesGodwin]

Making progress. We've recreated the 3 graphs from the normal MagWeb site... its a work in progress, but I'm excited to get some more functionality than we have with magweb!

solar monitor graph.png

I don't want this topic to branch into a discourse on statistics and graphing but.....

I just came across https://grafana.com/ that looks interesting.

[CharlesGodwin]

I am ready to publish release 1.0.0 but I am asking for input before I do so.

If anyone has installed and tested this software and has NOT contacted me I'd like to know of any problems or requests.

I plan to publish this week. The only changes since my last test release are:

• Allow override of text for Date field
• Allow override, using a filter, of JSON field names
• Provide access, via GitLab, to my entire Eclipse project used to develop this software

Note: The exact link to access the the release software may change but I'll clarify that when I publish.

Future Plans:

• Support to generate a specific MySQL schema for a site (using filter or no filter)
• Support to log MySQL INSERT statements to a file
• Support for logging directly to a MySQL database. (This will take a little longer)
• Any requests?

Read this part if you have customized any of the projects code

If you have been using the software without modification you don't need to read any further.

I have decided to review the architecture of the product before it goes to release 1.0.
I will be making these changes:

• Enhancing time stamp logic to allow setting the servers timezone as some statistical software like UTC/GMT timezone data.
• Adding timezone as a immutable field in the data
• Stripping all enhanced formatting from server. Everything to do with CSV, SQL and formatting will be offloaded to the client access code

If you have been customizing my code, let me know so I can help you understand the changes.

[CharlesGodwin]

__The deed is done!__

The software has been significantly simplified since earlier versions. It is now just a REST website. Going to the server's URL (i.e. http://localhost:19450) will return a JSON file of all the data points in the Magnum Energy network.

All of the value added processing is now handled by client side utilities with examples implemented in command line PHP.. The plugins interface has been simplified to make adding data providers easier but since it is now just a curl interface the ME server can be combined with separate programs to generate a log record without doing any Java work.

Anyone who worked with the pre-release versions should review the current software and reference manual as much has changed.

I hope this can be useful to some of you.

Thank you to David, Lee and Yam for help during this project.

Always take the newest release from here: https://gitlab.com/Magnum_Energy/distribution/tags

Charles Godwin

[sagreen83]

Charles,

I just tried to hit the new link, and im getting a 404 error. Seems the link doesnt exist yet.

Scott...

[CharlesGodwin]

Standby. It's an access control problem. i'm new to Gitlab so I'm learning.

[CharlesGodwin]

Fixed. Try it now.

[CharlesGodwin]

I have release a revised version of the server software. There was a bug that caused it to fail when processing PT-100 packets.

It is available here: Take the newest version

https://gitlab.com/Magnum_Energy/distribution/tags

If you have an RSS news reader, add this to get updates of any new release.
https://gitlab.com/Magnum_Energy/distri ... ormat=atom

[sagreen83]

Plugins
Adding extra "device" data is straight forward but not yet documented. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is dynamically loaded at run-time using the plugins= option. Your can have multiple classes by coma delimiting their class names in the option. This class is responsible for returning an ArrayList of objects that implement ca.godwin.magnum.DeviceRecord. How you collect the data in these objects is implementation specific. For example my MorningStar controller has a ModBus interface so I communicate using their published interface, extract the data I want, populate the object so the two methods provide data and I'm done. I also have a class that returns a weather record with current temperature, pressure and humidity. The values are appended after the magnum fields in the order that the plugins are defined and order of each DeviceRecord object in the ArrayList.

Charles,

I'm getting close to having a REST server ready that will read data from my charge controller. Any chance that I could get a copy of the java that you used for your Morningstar plugin so that I can use it as a start for my charge controller reader?

Thank you!
Scott...

[CharlesGodwin]

@scott

Yes, I can send you the Morningstar java. Just send me an email and I forward it to you. I do not want to publish it.

But....
I think that code will be overkill as it deals in painful detail with extracting data from a ModBus interface. If you're looking for an example of inserting some extra JSON data into the JSON generated by my reader I recommend you look at the code in JSONReader.java in the GitLab repository (https://gitlab.com/Magnum_Energy/distri ... win/magnum) as this code is a simple process of: read a JSON record, stuff it in the stream, repeat. Changing the file reader to a REST call would be straightforward.

If you want the Morningstar stuff, just email. You can find my email address in the GitLab project GETTING_STARTED.md doc.

[sagreen83]

Charles,

My downstream system needs the date to be in the format of...

2001-07-04T12:08:56.235-0700

So, per the directions for SimpleDateFormat Im setting the line in magnum_server.options as follows.

dateformat=yyyy-MM-dd'T'HH:mm:ss.SSSZ

When I make this change the server is returning nothing on port 19450. Neither the 'T' or the .SSSZ parts of the format string work.

Thoughts?

Scott...

[CharlesGodwin]

Thoughts

1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream

I'll test tomorrow when I'm near my system

[sagreen83]

Thoughts

1 does the server work if you don't override dateformat
2 are there any messages in the server log file
3 have you considered reformating the date downstream

I'll test tomorrow when I'm near my system

1 : The server works fine with the default dateformat (without override)
2 : No messages show up in the log file indicating a problem
3 : Haven't thought about reformating the downstream system. I was hoping to be able to deal with this server side.

[CharlesGodwin]

I have tested and the problem is the 'Z'. It works, in a test setup, without the Z.

This code, same as that in the server, works.

Code: Select all

DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS");
String now = LocalDateTime.now().format(formatter);
System.out.println(now);

If you put the Z back in it fails. I know the solution but it won't be ready for a day or two. I need to change LocalDateTime to ZonedDateTime.

A work around is to use the time zone string provided in the field named 'timezone' in the JSON output.

I'll notify this forum when it's fixed.

[CharlesGodwin]

@scott

I may have a fix. Please try it and if it is working, I will officially release it.

for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.

[sagreen83]

@scott

I may have a fix. Please try it and if it is working, I will officially release it.

for now, go to this branch page, read the instructions titled Update without internet, restore older version or install pre-release version, download the two Release_1.0.6 files, run setup, and test.

Trying to test now... Oddly I'm getting the following error in the server log, and I have no idea why yet... I've not seen that error before and I've rebuilt and installed 100's of times.

Starting server Sun Nov 18 16:56:03 CST 2018 1.0.6 options:
Error: Could not find or load main class MagnumServer

Scott...

[CharlesGodwin]

You said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?

[sagreen83]

You said "rebuild" are you really building from source or downloading the files and running setup_1.0.6 or...?

actually building from source. Using your ANT process. Thats been working fine for 1.0.5 (Using this process to compile my plugin into the main jar files)

Scott..

[CharlesGodwin]

I have not tested that way for a while. I use eclipse which is slightly different. I'll look at it later. I'm not near my PC.