Using a Pi to Log Data in a Magnum Energy System

[sagreen83]

I have not tested that way for a while. I use eclipse which is slightly different. I'll look at it later. I'm not near my PC.

Ok, I just installed the normal (non build) way. If I add .SSSZ to the date format, as follows, the webserver does not return results (must be crashing somewhere in the json build).

dateformat=yyyy-MM-dd HH:mm:ss.SSSZ

If I drop the SSSZ it works fine. I'm not seeing anything in the server log, and on a hope I added trace=true to the server.options file but that didnt produce any more output.

Scott...

[CharlesGodwin]

Strange. I did that earlier and it worked. The old code worked by just dropping the Z.

I'm still not near my PC

[sagreen83]

Strange. I did that earlier and it worked. The old code worked by just dropping the Z.

I'm still not near my PC

No hurry. The new code works by dropping the Z as well. so if i do .SSS it works. Its the Z that is killing it. Hoping I'm not confusing things. I assumed that you were trying to fix the Z (timezone) part of dateformat.

Scott...

[CharlesGodwin]

@scott

I am confused as all my tests with release 1.0.6 supports the Z. This morning I did the following:

Note: There are some changes to prior build instructions. I will list them at the bottom. I am also attaching a log of all these activities.

• I created a new directory on a Pi
• I installed ant and a Java Development Kit (JDK) as all I had previously was just a runtime.
• I downloaded a zip of the source for Release_1.0.6 branch from gitHub
  https://gitlab.com/Magnum_Energy/distri ... _1.0.6.zip
• I deleted 2 files that should not be in the zip file(No longer needed)
• I followed build instructions in BUILD_INFO.md
• I created a new test directory
• I copied the two files needed for an install into the test directory
• I ran the setup program
• Because I have no Magnum system nearby I modified the magnum_server.options file to reference a mock json file and I copied the .json file into the folder
• I started the server with an override of the dateformat

  Code: Select all

  ./startserver dateformat="yyyy-MM-dd'T'HH:mm:ss.SSSZ"

• I got the expected result from the server using

  Code: Select all

  curl localhost:19450

  . The full record is in the log but it starts with {"Date":"2018-11-19T09:05:49.373-0500","timezone":"-05:00"
• I then modified the options file to include dateformat=yyyy-MM-dd'T'HH:mm:ss.SSSZ setting instead of using the command line override
• I restarted the server and got the same result.

I'm not sure where your problem is. I can only guess. Did you copy the pair of files for an install to your target directory (magnum_xxxx.zip and setup_xxxx) and then run the setup_xxxxx script? This is important as it modifies some jar file names in the lib directory. Can you send me a directory listing of the lib folder? "ls -l lib"

What is 'wrong' with the zip file?

The interim release source zip file contains two files that should NOT be included. Once you have unzipped the source files delete these two files before the first ant step.

As of 2018/11/19 10:20 Eastern these files were removed from the zip file.

The document BUILD_INFO.md has been revised to include how to install ant and JDK for Pi systems and some refinements to the steps. The fundamentals remain unchanged.

I will remove the two extra files before the official release.

[CharlesGodwin]

@scott

I notice that building extra plugins can be complicated if you need to checkout the entire source, add your code and then build a private version. That's a lot of work.

My method is to have a separate project for just compiling the extra plugin code. I then build a jar file with a different name containing just that code. I then place that jar into the pi lib directory and all is good. I only need to rebuild this jar when my code changes.

It would not be too much effort to create an ant build file and short document to support this. Is anyone interested?

[sagreen83]

Thats interesting... I just ran the same commandline as you did.

./startserver dateformat="yyyy-MM-dd'T'HH:mm:ss.SSSZ"

I get the following in the logger log...

2018-11-29 05:53:55 Error encountered:file_get_contents(http://localhost:19450/): failed to open stream: HTTP request failed!
2018-11-29 05:53:55 Sleeping for 60 seconds

I'll try to do this with the process you described in your message.

Scott...

[CharlesGodwin]

I hope its' something simple.
Before you rebuild and replace I suggest you delete all your magnum*.jar files in the lib directory so there are no old rogue versions getting in the way.

Code: Select all

rm lib/magnum*.jar

If you hit glitches please email me as my responses can be quicker and more specific. Thanks

[CharlesGodwin]

This is addressed to anyone using my software.
I haven’t published any changes to my software but I plan a cleanup. More like pruning dead branches. Take a look at this https://gitlab.com/Magnum_Energy/distri ... FOR_2.0.md and tell me if there are any problems. I’m trying to make it simpler to use, maintain and document.

If you are using my software please email me so I know who is out there when I plan changes.

magnum-at-godwin-dot-ca

[RDK]

Charles....It has been a while since I communicated on this subject. My system, which spies on the MAG-Web communications, has been working just fine, although it does stop working, as documented, when we lose internet. Now I would like to look into setting up your system on a different Pi.

We have a MS4448PAE inverter, ME-ARC, and the ME-BMK, and the Outback Flexmax 80 MPPT charge controller with a ASX-Port. The ASX-Port allow us to remotely monitor and modify the charge controller parameters.

Will your system see the data for the Battery and Inverter units and also the Outback charge controler's data?

By the way, you've done an excellent job of developing and documenting your work - congrats......RDK

[CharlesGodwin]

Hi

I hope my system can help.

The software as published will handle all your Magnum Energy (ME) components but, alas, not the Outback Energy ASX-Port.

I presume, since you have managed to deal with the ME magweb device, that you have programming skills. I don't think it would be hard to extend the reader to support the ASX-Port but I have no experience. I have an unpublished java extension that reads the ModBus interface of my MorningStar charge controller and provides the data as part of the JSON record produced. A quick Google search for the Outback interface reveals that it uses a semi-public protocol called Sunspec (https://sunspec.org). They officially publish a Python library and I see there are various third-party Java implementations.

In this document https://gitlab.com/Magnum_Energy/distri ... ASERVER.md I discuss alternative ways to building extensions to the server, either as a Java plugin or using a wrapper. I have no opinion which would be preferable in your situation. I provide an example plugin written in Java that can be a model. Read this: https://gitlab.com/Magnum_Energy/distri ... N_BUILD.md

If you would like detailed help write me: magnum-at-godwin-dot-ca.

Thanks for the kind words.

[RDK]

Charles....OK, thanks. I will let you know if we decide to got head with this idea.....Rob

[owen_a]

Ah, I've just come across this post as I lost the code to communicate with my TS-MPPT-60. Charles, you sent me the code a few years ago and the software worked flawlessly. Any chance this is in git?

Also, I'm having trouble understanding the documentation for the Modbus protocol for the Morningstar's, any chance you could please explain how one can make a call to the controller for, say, battery voltage? As in, the PDUAddr on the table in that PDF, the battery voltage is 0x0018. The logical address is 25. From what I can see online, you give a starting address and a quantity, whatever this means...

Just need someone to shed some light onto this.

Thanks,
Owen

[CharlesGodwin]

The Morningstar code is not on Gitlab - yet. I hadn't considered it as no one asked. But I can share if you send an email to magnum-at-godwin-dot-ca. I'll look for your email.

The code is a bit different that what I would have sent you before. The difference is not in reading the ModBus information but in the way the data is sent back to you. The messaging has changed. I developed the software for the data reader to be modular and it will support multiple, all optional, readers. I ship it with a magnum device reader but I have developed the Morningstar reader as well as a temperature sensor reader. In you case, you could use it with ONLY the morningstar reader.

Concerning how to read the data using modbus. Each read request is to an address and the quantity is "how many consecutive registers do you want?" For example a request at logical address 25 for a quantity of 4 would return the values at 25,26,27, and 28.

The confusing part with the morningstar data is that they have a few registers with base multiplier values. These registers must be read, massaged and then used as multipliers to calculate true values in other registers.

[CharlesGodwin]

I'm going to try to explain getting battery voltage from MorningStar. It's not simple and I fear my language skills will get in the way.

Voltages, as I mentioned in a prior post, are based on several values. Referring to page 3 of the document you referenced.
A voltage value( adc_vb_f_med) is the value in register at 0x0018 (25) multiplied by voltage scale multiplier (V_PU) multiplied by 2 to the power of -15. This won't copy and paste.
What is V_PU you ask? It is the float value created by using the value in register 0x0000 (1) as the whole part and the value in 0x0001 (2) as the fractional part. In Java I use this to calculate V_PU

Code: Select all

v_pu =  vpuhi + (float) (vpulo / 0x8000);

vpuhi is the the whole part and vpulo is the fractional part.

so voltage adc_vb_f_med is

Code: Select all

adc_vb_f_med  = (float) (value at 0x0018) * v_pu / 0x8000;

Similar formulae for amps.

May not be pretty, may not be the best code, but it worked so I stopped revising it.

[owen_a]

Hi Charles,

Thank you for getting back to me (I sent you an email also).

Thank you for trying to explain this to me, that does not seem easy at all haha! I was under the impression it would return me the value as is.

- Owen

[CharlesGodwin]

It's not as complicated as it sounds. I have a small function that does all the fiddly calculations and it's also code that's forgotten once it works.

[dogman1155]

I have a Magnum system with two MS4448PAE inverters and the BMK battery monitor.
I just recently upgraded from a single inverter to a dual inverter system. In doing so I had to
replace the ME-ARC monitor for a ME-ARTR. I was expecting to see both inverters in the data
stream but i only see one. I am able to get both by connecting to both inverters network port.

I do not have a mag-web but would assume that some how it gets both inverter streams or
at least a total for the two so that it can report to the magnum web page.

I had to read both streams and add the totals, I was hoping for a better way.
Has anyone with a mag-web captured the data stream so that what I see can be confirmed ?
Has anyone setup a dual inverter system and captured the data stream ?

I have the old original copy of the Magnum protocol document, I wonder if there is a newer
one somewhere that can be had ?

I also have two morningstar TSMPPT60 charge controllers that am collecting data from
using the documented modbus, I see that Charles has found an undocumented way to
get the data from the mornningstar controllers using java, would it be possible to get that info ?

Howard

[CharlesGodwin]

I have limited experience with multiple inverter setup. I had one user of my software https://gitlab.com/Magnum_Energy/distribution that had a multi-inverter and multi PT-100 setup but due to circumstances outside the scope of this software he was unable to implement and validate the software. However he did have contact with Magnum Energy and I was able to indirectly ask some questions about reading multi device data. The answer I got, again indirectly, was that the ARTR and inverters are supposed to work this all out between themselves and display the net result. Also I was told the ARTR sent similar packets to the RTR. But this is third hand information.

I assume you have set up the ARTR as shown in the user's manual, page 4, Figure 2-1, System Overview. I recommend you look at data from communications port 5 or port 6 or perhaps the remote port on the slave inverter.

You talk of "data stream". what do you mean by this? Are you using my Magnum reader soft ware or something else?

The latest documentation I have from Magnum is dated May 2013 but does not include the ARTR. The document name that I have is "Magnum network protocol 7-13.pdf". I was given to me from an outside source so the official file name may be different.. I was asked not to redistribute it. If anyone has a newer version I would be very grateful for a copy.

You should verify this with Magnum energy but I think the Mag-Web interface does NOT support multiple inverters.

Re the Morningstar controllers. I am not using an undocumented way of extracting data. I used the Morningstar published modbus documentation. The code I use is published in my software project, referenced above, in the examples folder. It is posted as an example as I am not able to give users significant support. The code only extracts data points that interested me at the time I built the software. The controller has many, many data points and I ignore most of them. I can suggest solutions to puzzles and I try to correct any bugs I detect. I run my system with a slightly different version that I do not publish.

Using two controllers introduces an interesting problem of identifying data points. The software, as written, does not consider the possibility of more than one controller. You would have to decide what points are common, what points are unique and discretely labelled and which points to ignore. That is mostly a design decision and some code changes, not a technical problem.

[dogman1155]

Charles thanks for the reply.

I may have an older version of the magnum protocol document date Oct-15-09.
I wrote my software in C back in 2010-2011 when I setup my system and it worked fine until
I replaced the ME-ARC, version 2.2, controller, my software was too strict and required one of each
packet type before it would return anything and the new ME-ARTR, version 4.x, doesn't send an A3 or A4 message.
I think it only sends them when it finds the ME-AGS attached to the system but not sure.

Once I got that fixed I installed inverter number two and I looked at data on all ports.
They have made substantial changes to the protocol, extra packets and removed some.
I expected to see everything on port 5 or 6 as you suggested but I did not.
Also it doesn't seem to send a 91 RTR packet anymore.

I did find that connecting to the remote or network port on each inverter gives all packets
except the inverter packet for the other connected inverter. At this time I have a serial connection
to both inverters and on the one with slave setting I only grab the inverter packet, all the
rest are duplicate of what comes out of the other inverter. Also most of the data in the inverter
packet is repeated since the data mostly comes fro the same source. I add up the watts and amps
from all inverters for the total.

I tied talking to Magnum and the message from them was that the protocol is proprietary and they would
not help me. I like there equipment but not there support, knowing what I know now I may have started with
other equipment, but it's too late now and the hardware seems to be solid.
The first inverter and BMK has been running 24x7 since late 2010 supplying about 90% of my needs.
The book and Magnum support told me my ME-ARC can be attached to port 5/6 of the ME-ARTR and
it should work, but so far it seems incompatible with the newer firmware. The ME-ARTR won't even see
it connected as it should, I have hope that if I can get it working maybe all the packets may show up there.

By your comments about using undocumented java I thought maybe you had done what I tried, looking
at the html/javascript from the morningstar controller and extracting what they do. I almost got it working
but it was complaining about cross domain, CORS, security issues.

After that did not work I wrote code using C and the modbus as documented, I sometimes get errors
that the connection was refused because I have more than one app getting data at the same time.
I am going to make a design change to make a server/client to fixed that.

I took a quick look at your software but I have not learned java and I prefer pure C code anyway.
It was useful to determine that C0, C1, D0, 11 etc packets that i was seeing were in fact real but
not useful to me. These were not in my document and must be new with the ME-ARTR.
At some point when i have time I will setup a spare raspberry pi and try your software.
I wish I had found it way back when I first started.

Howard.

[CharlesGodwin]

It's a pity that Magnum is so secretive about their protocol. There are days when I wonder if they are concerned about trade secrets or just embarrassed about its design or lack of design. It is a very rigid protocol that is hard to expand. In fairness to them it was developed in early days of modern instrumentation and seemed to have been developed for just the matched pair of an inverter and remote. But their subsequent decisions when the world changed and they wanted more devices included are questionable. Then they developed an interface device, the MagWeb, that relies on an always on internet interface for an off grid product? What were they thinking. That's when I started my work. I needed to know when my generator ran as my system is at a cottage and I was then only a weekend visitor. The arrival of the Raspberry Pi is when I started work.

Currently my code is the only public code I've found that tries, mostly successfully, to read and extract each packet discretely. I had to revise it recently to support a RS495 HAT device that behaves a little differently than the RS485/USB adapters. This new code is not mainstream yet as it's still being tested. It is available in my repository in a branch named Revise_RS485_Reader and is much more fussy about what a packet is. If you want to know what all the packet type are, that I know about, start in the program MagnumPacket.java. This program dissects the packets to determine type. Program MagnumReader.java dispatchers the packets to the appropriate program for translation.

I have no passionate loyalty to Java. I have written code with C, C++ and many other languages. It was chosen as I was working with it professionally at the time and then the Pi didn't have much of a development infrastructure. f you or someone else can learn from my work and develop it in another language then go for it. I have days when I wonder if a rewrite in Python would be a good idea but it is a lot of labour and there is not much demand.

The newest ARC remotes are compatible with most of the same packets as the ARTR. The biggest change is the addition of the PT-100 charge controller.

I get read/connection errors with my Morningstar controller regularly but I have never detected a pattern. I've just written resilient code that reports but survives the error and goes on in the next log cycle. There is new firmware for the controller dated 2018 but I haven't installed it yet. It's on my "B" list.

I may not have been helpful when you started out. Although I started this thread in 2014 I only published a version of the software in 2018. No one asked before then. Making it public was an interesting exercise as I had to knock off all the sharp edges.

[dogman1155]

I wrote my code for just the same reason you did although my system has grid power, I did not want to depend on it since I
also am a ham radio operator/experimenter and wanted to have power even if the grid goes away. The system started out
just to power my radios and ended up powering the whole house. I do not always have internet and don't want to rely on or
be held ransom by someone else's server. I also have integrated network switchable AC plug so I can turn thing on/off
remotely when I have internet and I have automatic transfer switches for the computer and refrigerator circuits.
The local electric company is only there as a backup if the solar should fail or we have too many non solar days.

When I looked at the mag-web I was just not that impressed with it and if as was suggested it will only handle one
inverter, than that makes sense with what I see on the network and I am very glad i did not invest in one.
My original code was build using windows and Microsoft studio so I was very excited about the raspberry when it came
out and I have one or more of each version from the beginning. I have found for the last few years I hardly ever turn on
my big monster windows computers anymore except my laptop. I use the raspberry as a desktop and have more than one
that runs 24x7 for many uses. I am going to order the new raspberry 4 soon.

There is some public python code on the web that I just found last year, but I don't know python either and it did not show
anything I did not already know. I do have some RS485 serial chips and little RS485 boards that I plan to try later.
I did not expect that it would be much change to the software. At this time I have USB to serial adapters and then serial RS232
to RS485 adapters. It was just stuff I had around already, I sure wish I had found the adapters that you show way back then.
It kinda started out as a proof of concept and just grew LOL. I think there may be more interest than you may think
based on the number of responses to this thread, I have seen many people asking question about the
Magnum system and software, google magpy, this is the python version, and you will find a fair amount of people talking about it.
That was what I was doing when I found this thread and was delighted to find someone else who has made progress with it
and especially sense it involved a raspberry. I was hopping to find a newer version of the protocol document that might have
clues about the dual inverter and some of the packet types that I was seeing. They seem to have pulled the document
from the web, I bet they want people to buy there overpriced mag-web system.

I also have not updated the Morningstar controllers, I did not want to introduce too much new all at once.
I get connection busy or refused errors and just like what you did I just go on without updating. The update
will happen soon enough anyway on the next cycle. The change log suggested that they changed/corrected
something to do with a scaling error, that is what I had to most problem with and i am some what apprehensive
about breaking my code while I have a lot of other changes going about with hardware and software.

I do the same thing with the Magnum inverters network. It updates very fast with the amount of excess data
they have on the network, I have never understood why they repeat the packets in so many different ways,
It seems to be the most convoluted protocol I have ever seen in my many years of witting and designing computer software.
What I have noticed repeatedly is that for many hardware companies, they can design and build good hardware and
software seems to be a necessary after thought that they don't seem to put the resources too.

[CharlesGodwin]

Replies may be out of order.

I only discovered the USB/RS485 dongle about 18 months ago. Before that i was using an expensive add on board that became obsolete as the vendor stopped upgrading the Pi software. I have recently added support for a RS485 HAT as the HAT behaves slightly different that the USB dongle. the code is still in beta testing . If there is urgent interest let me know, otherwise it will be published shortly.

I studied the python code and did some testing. One of my users tested too. I could not find a python solution that could handle all packets as cleanly as my java solution. I'm not saying ti can't be done, I just haven't seen it. I may write one, based on my Java code, in python later this year.

I agree the Magnum protocol is really weird. I think it started as a simple idea and then grew out of control and no-one said "wait a minute, lets do this differently". Even today I think they could build a logging/control box to clean up the interface. OutBack Power has one, a model AXS.

Although there seems to be a lot of interest in the Magnum interface, I don't have that many users of my software. I don't know the actual count as GitLab doesn't supply download statistics but based on correspondence I would guess in the 15-30 range.

[CharlesGodwin]

@dogman1155

I implemented a serial modbus version of my Morningstar controller interface. It has been running for a day with no faults. In the same interval my TCP version has had 8 faults. The only downside is the two must be near each other. That is not a problem for me.

[CharlesGodwin]

I have released a separate project that is a python implementation of a reader that is simple and straightforward. It is available here https://pypi.org/project/pymagnum/

Please let me know if it's useful.