CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Using a Pi to Log Data in a Magnum Energy System

Thu Jun 19, 2014 3:12 pm

Introduction
I have implemented a program for logging all messaging events in a Magnum Energy (Magnum) system using a Raspberry Pi Model B and a RaspiComm expansion board. This program logs all messages on the network for subsequent analysis and display. The solar system is off-grid at a summer cottage on an island, far from electricity and internet connectivity. The project started as I wanted to track all generator activity and Magnum had no supporting methodology. This logger will run unsupervised for 6-7 months of the year.
The Project
I have a Magnum MS4024PAE Inverter Charger, an ME-ARC Remote, a BMK Battery Monitor Kit and an AGS-N Automatic Generator Start module. There is also an 810 Watt solar array, a Morningstar Corporation TS-MPPT-60 charge controller and a Yamaha EF5200-DE remote start generator. The AGS detects a low State of Charge (SOC) and automatically starts the generator to top-up the batteries. The magnum units communicate with each other using a proprietary protocol on a RS-485 serial network. Mangum, at one time, published this protocol but withdrew it from public circulation a few years ago. I was able to persuade them to give me a copy with my assurances that I would NEVER contact them for support. I have lived up to that commitment. Mangum has asked me to not publish their copyrighted document.
My Approach
Magnum sells a Web Monitoring Kit that uses a continuously stay-alive internet connection to log information to their data server. The data displayed is essentially the same as the live data displayed on the ARC Remote and monthly logging just shows high – low values on a daily basis. I couldn’t use this for these reasons:
  1. I don’t have a continuous internet connection
  2. It didn’t provide any historic logging of generator events. (I have an electric refrigerator so keeping it alive is important and generator activity will tell me approximate fuel consumption and time left before service)
  3. I had just bought a Pi and was looking for a good project.
I implemented the logger using Java, a language I’m professionally comfortable using. I’m sure a similar program could be developed using C or Python, but I haven’t tried. I used Eclipse IDE on a Windows laptop for development and just copied the runtime jar file to the Pi as part of the build. I use the Pi4J library for RS-485 Serial support.
The connection from the Magnum network to the Pi is a simple phone splitter jack on the AGS unit, a 4-wire RJ-11 cable from the splitter to a standard RJ-11 wall jack beside the Pi and wiring from the jack to the RaspiComm RS-485 connector. I did not have to make any special changes to either the Magnum or the Pi configuration.
The first thing I needed to do was figure out what a “message” was. The Magnum protocol was a simple protocol of the inverter sending a status message and waiting, then the Remote would send a message with settings content pertinent to each of the other components and wait. These components would respond when there was a pause in the message traffic. The message content was a combination of facts, such as current voltage, settings (battery size) or request to alter a setting. Each unit in the network behaves autonomously. For example, the Remote will allow a user to configure the AGS to know when it will start, how long it will run, and when it can run (excluding quiet time). But the AGS makes its own decision about actually starting the generator. This means there are no command messages, just settings messages. These settings messages are sent repeatedly so the network is very resistant to missed messages and all units will do their tasks even when not connected to the network.
This message stream is continuous and there is no typical server/client request/respond behaviour. The data logger is continuously listening to a non-stop message stream and figuring out what is being said AND it can’t interrupt.
My solution was to determine the end of a message by detecting when the next message started. I was using serial blocking input so I read one character at a time. I check how long the last character took to read and if this exceeded 2 milliseconds, I assume that one message ended and a new was starting. So I would process the buffered characters, excluding the last one, as a message. This works well and I get a very high success rate of meaningful messages. To distinguish messages I rely on the documentation’s definitions and examine message length and some key character values, sometimes at the end of the message and sometimes at the head of the message. Occasionally I’ll get a garbled message that is the wrong size or has no meaningful identifying content. I discard these. My rational is that I’m logging data for later review and I am getting so much data that some losses are not important. I also ignore the first message as I have no idea when I joined the message stream so I start with my first compete message.
Here is an edited code fragment of the message read loop:

Code: Select all

long wait = 2;
char one;
while (getmore) {
  long start_time = System.currentTimeMillis();
  one = serial.read();
  interval = System.currentTimeMillis() - start_time;
  if (interval > wait || ix > 22) {
    getmore = false;
  } else {
    buffer[ix++] = (byte) one;
  }
}
This code is the core of the logger but I have wrapped this in a more complex set of routines. The pertinent features are:
  • Add a shutdown hook to enable an orderly logging of the last message and file close.
  • Discard consecutive duplicate messages. As each unit transmits the same message over and over again, I discard any consecutive identical message on a message type basis. This reduces file size.
  • I write the log to a Gzipped data stream – again for storage reduction.
  • Create a new file every hour, on the hour. I am not sure I can defend this time span decision but it seemed like a good idea at the time. I may extend the span.
  • Set the program to be started at system start-up by a crontab entry.
  • Use separate program to read and format the logged messages. This reduces processing load on the Pi. I do my reporting on the laptop.
I am currently generating about 22KB of zipped data an hour so a year’s data is less than 250MB, not a lot of disk space.
What’s Next
I have several things I plan to do in the next few months / years. A lot of the work can only be done on site as I need to interact directly with the live devices.
• Add logging of the MorningStar charge controller. This is relatively easy as it has completely documented MODBUS interface. I just have to decide what to log.
  • Review and revise my logging criteria. I probably don’t need all the information I have. I may just do a data slurp every 60 seconds. This will reduce the load on the CPU which is currently running in a tight loop.
  • Integrate the collected data in RRDTool for logging and reporting.
  • Add a web server interface so I can see what’s happing from any device at my cottage. I have a local wireless network with mobile broadband access when I’m at the cottage.
  • Add an email/ftp interface when/if I get more persistent internet access.
  • Add weather statistics
  • Add camera images
Miscellaneous Notes
This is my first posting of a personal project so I have been unsure of what exactly to include / discard. I look to feedback to help.
  • Should I post all the source?
  • The full source is available on request as an Eclipse project.
  • The Raspberry Pi and an Ethernet switch are running directly off the batteries using a set of LM2595 DC/DC buck converters. This allows logging to continue with the inverter shutdown.
  • I cannot provide the Magnum Protocol Document
Last edited by CharlesGodwin on Sat Oct 11, 2014 9:31 pm, edited 1 time in total.

salts
Posts: 13
Joined: Fri Nov 15, 2013 2:38 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sat Aug 23, 2014 7:29 pm

Hi Charles,

I was hoping you could send me your source code?

I have the Magnum Energy RD3924 with remote control panel ME-ARC and would like to implement a similar project, though I will probably use C as I have an Arduino with SD card and RTC I may use instead of the RPi.

I have also found this, http://static.us-inverters.com/docs/support/Magnum-Networking-Communications-Protocol-(2009-10-15).pdf a yes/no to the question 'is this the correct protocol?' would be nice, but if you feel it is against your agreement with Magnum Energy, don't worry. I can log the data to a serial data logger and just reverse engineer, just a bit more work.

All the best

Ian
Last edited by salts on Sun Aug 24, 2014 5:06 pm, edited 2 times in total.

CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Aug 24, 2014 12:55 pm

Ian
Please repost the link as I get 404 -not found with the link provided. It's probably the right document as it was available on their website for a while and then they withdrew it. My document includes your inverter as a supported model.

I can advise on using C to read the data. I considered C but my java skills are better, it's been a while since I used C. The major problem I had is that the devices send out continuous messages, only separated by timed pauses. And the messages are far from well structured. I ended up using the delay between each character read to determine EOM "end of message" and then examined what I had since last EOM to decide what to do with it.

I don't have a tight non-disclosure so I should be able to help.

ame
Posts: 3172
Joined: Sat Aug 18, 2012 1:21 am
Location: Korea

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Aug 24, 2014 1:39 pm

CharlesGodwin wrote:Ian
Please repost the link as I get 404 -not found with the link provided. It's probably the right document as it was available on their website for a while and then they withdrew it. My document includes your inverter as a supported model.

I can advise on using C to read the data. I considered C but my java skills are better, it's been a while since I used C. The major problem I had is that the devices send out continuous messages, only separated by timed pauses. And the messages are far from well structured. I ended up using the delay between each character read to determine EOM "end of message" and then examined what I had since last EOM to decide what to do with it.

I don't have a tight non-disclosure so I should be able to help.
Just add a closing bracket and .pdf to the URL and it will work, i.e. ").pdf"

It looks like the link was mangled a bit in the forum post.

salts
Posts: 13
Joined: Fri Nov 15, 2013 2:38 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Aug 24, 2014 4:06 pm

Hi,

User Ame is right about the link, I went to edit the link and it is correct, just won't post correctly, I have pasted below the link and document name on two separate lines, the link will take you to the index page and the document is the fifth link.

http://static.us-inverters.com/docs/support/
Magnum-Networking-Communications-Protocol-(2009-10-15).pdf

Edit: updated original link, enabling 'do not automatically parse URL'
I will be happy for any help, the source file in Java will be a great start.

From the protocol it looks as if byte 14 is always the model number and byte 15 is always zero for revisions < 3.9, revision 4 > byte 14 is still the model and byte 20 is zero, I was thinking, for no other reason than to be different, to capture 1 second of serial data, search for my model number and build the message from there, though it does look problematic, as it looks possible for the model number and a zero to appear together elsewhere in the string because of other variables.

I will probably, which ever way I go, capture 1 second of data, extract just 1 complete set of values, discarding the rest and only capture every 30 seconds, this is enough for my needs, but will probably use a config file to allow setting different values, as we are going to install a separate solar array in the near future and will probably use a Magnum Energy inverter for this as well.

For information our current system is a battery bank of 12 Trojan T105 batteries these are charged when we have shore power, which is about 8 hours per day, this means monitoring energy usage is quite important to us. Though it is true the remote panel supplies all the information to do this, it is boring!

CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Aug 25, 2014 1:12 am

My version is dated 12-16-2010, slightly newer. The one you have seems good enough.

I'll look at my source code to ensure it is clean and clear.
my code supports a system with an inverter, an ARC, an Automatic Generator Start (AGS) and a Battery Monitor Kit (BMK) so there's lots you would want to strip out / ignore.

My experience says a 2 millisecond delay is enough to define a message break. I check the revision number to be sure I have the right message. What is not documented is (at least with my remote) the remote sends a message with byte 17-21 all 0x00. I call it Remote_00 to distinguish it from the defined remote message with a defined value in byte 21. (offset 20).

i'm also pragmatic about the data. There is so much data streaming that I don't care if the odd message is dropped, there's going to be another one in a few milliseconds.

Since I wrote the blog entry I now just build a set of messages, on demand, once a minute. In my situation there are 11 distinct messages with about 95% of the data of no use to me. I log it all just in case. With a one minute interval and including MorningStar data, it's only about 500KB a day. In your situation you may ONLY need the inverter record.

salts
Posts: 13
Joined: Fri Nov 15, 2013 2:38 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Aug 25, 2014 3:52 am

Really appreciate you taking the time to answer my posts.

Great to know the protocol version I posted is good enough, I would use a serial data scope in development, so perhaps you may get the odd direct question, if something does not match my copy of the protocol :-)

I would also not release your code to anyone, without asking you, in addition I will not release any of my own code, without your prior approval as I will consider it a derivative work.

All the best

Ian

Von Baron
Posts: 4
Joined: Thu Oct 23, 2014 1:13 am

Re: Using a Pi to Log Data in a Magnum Energy System

Thu Oct 23, 2014 4:29 am

Hello Charles,

Found this post of yours while looking for means to communicate with Magnum inverters.
What I am doing is developing an application in C that communicates via Modbus to connected devices and provides a HTTP interface back to that data. Typically these Modbus devices might be IO modules or for renewable energy systems, any number of Solar charge controllers and inverters. What I want to include is the ability to communicate with Magnum inverters, such as MS4448PAE units.

In essence, it is a SCADA application, that is lightweight. It does not use or need Apaché or PHP.
But rather that running on RPi, I am using the Cubieboard II, operation on Debian Wheezy text mode.
The aim is that the Cubieboard acts as a headless server, talking Modbus TCP/RTU with the HTTP server built in.
Two libraries are used, one for Modbus, libmodbus-3.1.1 from https://github.com/stephane/libmodbus and the second is libmicrohttpd-0.9.34 from http://www.gnu.org/software/libmicrohttpd/

It's up and running presently, but is still early days, and I have it talking to Arduino with Modbus TCP and RTU modes.
You made mention about wanting to do Modbus as well for your system and I wonder how far you have progressed with this part. My application scans all configured Modbus devices, whether TCP or RTU for all configures data tags on a poll basis in separate threads, so they don't impact upon each other, and polling is defined and set for each device. All data is stored in a large data structure that maintains many things such as alarm conditions, statistical data, and historical trending saved into files. Everything about the system is configure in text file for Modbus device and CSV for data tags configuration.

To access the data from the external devices, you simply make a HTTP request to the application and the data is returned.
Typically, the HTTP flow is such that the initial requests call the html/css and main javascript files, follwed by AJAX requests for the data at the set period you need. This can be either as a local HTTP client as in your case with no internet, or via the internet.

So, getting to my main question, is I would like to develop soem code in C to allow collecting of data from the two Magnums.
The system has a AGS and BMK and a RTR installed. The two Magnums are in a stacked formation.

I understand from reading your post you connected the RS-485 end of your converter to the AGS unit with a splitter.
Is this the best or only approach in terms of where to connect into the Magnum communications?

I would be interested to know more from you in terms of the code you developed in Python and how I might re-adapt that in C.
If you're interested at all in knowing more about my application I mentioned above, which I'm sure will run quite nicely on a RPi, then give me a shout or contact me through http://paulalting.com.
I am off grid myself, having a hybrid solar and micro hydro system.
____
Paul

CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Thu Oct 23, 2014 3:58 pm

Thanks for your interest

I have not yet written a modbus version but it is planned for this winter. My system is at my summer cottage, which is closed until May.

I recommend you read the document titled Magnum-Networking-Communications-Protocol-(2009-10-15) which you can get from an URL mentioned earlier in this topic. RS485 is only way you can communicate with the Magnum network. Once I got my serial reader working it was just work to manage all the data, nothing technically challenging. The RS485 protocol is such that it doesn't matter where in the chain of connected network cables you add one more, it's always a splitter connection. Magnum uses fancy labels but that's to make it easier to explain, I expect its a Y connection in the Inverter and the BMK and perhaps the RTR. On my system I used a splitter on the AGS as all my other connections were inside the main box with only the AGS outside it was a convenience decision.

My plan for a modbus implementation was to present a standards based interface layer. Adding a CSV layer would be an extension exploiting the modbus interface.

If you're writing in C you may be able to use unblocked serial reads so you can detect an end of message, as I described earlier, I implemented in Java and my version has to used blocked reads, making it harder. I did not use Python. Although I have coded in many languages, I've avoided Python so far.

Reading the document you will discover that most of the message content is semi permanent data such as battery capacity, version numbers, and AGS start/stop parameters. Very little is changing data such as SOC and voltages.

As you've described, getting the data is easy (mostly) but logging it can be an endless job. I currently just dump a copy of all the info to a file and change logs every midnight.

I have not considered a dual inverter implementation but having looked at it I may do that for the modbus. If you can wait a few months I expect to have a modbus version working. I've designed it so it will be a standalone Raspberry Pi with direct battery power and a standard RJ11 connection for RS485 and and Cat5 for TCP/IP. Your CubieBoard would do the same thing. Arduino? I have no experience, but probably. (BTW My battery power scheme wouldn't work for 48VDC units as my DC-to-DC converters only work to 37VDC).

Let me know if you want more info.

Von Baron
Posts: 4
Joined: Thu Oct 23, 2014 1:13 am

Re: Using a Pi to Log Data in a Magnum Energy System

Thu Oct 23, 2014 9:03 pm

Charles,
Yes, I alread have the document, Magnum-Networking-Communications-Protocol-(2009-10-15).pdf and have read through it prior to finding your post. The document is vague at best, in that it lacks clarity and detail in exactly how the protocol works.

So, it would seem from what you have seen, it might be also posssible to connect in at the RTR or anywhere on that loop.

You say 'My plan for a modbus implementation was to present a standards based interface layer. Adding a CSV layer would be an extension exploiting the modbus interface.' When I said 'Everything about the system is configure in text file for Modbus device and CSV for data tags configuration' I think you may have thought what I am doing is not based on a standard. I have not gone away from standards at all, simply that my application is such that all the data tags you wish to read from all connected devices is laid out in a text file, based on CSV. This file contains all the configurational information about each tag. Things like from which device, the actual modbus register, the raw and engineering scaling as well as device ADC error offset, whether it will be logged in one of a number of log files, what the alarm setpoint condition are and if the alarm check is enabled. The application reads this information on startup and creates all the data structures it needs to then spawn the separate threads which go about doing pure Modbus transactions as implemented by the libmodbus library.
There are additional config files that describe the IO devices, such as whether they are TCP or RTU and what address, (IP or async /dev/ttyUSB) and poll rates and so forth. Again, my application uses this on startup, so it is not hard coded into the actual application. This makes it very easy for anyone to use the application and simply create the needed text files.
I have a chap in the states where I have just done this for their 'off grid' home, and another person in Turkey shortly for their 'hydroponics' system. The application remains the same. I hope that clarifies it somewhat for you.

You mention 'The RS485 protocol is such that it doesn't matter where in the chain of connected network cables you add one more', I guess I would like to clear that up in terms that there is no such thing as a RS485 protocol, RS485 is purely an electrical specification. People so often get this confused. It is the actual data protocol that allows for device addressing and therfore for multidroped networking to work. In the Magnum protocol document I don't see any reference to device addressing as you would expect in a master slave configuration, only a header byte which tells where the following data originates from.

You mentioned 'Very little is changing data such as SOC and voltages', but from the document I see the BMK sends this information at byte 1.

You mention 'I currently just dump a copy of all the info to a file and change logs every midnight.' This is where my application works nicely, it can log any data tag to multiple log files, 60, 120, 300, 900, 3600 and 86400 second based log files. You can have the engineering tag value logged to any or each one of those files. And that data is retrievable via a simple HTTP request, where you specifiy the tags and the start time and number of records.
For example, http://yourwebsite/?id=requestTrend&trend=trend60&tags=4/5/6/7&start=5&size=50
Each tag in your system has a reference number, which makes it easy and quick to parse.
If you're interested and to give you an idea of all how this applies in a working system, take a look at my own little system http://paulalting.com and then scroll down to see the orange label titled 'HydroSolar'.

You say 'If you can wait a few months I expect to have a modbus version working' I'm not sure what I need to wait for here, I have an operation Modbus SCADA system operating right now. My questions are about adding in the Magnum data protocol which you have appeared to have done and this is what I am interested in from you. I mentioned my system as it might be of interest to you and would share with you if you were interested, but it is all in C. Like I say, my development is on my Cubieboard II and I expect it would compile and run fine on the RPi. The Arduino is simply used as Modbus slaves, getting analog data and doing PID loop calculations to control PWM to SSR (Solid State Relay), so, simple direct IO stuff. The Cubieboard runs my Quadlog SCADA application.
____
Paul

oceanofthelost
Posts: 2
Joined: Tue May 19, 2015 9:43 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Tue May 19, 2015 9:48 pm

I just purchased a magnum energy inverter for my solar array. I was wondering If i could get the source code for the eclipse project. I would like to experiment with the magnum inverter. I plan to control the inverter through the rs485 port and also to log all of the data that comes out of the port as well. Any help would be awesome.

CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Wed May 20, 2015 1:01 pm

I will build a workspace zip file over the next few days. I have one now but it has a lot of extraneous code I'd rather not publish. I hope you're an experienced Eclipse/Java programmer. ;)

I am concerned about your desire to "control the inverter using RS485". I'm not sure my code will help in this area as I made no provision for the timing required to know when to write a message to the magnum device. But that can be your challenge.

If I don't send you the file in a week, remind me.

oceanofthelost
Posts: 2
Joined: Tue May 19, 2015 9:43 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Wed May 20, 2015 3:19 pm

Yea I really want to build upon what you have started. I plan to start with the data logging and then once that is working I will work on getting the timing done so I can send response messages to the magnum. My final challenge will be getting the magnum to act as a slave device which if a few things happen I should be able to do that. Would make integrating it with a modbus system extremely easy. But we shall see.

owen_a
Posts: 54
Joined: Sun Sep 09, 2012 3:31 pm
Location: United Kingdom
Contact: Website

Re: Using a Pi to Log Data in a Magnum Energy System

Mon Mar 21, 2016 7:45 pm

@CharlesGodwin - Did you manage to interface with the TS-MPPT-60A charge controller over Ethernet? I'm currently trying to do this my self and struggling quite a lot.

Willeert
Posts: 5
Joined: Tue Jun 14, 2016 2:06 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Jun 14, 2016 2:12 pm

I have a Magnum / Midnite offgrid system. I can do Modbus communication with the Midnite Classic but would like to add the Magnum equipment to my system. I code in Arduino C but any help in any code would help me puzzle out the Magnum system I am sure.

Thank You

Will

User avatar
RDK
Posts: 181
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Jul 05, 2016 10:25 pm

Guys.....Really got my juices flowing as I read this thread. My son has MS4448PAE inverter and ME-BMK battery monitor at his "off grid" cabin. Oddly, even though it is truly "off-grid" he has a way to get internet services. Thus, he also has the MagWeb Monitoring Kit (ME-MW-E) so he can check out the system via the Magnum site.

Neither of us like the way Magnum handles the data and want a "better way" which I think can be had using some of your expertize. However, I would like to install a Raspberry Pi Internet Bridge between the MagWeb device and the Internet router. Then, using TCPDUMP, I could "read" the records coming from the MagWeb, and process them similarly to what you are doing via the RS485 device.

OK, the above is an outline of what I want to do, but so far no code has been developed and I am only now experimenting with TCPDUMP to see what the transmissions from the MagWeb look like.

Can someone post a sample data file which I could use to see if the TCDUMP stuff looks at all like what you are seeing.

Perhaps someone has already developed a system like I'm proposing?

By the way, I have done this for my weather stations so I know, in principle, that is it possible. Also, I code in Python, but could most likely work my way through either Java or C.

Thanks.....RDK

Von Baron
Posts: 4
Joined: Thu Oct 23, 2014 1:13 am

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Jul 12, 2016 2:00 pm

Hello RDK and Will,

I am guessing, RDK, that you may have seen my posts over on the Midnight Solar forums, in particular, the following thread:
http://kb1uas.com/mnsforum/index.php?topic=2458.0, where the last post was by RDK45, so guessing this maybe you ?

I know Will has, as we have worked on other projects together in the past.

For others interested, the following is a description of a fairly simple bit of Python code to decode the data stream from the Magnum MagnaSine inverter system of components as described on the Midnight Solar forum as indicated above, called MagPy.

It was basically a fun project for me to assist others over on the Midnight Solar forum back in June 2015.

The decoder was coded up basically on the published protocol specification sheet I was able to obtain online, and no much else.
I don't use or have a Magnum system, we don't see a lot of this brand here in Australia.

There are many combinations of various equipment that can make up the Magnum network, and I haven't gone through all these possibilities.
Also, at the time I wrote this, I didn't then have an up to date protocol specification and that newer equipment from Magnum may in fact use a different more up to date protocol specification. What this means is that, the program as written at the time, may not perform as well with other or more up to date equipment than I had to test at the time. You are free to experiment and enhance the code.

I was able to test the decoder by two methods, first by using an Arduino as a Magnum network simulator and secondly by uploading the Python code via SSH to a small GNU/Linux board located in North America which was connected to a Magnum MagnaSine inverter system and see that it decoded useful data from the continuous data stream.

The first test, using an Arduino as a simulator for the Magnum network, is, again, a simple little program that simply regurgitates actual captured data from a real Magnum network in the various packets as specified by the Magnum protocol specification.
Simply connect the Arduino via USB to your system running MagPy to experiment and extend the code to suit your needs.

Unfortunately the protocol used by the manufactures of the Magnum inverter system did not follow a standard well understood or used data protocol, but rather, it seems, they invented their own, of which the protocol data sheet mentioned in this thread, is the only actual piece of information I had on hand to go by.

I chose Python as the language as it allowed rapid development and ease of testing and also ease of deployment for other users without a lot of fuss. Simply ensure you have Python installed on you system of choice, together with one important library, PySerial, which can be installed from https://pypi.python.org/pypi/pyserial. Also, consider reading some of the documentation on PySerial, especially about installing PySerial, at http://pythonhosted.org/pyserial/pyseri ... stallation

The decoder works as a simple data packet analyser and uses a standard communications port to get data from the Magnum network.
To make things easy, it is possible to use a USB to serial communications port adapter to connect to the Magnum communications network. This can be either as USB to RS-232 and then RS-232 to EIA-485 or simply have a USB to EIA-485 adapter.

The Python code should work across all platforms, whether that be a simple RPi or more powerful ARM or Intel based systems.
My only real world tests are with a Cubieboard II (ARM A20 CPU ), running Debian and also on my Mac Mini running OSX, where I used the Arduino simulator as my pseudo Magnum network.

To run MagPy you have a few command line options.
Running the program without arguments will use a default communications port of "/dev/ttyUSB0'
You can specify an override with the -p or --port argument, so for example;

Code: Select all

python MagPy.py -p /dev/ttyUSB1
There is also a debug option, but will leave that for users to read into the code to understand better what that does.

Will, I think the Python code will lend itself to being translated to C/C++ for integration into your Arduino project if you are wanting to do this.

So, two pieces of code, first the Python code itself to decode, and secondly the Arduino simulator code I used for testing.
Since, for some reason, I am unable to add an attachment directly here on the forum at present, I'll provide two DropBox links.
MagPy https://www.dropbox.com/s/hobhg6ncu5iwc2v/MagPy.py?dl=0
Simulator https://www.dropbox.com/s/g3cuka904offb ... r.cpp?dl=0

Have fun and experiment with it.
Paul Alting van Geusau

User avatar
RDK
Posts: 181
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Jul 13, 2016 12:22 am

Paul..Thanks for the reply and yes, I am RDK45!!!

The technique you are emulating is NOT the same as I wish to use. I have experience using a Raspberry Pi as an Internet bridge, that is a device which spies on traffic passing through it. In practice, I have the Pi attached to the internet (actually a switch on my router network) and the MagWeb device attached to the PI. In this configuration the Pi can see all of the traffic from MagWeb to the Magnum server. Magnum uses a "different" TCP protocol which wraps their binary information in a HTML wrapper. Using TCPDUMP on my Pi have trapped their traffic. The records are about 273 hex characters long. Here is an example:
D47F199D160000000706050402FFFFFF024000022700037A000100330F2C1B7300000102580000053C04641E170600
E19B840C14032C032C14007300000090787878005A00781478502800200A003C3C00000000000000002800167B2E310F969
B8AFBFC906416160010114319200006FFFF00580A01F00B066D285B60A082572E4A06003C0000003C00000000204AE95C
9708863B27B6F60800450000289CC340003206F7CA4509AE86C0A8000A0FB4C7258361701B0016C7D2501016D05283000
00000000000007EA08257605D0C00B5000000B500000008863B27B6F600204AE95C970800450000A7ACBF40004006D94F
C0A8000A4509AE86C7250FB40016C7D28361701B501807FFCB860000234D13
I have identified the Inverter section but so far have not found the battery section. The inverter section starts at about character 17.
Hex Description
40 Inverter Status, See PDF for codes
0 Inverter Fault. See PDF for codes
2 DC Volts
2B
0 DC Amps
0
77 AC Volts out, 0-255
0 AC Volts in, 0-254
1 Inverter LED, 0=off, else on
0 Charger LED, 0=off, else on
33 Invert Revision 51=5.1
E Battery Temp
21 Transformer Temp
18 FET Temp
73 Inverter Model
0 Stack Mode
0 AC Input Amps
0 AC Output Amps
2 AC Hz
58
where DC volts, DC Amps and AC Hz are double words

My method avoids the timing issues with reading the records coming via the serial port on the inverter, I think?

Anyway, for now I have nothing which actually works real time and will probably delay that until I decode the battery data and perhaps the time/date.

Any ideas? Have you had any success trying to work with the Magnum engineers to decode the records?

Thanks....RDK

Von Baron
Posts: 4
Joined: Thu Oct 23, 2014 1:13 am

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Jul 13, 2016 1:21 am

Bonjour RDK,
Je suis curieux de ce que RDK est :roll:

I understand better now that you wish to use a Ethernet TCP connection rather than a async serial connection.
Yes, you would be correct, with TCP, you will not have the critical timing issues that are present with async serial, as it will follow standard TCP packet flow control, which will be handled by the low level Ethernet drivers at both ends for you.

Maybe you can take my Python code and simply re-code the way you get the data, via a TCP connection rather than an async serial connection. Use some TCP library and pull the data into a buffer and decode, much like how MagPy does.

I understand you are more conversant with Python than Java, and that your son is more with Java. Both should be suitable to the task.

For me, this was only a small side project, and one where I might re-code my work into C/C++ to integrate into my ongoing SCADA application which currently only uses Modbus as its communications protocol.

Since I last work on this Python code, over twelve months ago, I have not progressed it or had any communications with people at Magnum.

Santé
PRAVG 8-)

User avatar
RDK
Posts: 181
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Jul 19, 2016 7:01 pm

Paul...Well, I'm back in France after 3 weeks with my family and friends in Colorado and Wyoming. While at my son's cabin I captured several sequences of traffic between the MagWeb unit and the Magnum server. So far I'm at a brick wall to decode it all. I have found the inverter section but so far can not find time/date or battery information. With several dates, etc. I would have thought I could find a few things, but nada... I'm using the Oct 15, 2009 Protocol PDF file as a guide.

If I don't crack this soon, I may send a note to Magnum to see if they will offer any assistance????

If you have any ideas, let me know....RDK

PS....RDK est moi, un chimiste/mathématicien retraité avec une belle femme française !!...

CharlesGodwin
Posts: 27
Joined: Wed Aug 22, 2012 9:57 pm
Location: Ottawa Canada

Re: Using a Pi to Log Data in a Magnum Energy System

Tue Jul 19, 2016 7:07 pm

If Magnum is sending just the raw data from the system there may be no date time data. The system keeps track of time of day but no date stuff. None of the messages are time stamped. The best it does for dates is days since last generator run.

By all means contact Magnum but i'm not expecting much help. I hope i'm wrong.

Good luck and keep us informed.

DMackG
Posts: 4
Joined: Tue Jun 16, 2015 7:29 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Dec 21, 2016 2:57 am

Hi Guys,
I know this is a bit dated but was wondering if there has been any further development on this project? Definitely has potential!

User avatar
RDK
Posts: 181
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Using a Pi to Log Data in a Magnum Energy System

Wed Dec 21, 2016 7:48 am

Guys.....I'm heading back to the USA in about a month and will have local access to the Magnum for experimentation. I have most of the information I need, as posted above. Now looking to "Charge State" and a few other minor items.

If I have any success I will post something here...RDK

User avatar
RDK
Posts: 181
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Feb 12, 2017 2:07 pm

Guys.......I think I have it all working now. I have had access to the Magnum setup for the past three weeks and now I believe I have decoded both the Battery and Inverter data records that are being sent to the Magnum website via the Internet from the Mag Web unit.

I will be returning to France tomorrow and when I get back I will set down and document my findings. Are you still interested?...RDK

Willeert
Posts: 5
Joined: Tue Jun 14, 2016 2:06 pm

Re: Using a Pi to Log Data in a Magnum Energy System

Sun Feb 12, 2017 10:02 pm

I am interested. I have been reading a book about coding in Java to help me with the code that Charles Godwin was kind enough to share with me. My goal is to convert it into Arduino C and then write to the Magnum inverter to turn it on and off in as the Arduino requires.

Thanks for offering to share.

I will also post up when I have something working. I have ordered rs485 hardware to start the read process of the Magnum system.

Will

Return to “Automation, sensing and robotics”

Who is online

Users browsing this forum: No registered users and 10 guests