OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Hanging in background causing failures. Swappiness? Cron?

Fri Nov 02, 2018 10:51 pm

Short Summary of Problem

I am using a RPi 3 to run a Python script to record a moving object over long periods of time. Occasionally, the program will hang, which causes the camera to stop recording. This is bad because while hanging, the object can move outside of the camera visual range and cause the program to crash. I am having trouble diagnosing the problem and finding a solution.

Description of Hardware, OS, and Software

The hardware is an RPi 3 with a Pi Camera and external hardware attached to the GPIO. The hardware attached to the GPIO is a turret mount for the camera. The power supply is providing 5.1 V and the RPi draws about 0.8A. The GPIO is only supplying power to the motor control circuit. It is NOT supplying power to the motors themselves. There are no low-power alerts with the current power supply. Recorded video is saved to a USB drive and does not touch the SD drive. The SD card has a speed rating of 10, so even if something was touching it I doubt it would have issues.

The RPi is running a mostly vanilla up-to-date Raspbian image. I have made changes to the swappiness in an attempt to deal with the crashes. Note that I have adjusted swappiness and some of the expiration times.

Code: Select all

vm.swappiness = 60
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
The Python script is fast, as it is mostly being used as a wrapper for Numpy and OpenCV, programs that are compiled in C. Despite running as a recording program, very little visual memory is required. A small live feed is shown in a Pygame interface. The calculations that are handling the object tracking happen with Numpy and OpenCV on arrays. This is more calculation heavy than rendering heavy. If you would like to inspect the code, you can find it on Github.

Crash Diagnostics

I have made some changes to things to attempt to zero in on the problem. Below are things which are reported in things I have googled about similar problems. I have tried fixes related to them and was able to rule out these issues. I recorded what happened directly after shutdown (by crash or normal shutdown when the object being tracked is legitimately lost) for the last 58 runs. So I have a nice size dataset to look at for issues.
  • To determine if there was a memory leak being caused by my program, I added a Cron job. This was based on a suggestion in an older thread (here):

    Code: Select all

    */5 *  *  *  *  free -m >> /var/log/memory_log
    The logged memory usage is pretty consistent, so I'm not going to dump everything here. Instead, this was the latest addition:

    Code: Select all

                  total        used        free      shared  buff/cache   available
    Mem:            748          76          54          10         617         598
    Swap:            99           0          99
  • While viewing the recent logs after a crash with

    Code: Select all

    journalctl -n 10 | cat
    upon shutdown, I looked for patterns reported by other users who have had uptime issues. On the apparent connection to an hourly cron job, we can see from the logs:

    Code: Select all

    1540372707.89 , 2018-10-24 04:18:27
    
    -- Logs begin at Thu 2016-11-03 12:16:42 CDT, end at Wed 2018-10-24 04:17:01 CDT. --
    Oct 24 03:04:23 raspberrypi sudo[14909]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
    Oct 24 03:04:23 raspberrypi crontab[14913]: (root) BEGIN EDIT (root)
    Oct 24 03:04:26 raspberrypi crontab[14913]: (root) END EDIT (root)
    Oct 24 03:04:26 raspberrypi sudo[14909]: pam_unix(sudo:session): session closed for user root
    Oct 24 03:17:01 raspberrypi CRON[15037]: pam_unix(cron:session): session opened for user root by (uid=0)
    Oct 24 03:17:01 raspberrypi CRON[15041]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
    Oct 24 03:17:01 raspberrypi CRON[15037]: pam_unix(cron:session): session closed for user root
    Oct 24 04:17:01 raspberrypi CRON[17227]: pam_unix(cron:session): session opened for user root by (uid=0)
    Oct 24 04:17:01 raspberrypi CRON[17231]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
    Oct 24 04:17:01 raspberrypi CRON[17227]: pam_unix(cron:session): session closed for user root
    There are several entries like this in my crash log. It appears that the hourly cron job is NOT causally related to the crashes. The crash time, (listed at the top of each entry) does not correlate to the job time. I disabled this job just in case.
  • Another reported issue is a daily crash after time updates. Again we can see from the logs:

    Code: Select all

    1540523342.24 , 2018-10-25 22:09:02
    
    -- Logs begin at Thu 2016-11-03 12:16:42 CDT, end at Thu 2018-10-25 22:01:52 CDT. --
    Oct 25 22:01:48 raspberrypi systemd[1]: apt-daily-upgrade.timer: Adding 59min 59.245696s random time.
    Oct 25 22:01:48 raspberrypi systemd[1]: Reached target Network is Online.
    Oct 25 22:01:48 raspberrypi systemd[1]: Starting Daily apt download activities...
    Oct 25 22:01:51 raspberrypi systemd[1]: Started Daily apt download activities.
    Oct 25 22:01:51 raspberrypi systemd[1]: apt-daily.timer: Adding 6h 36min 44.508040s random time.
    Oct 25 22:01:51 raspberrypi systemd[1]: apt-daily.timer: Adding 6h 53min 53.014271s random time.
    Oct 25 22:01:51 raspberrypi systemd[1]: Starting Daily apt upgrade and clean activities...
    Oct 25 22:01:52 raspberrypi systemd[1]: Started Daily apt upgrade and clean activities.
    Oct 25 22:01:52 raspberrypi systemd[1]: apt-daily-upgrade.timer: Adding 13min 42.753046s random time.
    Oct 25 22:01:52 raspberrypi systemd[1]: apt-daily-upgrade.timer: Adding 19min 55.751872s random time.
    After looking at multiple crash logs that look like this, again I can say that this is not causally related to the problem as there is no correlation between time to crash after the daily cron job. I kept this job running unlike the daily one since time correction is important to my experiments.
  • I dropped some unnecessary lines of code from the Python script to speed things up. Now it no longer opens and closes gpicview as that was not needed outside of early testing. I also reduced the number of variables in the program. I could convert to Cython, but since most of the heavy calculations are happening in Numpy, that seems like it would not do much.
The Big Questions

How can I run this dang thing all night without it having a stroke and losing the tracking target?

What is going on here, and how do I fix it? These crashes are unacceptable, so I need to find a way to stop them. Cron appears to not be causing issues like is usually reported for similar issues. I can play with the swappiness more, but I don't know what would be optimal for this type of setup. Is there a write to disk issue with the USB (seen here)? Suggestions for more things to test are welcome.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Tue Nov 06, 2018 7:11 pm

Bump

(What are the rules on bumping here? I assume 1/day? The rules sticky just says "don't spam".)

n67
Posts: 938
Joined: Mon Oct 30, 2017 4:55 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Tue Nov 06, 2018 7:22 pm

Well, bumping is kinda by definition rude.

It's kind of like asking what is the acceptable level of line jumping.
"L'enfer, c'est les autres"

G fytc hsqr rum umpbq rm qyw rm rfc kmbq md rfgq dmpsk:

Epmu Sn!

J lnacjrw njbruh-carppnanm vxm rb mnuncrwp vh yxbcb!

DirkS
Posts: 10017
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: Hanging in background causing failures. Swappiness? Cron?

Wed Nov 07, 2018 9:32 am

n67 wrote:
Tue Nov 06, 2018 7:22 pm
Well, bumping is kinda by definition rude.
I think that bumping once, a few days after posting is acceptable.
I don't think it's rude. It is very easy to miss a post the first time around.
Especially the case with a first-time poster. It's possible to mark a first post as read, even before it's visible on the forum.

n67
Posts: 938
Joined: Mon Oct 30, 2017 4:55 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Wed Nov 07, 2018 10:29 am

The argument would be that it is acceptably rude. I.e., that a certain level of rudeness is inevitable in this world and thus acceptable.
"L'enfer, c'est les autres"

G fytc hsqr rum umpbq rm qyw rm rfc kmbq md rfgq dmpsk:

Epmu Sn!

J lnacjrw njbruh-carppnanm vxm rb mnuncrwp vh yxbcb!

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Thu Nov 08, 2018 11:01 pm

[arguably rude thread bump]

Bumping an involved problem like this is pretty much a necessity. I know one of you has the very specific knowledge of this problem, but I have to wait until you stumble across my question.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Tue Nov 13, 2018 12:00 am

bump

Still looking for direction on this.


OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Thu Nov 15, 2018 10:39 pm

bump

Can I get a confirmation that I am asking this in the correct forum? I am having some of my volunteers ask if I have found the fix yet, and I have nothing to show.

User avatar
bertlea
Posts: 301
Joined: Wed Dec 07, 2016 6:33 am
Location: Hong Kong

Re: Hanging in background causing failures. Swappiness? Cron?

Fri Nov 16, 2018 9:52 am

From your detailed description in the first post, it seems to me that the issue is not caused by memory issue, power or even SD card. However, I have some doubts on your results of memory check as it seems the system use very small amount of memory and the swap file was never used. I expect opencv will use quite a lot of memory. Are you sure you get the memory statistics during your script is running?

I can only suggest few things to do to try to isolate the cause of the problem in this situation:

1. Try running the system without the external hardware attaching to the GPIO, see if the system can run overnight without issue. If it does survives, then the direction is clearly about the GPIO connected hard ware.
2. Similar method can applies to narrow down the scope. Assume it is related to the GPIO and the hardware, then you try to change the program that do not do anything to control the hardware via GPIO. This can tell if it is the communication or action via GPIO causing the problem.

That is just a general try-and-error method to find the cause of the issue, as I am not expert on hard-ware related stuff. I hope others with better knowledge can look into that again if you have more findings.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 19, 2018 10:21 pm

Thanks for the recommendations @bertlea! I really appreciate your input.

Let me run through your suggestions one by one.
bertlea wrote: I expect opencv will use quite a lot of memory. Are you sure you get the memory statistics during your script is running?
Yes, my statistics are while the script is running. It seems I need to clarify my original post. We are using Numpy for this script right now. Earlier versions were using OpenCV, but I took it out since it was my first guess as to why I was having problems. Currently, the only Numpy calculation is once every few seconds. The relevant code is here:

Code: Select all

def getImg():
	#Capture an image and see how close it is to center
	global stream
	global imgThresh
	global theImg
	#stream = io.BytesIO()  #Variable to hold image
	camera.capture(stream, use_video_port=True, resize=(horDim, vertDim), format='jpeg')
	img = Image.open(stream)
	img = img.convert('L')        #convert to monochrome
	img = img.point(lambda x: 0 if x < imgThresh else 255, '1')
	arm = np.asarray(img)        #convert to numeric array
	armcol = np.sum(arm, axis=0) #sum down all columns to give a vector
	armrow = np.sum(arm, axis=1) #sum across all rows to give a vector
	armcol[armcol < 10] = 0        #quick filter to remove effects of any stray pixels
	armrow[armrow < 10] = 0        #quick filter to remove effects of any stray pixels
	cmx = float(np.mean(np.nonzero(armcol)))  #horizontal center of moon
	cmy = float(np.mean(np.nonzero(armrow)))  #vertical center of moon
	cenx, ceny = horDim/2, vertDim/2   #center of image
	ratio = arm.sum()/(float(horDim)*float(vertDim))     #ratio of white:black
	diffx = cenx - cmx  #image center x minus mass center x - if pos shift right, if negshift left
	diffy = ceny - cmy  #image center y minus mass center y - if pos shift down, if neg shift up
	print(ratio, cmx, cmy, diffx, diffy)
	return diffx, diffy, ratio
As you can see, the frames are captured from the stream, simplified to binary black and white, then subjected to some np operations. While Numpy can do some power-hungry things, this script is simple.
bertlea wrote: 1. Try running the system without the external hardware attaching to the GPIO, see if the system can run overnight without issue. If it does survives, then the direction is clearly about the GPIO connected hard ware.
I have done this before. The external hardware connection seems to have no effect on the system.
bertlea wrote: 2. Similar method can applies to narrow down the scope. Assume it is related to the GPIO and the hardware, then you try to change the program that do not do anything to control the hardware via GPIO. This can tell if it is the communication or action via GPIO causing the problem.
This remains to be tested. Removing the GPIO output from the code will be a physical challenge, and I will need to construct a stable hardware object for the unconnected device to 'track'. I will let you know the results of this test when I have time to attempt it.

User avatar
bertlea
Posts: 301
Joined: Wed Dec 07, 2016 6:33 am
Location: Hong Kong

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 8:56 am

Hum… I just read your original post again and I realized there are known scenario you know your program will crash:
“This is bad because while hanging, the object can move outside of the camera visual range and cause the program to crash.”
Is it possible something is more likely to happen at the night time that can cause your program “loose track” of the object and crash? For example, due to some lighting condition changes?

Idahowalker
Posts: 445
Joined: Wed Jan 03, 2018 5:43 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 12:34 pm

Perhaps, you might want to be a bit more granular with your troubleshooting by drilling down into the routines running your program. Start with the Python code and add in some 'try-catch-finally" statements and have those statements logged in some fashion.
Without knowing why you are deleting my postings, I will not know how...

User avatar
scruss
Posts: 2628
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 2:41 pm

Looks like you've tried a bunch of the right things, so it might be time to look in other directions:
  • you mention you're using a Raspberry Pi 3 and it's under some CPU load: how's your CPU temperature through all this? It might be going into thermal throttling and then failing when the processing routine gets too slow to track your object.
  • long-running interpreted programs can sometimes fall foul of garbage collection issues. Are you ever explicitly cleaning up the objects created by your library calls? The Python gc garbage collection interface has some control/diagnostics that might prove useful. I'm not suggesting you're running out of memory — you've already shown us that you're not — just that sometimes garbage collectors get to a state where they halt everything to clean up.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 4:10 pm

bertlea wrote:
Mon Nov 26, 2018 8:56 am
Hum… I just read your original post again and I realized there are known scenario you know your program will crash:
“This is bad because while hanging, the object can move outside of the camera visual range and cause the program to crash.”
Is it possible something is more likely to happen at the night time that can cause your program “loose track” of the object and crash? For example, due to some lighting condition changes?
Good question.

This absolutely can and DOES happen. I'm tracking the moon, and little changes in the lighting can cause problems. The big one is clouds. As soon as a cloud passes between the moon and my scope, the moon develops a diffuse edge/halo and darkens slightly. This plays havoc with the lighting thresholds and causes major tracking problems. Rather than tracking an ellipsoid with crisp edges and an obvious L threshold, the cloudy moon has a gradient of luminosity which changes shape with cloud thickness and is larger than the viewing area.

However, this is a known issue with distinct failure states. The recorded videos show the obvious clouds moving over as well to confirm the type of problem. To my knowledge, nothing in lighting conditions changes and causes a failure.

I cannot rule out that there is something else in the night that causes issues. I have already tackled temperature and humidity related problems on my own. I cannot be certain that errant dew is not causing problems with the hardware. Such an issue would be isolated to a single component, as I do not experience other obvious electrical shorts. Furthermore, this failure has been observed in low humidity, temperate conditions.

Example of the moon being lost due to lighting changes: https://youtu.be/6IXiuzAC_NU

Example of the moon being lost due to unknown glitch: https://youtu.be/ZHoGnqVQe5w

In the glitch video, you can see that there are missing frames. These frames are lost during the glitch.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 4:13 pm

Idahowalker wrote:
Mon Nov 26, 2018 12:34 pm
Perhaps, you might want to be a bit more granular with your troubleshooting by drilling down into the routines running your program. Start with the Python code and add in some 'try-catch-finally" statements and have those statements logged in some fashion.
This was one of my first thoughts. I did not use the try-catch technique, as there is nothing in the errors to catch (that I know of). Instead, I ran a version of the code which tediously printed the line number each time something was executed.

This approach showed that the Python was running as expected, even if there were pauses during the routine.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Mon Nov 26, 2018 4:18 pm

scruss wrote:
Mon Nov 26, 2018 2:41 pm
Looks like you've tried a bunch of the right things, so it might be time to look in other directions:
  • you mention you're using a Raspberry Pi 3 and it's under some CPU load: how's your CPU temperature through all this? It might be going into thermal throttling and then failing when the processing routine gets too slow to track your object.
I will keep this in mind for the next tests. I have not been recording the temperature.

Anecdotally, the important chips are warm to the touch, but don't blister my skin (like the external motor controller sometimes does). The RasPi has some of those cheapo add on thermal baffles, and tests are occurring in areas with good (sometimes so much it jiggles my scope) airflow.
scruss wrote:
Mon Nov 26, 2018 2:41 pm
  • long-running interpreted programs can sometimes fall foul of garbage collection issues. Are you ever explicitly cleaning up the objects created by your library calls? The Python gc garbage collection interface has some control/diagnostics that might prove useful. I'm not suggesting you're running out of memory — you've already shown us that you're not — just that sometimes garbage collectors get to a state where they halt everything to clean up.
Ahhhh, this is a good lead. I did not know this about the garbage collectors. I will look into this.

User avatar
scruss
Posts: 2628
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Hanging in background causing failures. Swappiness? Cron?

Tue Nov 27, 2018 3:46 am

Just a thought — and pardon my ignorance of your very neat LunAero project — but what's stopping you using a GPS and Skyfield/AstroPy/PyEphem to maintain camera position?
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Tue Nov 27, 2018 4:34 pm

scruss wrote:
Tue Nov 27, 2018 3:46 am
Just a thought — and pardon my ignorance of your very neat LunAero project — but what's stopping you using a GPS and Skyfield/AstroPy/PyEphem to maintain camera position?
Cost and Newness.

Using positioning was something we have considered. It would be trivial to set up the software to get the lunar position down to seconds of arc every minute. I know that because I am actually using AstroPy on the back end for some calculations. But, this makes the hardware expensive. To implement two stepper motors that are able to put the moon in frame would require some complex gearing. I have plenty of shop experience, and I think I would have no problem machining the parts to do it, but making a single unit is not the goal of this project. The goal is to make hundreds of these things and deploy them across the continent. That way, we can run them in sync and track the migratory flux over the large area. So, our gears and parts are made entirely out of 2D laser cut acrylic.

The choice to use a single material with a single manufacturing method is quite deliberate. As this is an open science project, we want anybody who wants to replicate or participate to be able to DIY it. Laser cutters (and sometimes waterjets) are becoming much more common in community fab labs across the US. Really, we would be happy with any single-method/material design. We originally were working with a 3D design, but the low resolution you get on community printers makes for poor motion control.

So now we have something anyone can access, and we can make it cheaply enough to deploy a lot of these. I don't know if you hang out with many birders, so I'll let you in on a little secret. They are all crazy. Seriously. Amateur and professional bird-watchers are chomping at the bit to get one of our volunteer kits. We were able to fund giving these kits out at no cost to volunteers because of our crowdfunding campaign. They get to watch birds, and we get good data. Symbiosis.

Newness is also important. From an academic perspective, it is not worth doing unless you can get a paper out of it. There is a big prospective dataset which can be gleaned from this equipment about the birds that cross the moon. But I am not a biologist, so that is less important to me. I like to design hardware (with a PhD in chemistry...go figure). Tracking the moon with computer vision is a challenge which has not been attempted. It makes sense astronomically, because the moon is notoriously difficult to track compared to other celestial objects. The stars appear to move with the night sky, but this is entirely due to the earth's rotation. Since they are so far away, we can negate any local motion they might have. But the moon's location changes due to the lunar precession, earth tilt, lunar revolutions, and the angle the sun is striking it. It ends up being annoyingly complicated to predict where the moon will rise, and what path it will take for "dumb" hardware. So we do it with computer vision. Now we have this method ready to go for something else. Let's say we take something to Mars and want to track Phobos and Deimos, how do we do it? What if we want to do comet tracking? What if this can be useful to other industries? Do you think the military would ever be interested in a DIY turret that can track objects in the sky? Building this hardware is a novel challenge which adds to the scientific knowledge of humanity.


/walloftext

Cost and Newness

User avatar
scruss
Posts: 2628
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Hanging in background causing failures. Swappiness? Cron?

Wed Nov 28, 2018 4:08 am

Okay, that's your approach. But if your deployments are to be in sync across the continent, they either have a GPS or you're using some kind of network — which can almost always provide you with timing and position. If you've got the hardware to track the moon optically, you've got the hardware (some perhaps difficult maths notwithstanding) to use ephemeris to do so.

I do like your low-cost approach. I do open-source assistive technology myself, and it can be impressive what can be made in makerspaces. I know all about birders, too. Spent a good bit of time on Fair Isle, one of the world's destination birding spots. I've also experienced Long point and Leslie St Spit (both Ontario) when there are reports of a rare bird - folks fly in from all over.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

OUBiologging
Posts: 14
Joined: Fri Nov 02, 2018 8:24 pm

Re: Hanging in background causing failures. Swappiness? Cron?

Wed Nov 28, 2018 7:36 pm

scruss wrote:
Wed Nov 28, 2018 4:08 am
Okay, that's your approach. But if your deployments are to be in sync across the continent, they either have a GPS or you're using some kind of network — which can almost always provide you with timing and position. If you've got the hardware to track the moon optically, you've got the hardware (some perhaps difficult maths notwithstanding) to use ephemeris to do so.

I do like your low-cost approach. I do open-source assistive technology myself, and it can be impressive what can be made in makerspaces. I know all about birders, too. Spent a good bit of time on Fair Isle, one of the world's destination birding spots. I've also experienced Long point and Leslie St Spit (both Ontario) when there are reports of a rare bird - folks fly in from all over.
The units do not operate synchronously, they are in sync via timestamps. Nocturnal migration occurs within a closed window, so as long as we have a reliable timestamp through a wifi connection or RTC, we can sync up what observers record. The GPS coordinates of the recording are based on the location the device was set up according to the volunteers.

The hardware to track the moon which I was referring to was the gearing. The downside of laser cut plastic is the kerf and melt. Gears made in this manner will be at least an order of magnitude less precise than milled gears. The loss of precision of a mil in each gear tooth when tracking celestial objects at magnification is enough to cause failures and deviation from the ephemera. However, milling parts out of metal is much more expensive and much less DIY-able for the average bird-nerd. The fact that they hired me at the university level is evidence enough that they generally don't have the tech/engi skillset or desire. Our hardware must be able to be manufactured with a press of a button by a kindly retiree that likes robins a bit too much.

Return to “Troubleshooting”