Page 1 of 1

SSP Troubleshooting

Posted: Sun Jun 09, 2019 2:33 am
by KieranMITDUT
Hey Konstantin/SSP Crew, hope all is well

I had a few questions for you about the SSP, we’ve seemingly had a couple failures that I wanted to see if you could help troubleshoot.

1) Today one of our SSPs just suddenly stopped working (we had a very similar failure mode occur in the past). This occurred right after I switched the camera to BayerRGB. The system was continuously repeating (over and over again):

21:22:09 daemon.err watchdogd: Process stereod stalled for 10.5359s! Restarting!
21:22:11 stereod: Daemon started
21:22:11 stereod: Using cameras 22947068; 22947067
21:22:12 stereod: Resetting FPGA

and then for a while it switched to:

21:26:58 daemon.warn stereod: [UvDevice::read_memory] Command sending error: LIBUSB_ERROR_TIMEOUT

The steps we took for troubleshooting:
-Reset camera defaults
-Reset all configurations

Afterwards, we plugged another SSP into the camera (with the same harnessing and everything) and it worked just fine. The settings immediately after failure (or with in minutes at least) are attached below.

2) We are having issues with getting the SSP up to the quoted speed online. We are currently running 192 disparity at 800 x 320 (colour) and it maxes out at 20FPS. Even at 32 disparity it maxes out at 20FPS. The settings are attached below. Do you have any recommendations on what settings I should be using? Or what I can change?

3) We had an interesting failure mode where the SSP split an object in two (vertically), potentially because the object intersected with the horizon. See image below, do you have anyway we could penalize this in the algo? This happens quite often and causes two distinct point clouds. Rosbags available as well if needed

Thanks for your time Nerian,

Re: SSP Troubleshooting

Posted: Tue Jun 11, 2019 12:35 am
by KieranMITDUT
Adding image file for failure 3

Re: SSP Troubleshooting

Posted: Tue Jun 11, 2019 12:38 am
by KieranMITDUT
Unfortunately having issues attaching the 2nd failure mode config, please see direct email.

Re: SSP Troubleshooting

Posted: Tue Jun 11, 2019 9:05 am
by k.schauwecker
Hi Kieran,

1) I had a look at the configuration file and this was set to RGB rather than Bayer. Did you change it back before downloading the file? I could get this configuration running with a SceneScan and Karmin2 unit. However, I'm concerned about the low throughput limit that is configured. Did you have any problems that required you to reduce the thorughput limit? A low throughput limit can cause image acquisition problems. I recommend setting this to maximum.

Did you get the SceneScan unit working again after resetting the configuration? The problem could also be caused by the cameras not receiving the trigger signal. Please check the trigger cable. You can test if this is the cause by:

* Setting Trigger Mode to "off" in the cameras settings page
* Disable "Use trigger time as timestamp" on the trigger page
* Set a very large frame paring interval

If the cause is the trigger signal, then you should get image data again in this configuration. But please don't keep this configuration as the cameras will be unsynchronized.

2) I think the reason for this is the low throughput limit that you configured. This will very much limit the frame rate you can achieve. I was abler to run your configuration (with Bayer format) at 40 fps. Some other things you should keep in mind:

* Please use the current 3.2.0 firmware. There was a performance problem in the 3.1.2 release
* Achievable frame rate in Bayer pattern mode is slightly lower than for RGB mode. But with the configuration you use (800 x 320) you should be able to get 40 fps.

3) I have difficulties recognizing much in the very small image. Could you attach a higher resolution image?

Re: SSP Troubleshooting

Posted: Sat Jun 15, 2019 8:43 am
by KieranMITDUT
Hey Konstantin, thanks for the response

re issue 1: we figured out the two problems with it, for 1 (the Process stereod stalled issue) it turns out that the hardware we are using (Nvidia Jetson Xavier) has some issues with communicating with the SSP, this has been an issue we've faced for months and it shows up again here - for example we have never been able to talk to the SSP through a direct connection on it and have always had to go through a switch, we have been able to connect to the SSP directly through a USB to ethernet dongle but that doesn't help us much as we need to time sync the two. Additionally we've found (with no understanding why) that if you increase the MTU above 1500 on the Xavier (with jumbo frames disabled on the SSP) that the xavier will no longer communicate with the SSP. For the time being we'll keep our MTU at 1500 and keep it with out jumbo frames.

The second part of issue 1 (LIBUSB_ERROR_TIMEOUT) turned out to be a USB cable issue between Karmin and SSP. However this doesn't explain why the USB cable worked with one SSP and not the other.

For problem 2, the frame rate issue, we still have not been able to get any of our units above 20 FPS, we attempted it with a Lenovo X1 extreme with jumbo frames enabled and MTU on the host machine at 9000 with no luck - I will post the configuration file tomorrow along with the print out of my ifconfig to see if you can provide some insight into what is up with our settings.

Additionally tomorrow for issue 3 I'll post a link to the ROS bag for you to have a look to see if you can tell how we can deal with this failure mode

thanks for your help konstantin


Re: SSP Troubleshooting

Posted: Sun Jun 16, 2019 5:27 pm
by KieranMITDUT
Hey Konstantin,

1) please find a link to the bag with the failure mode here. The first 2 cones are a-okay but it is the third cone (on the left) that has its point cloud half in the background and half in the foreground. You can rviz the left_image and point_cloud topic to view. Is there anything you would recommend for us to set in the algo to change this?

2) Additionally, the cfg file which we use that can not go above 20Hz is attached below. Note that with this if I run it with out triggering (i.e. trigger mode off), then the system status will say/ up to 50FPS (if I set it to that) where as the disparity map preview will max out at 20FPS. However if I use software or hardware triggering the FPS will always max out at 20 in both the system status and the preview. Does this help isolate the problem?

Re: SSP Troubleshooting

Posted: Mon Jun 17, 2019 12:41 pm
by k.schauwecker
Hi Kieran,

can you check that both cameras appear with "USB Speed Mode: SuperSpeed" in the advance cameras ettings (see screenshot). If it says HighSpeed for one camera then this would explain your frame rate problem. Did you leave the camera modules in the Karmin2 enclosure or put them in your own enclosure? The USB cables going to the camera modules are rather sensitive. Twisting them a little too much can cause such problems.

Regarding the point cloud: I think the key problem is that the rear cone is too small in the image. Please try disabling the speckle filter (in the latest firmware this is done by setting the speckle filter iterations to 0). This might prevent prevent filtering on the small cone, but it will also increase your artifacts in the point cloud.