Hi Ramin,
Thanks, I will try the things you mentioned. I will only have access to the camera next week again but will get back to you then. Also, thanks preparing a debug build.
Best,
Sven
Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Hi Ramin,
I checked a few things you mentioned in your last reply.
1. I ran the camera with nvcom on my laptop and everything seems to work fine. One thing I noticed was that the software showed an update rate of 6-7fps, even though the camera was set to 20 Hz (might be due to my limited hardware though). I ran nvcom for a while and the data stream was stable.
2. I reran the ros node with TCP. You were right, I did not properly reconfigure the launch script last time. Changing to TCP seemed to make another big improvement regarding the issue I am facing. The node now ran significantly longer. However, after 1-2 minutes, it still got stuck, frames were timing out and the node froze with a CPU load of 100%. I am attaching another export from the camera. I tried to close all my browser instances before downloading it this time, I hope it worked.
Thanks again for your help!
I checked a few things you mentioned in your last reply.
1. I ran the camera with nvcom on my laptop and everything seems to work fine. One thing I noticed was that the software showed an update rate of 6-7fps, even though the camera was set to 20 Hz (might be due to my limited hardware though). I ran nvcom for a while and the data stream was stable.
2. I reran the ros node with TCP. You were right, I did not properly reconfigure the launch script last time. Changing to TCP seemed to make another big improvement regarding the issue I am facing. The node now ran significantly longer. However, after 1-2 minutes, it still got stuck, frames were timing out and the node froze with a CPU load of 100%. I am attaching another export from the camera. I tried to close all my browser instances before downloading it this time, I hope it worked.
Thanks again for your help!
- Attachments
-
- settings(2).tgz
- (28.84 KiB) Downloaded 70 times
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Hi Sven,
thank you, this was quite enlightening. I had hoped that TCP might be free of issues (there are sometimes problems with high-volume UDP transfers, in hard-to-predict configurations). But this might be in fact a bug / regression, and I have a suspicion.
a) You can try the "issue-t98" branch (from my ROS2 repo) for a quick regression check (it uses a prior version of the visiontransfer library), I would be very interested if that one behaves differently (UDP and TCP).
b) You can try with lower framerates (let's say, start low at 3 or 4) and see if that is stable (UDP and TCP).
c) I saw in your code fragment earlier that you had a different timeout on the receive function (likely from your own testing) -- this just to make sure that you use the original repo code for testing, so we have as identical a setup as possible.
Today I do not have a Scarlet here, so any further firmware logging steps would follow tomorrow after testing there. (I was not able to reproduce the described behavior on any of our systems thus far, but I want to give you a nice clean build.)
As for your observation of lower frame rate, this is likely an Ethernet bandwidth issue (although if the client CPU is quite slow this might also affect it). If you happen to just need just the disparity, you can try to disable the image output channels in the web interface; from the device processing point of view those don't matter much, but they do eat up a lot of bandwidth.
Just for the sake of completeness and reference, it would be nice to have the rough system specs of the client PC. Also you wrote you are running ROS 2 Galactic; I have to admit that I recently tested on its successor Humble only (but IMO it is quite unlikely a-priori that this is the cause).
Best regards,
Ramin
thank you, this was quite enlightening. I had hoped that TCP might be free of issues (there are sometimes problems with high-volume UDP transfers, in hard-to-predict configurations). But this might be in fact a bug / regression, and I have a suspicion.
a) You can try the "issue-t98" branch (from my ROS2 repo) for a quick regression check (it uses a prior version of the visiontransfer library), I would be very interested if that one behaves differently (UDP and TCP).
b) You can try with lower framerates (let's say, start low at 3 or 4) and see if that is stable (UDP and TCP).
c) I saw in your code fragment earlier that you had a different timeout on the receive function (likely from your own testing) -- this just to make sure that you use the original repo code for testing, so we have as identical a setup as possible.
Today I do not have a Scarlet here, so any further firmware logging steps would follow tomorrow after testing there. (I was not able to reproduce the described behavior on any of our systems thus far, but I want to give you a nice clean build.)
As for your observation of lower frame rate, this is likely an Ethernet bandwidth issue (although if the client CPU is quite slow this might also affect it). If you happen to just need just the disparity, you can try to disable the image output channels in the web interface; from the device processing point of view those don't matter much, but they do eat up a lot of bandwidth.
Just for the sake of completeness and reference, it would be nice to have the rough system specs of the client PC. Also you wrote you are running ROS 2 Galactic; I have to admit that I recently tested on its successor Humble only (but IMO it is quite unlikely a-priori that this is the cause).
Best regards,
Ramin
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Hi Ramin,
Great, thanks for the detailed answer. I'll test all the stuff tomorrow and get back to you, including a description of the hardware of our system.
Best,
Sven
Great, thanks for the detailed answer. I'll test all the stuff tomorrow and get back to you, including a description of the hardware of our system.
Best,
Sven
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Hi Ramin,
I already have a first update regarding our system specs. We are running the node (and camera) from a robot, whose PC has these specs:
Intel(R) Core(TM) i5-6440EQ CPU @ 2.70GHz
Intel HD Graphics 530
8GB RAM
We have several other sensors connected to the robot/PC at the same time (e.g. lidar) and put the camera on its own subnet (to rule out networking issues).
I also tested the code again today with a lower framerate (2fps), which seems to work fine on first sight. I will do further testing and update you. I was also wondering if eventually a quick/videochat call could make sense as it might make communication easier and help accelerate solving the issue.
Best,
Sven
I already have a first update regarding our system specs. We are running the node (and camera) from a robot, whose PC has these specs:
Intel(R) Core(TM) i5-6440EQ CPU @ 2.70GHz
Intel HD Graphics 530
8GB RAM
We have several other sensors connected to the robot/PC at the same time (e.g. lidar) and put the camera on its own subnet (to rule out networking issues).
I also tested the code again today with a lower framerate (2fps), which seems to work fine on first sight. I will do further testing and update you. I was also wondering if eventually a quick/videochat call could make sense as it might make communication easier and help accelerate solving the issue.
Best,
Sven
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Yes, makes sense. I will be out of office this Monday, but I can organize something and let you know early Tuesday. Thanks for your patience and have a good weekend for now.
Re: Scarlet: Nerian Stereo ROS2 utilizes 100% CPU and freezes
Sounds good. Thursday would be ideal for me, as I will have easy access to the camera all day.