360 Degree View form Multiple Point Cloud Registration

Hi There,

I would like to use pointcloud registration to stitch a 360* view of an object. I have captured 5 pointclouds from various perspectives. When I attempt to stitch (register) the pointclouds for a cohesive 360* view the registration algorithms (RANSAC, ICP) match the voids (depth shadow) rather than the features of the object. So I just end up with 5 superimposed poinclouds.

Does anyone know how to stitch pointclounds for a 360* view? I would imagine this process to be similar to photogrametry.

Thank you in advance for your help.

Sam0_pose 1_pose

4_pose 5_pose

Hi Ligthsolid,

So you are not met with resounding silence here…
I’ve been trying similar things at the sub-metre scale.

You say it is matching the depth shadows, rather than features?
What you mean is not obvious to me from your images.

However, if there are features of your point clouds that are resulting in eroneous matches, you could try removing them, completing ICP, then using the transformation matrix on the original source point cloud with the features included.

I am struggling with some images to get sensible ICP transformation matrices. Other than changing the threshold value (which makes an enormous difference, but seems to be difficult to do dynamically in code – ive had to do this interactively), you don’t have many levers to pull other than your
initial estimate of the transformation matrix.

One approach I have been experimenting with today is changing the order in which the images are fitted together – to give troublesome images a larger “target” cloud to fit to – and therefore a larger
number of legitimate correspondence points.


Hello runner/M,

Thank you for getting back to me.

Firstly, to clarify, each of the images I have posted is a point cloud from a different perspective of my object (a hollowed out stump). My intel RealSense camera only has 60* view. What the camera doesn’t see it just leaves as a void in the point cloud. What I am attempting to do is capture point clouds from different perspectives 72* apart (5 captures around the object) and stitch them together into a cohesive point cloud.

The first 5 images I posted are the individual point clouds spaced apart by 72*. The last image is the RANSAC registration of these 5 point clouds. As you can see, there is still a large void. So the registration of point clouds spaces apart by large angles doesn’t work.

Thank you for the suggestions, about removing the eroneous features. I believe my problems is that RANSAC and ICP registration cannot register point clouds with such large transformations. I am looking into the odometry class to determine what is the maximum transformation the 2 RGBD images can have before the registration falls apart.

I am open to any suggestions, comments and even light-hearted redicule.



A tree stump. I thought it was a mountain.

I actually don’t think it is the large transformation – I think it is having enough overlap in the images
with sufficient correspondence points that are unambiguous.

My continuing battles with ICP have revealed to me just how vulnerable the process is to being misled by “local minima”. So the initial guess transformation is critically important.
I have several hundred overlapping images (1m^2 of surface area for each image). I complete joining them together with ICP, but run an ongoing check on the sanity of the transformation matrix predicted.

I know my initial guesses can only be out by a couple of mm – so anything beyond a couple of degrees of rotation, and 10 mm of translation (total) is rubbish.

So I do a first pass through all the images, and compile a list of translations I don’t trust (they are not concatenated into the image). Then on a second pass, I try to fit the rejected images in . So on the second pass, there are far fewer gaps in the image and the rejected images now have more of a chance, as there are more correspondence points.

Does that make sense?

However, this is not perfect.
But it is better than what I had before.

I have solved many problems. I found the checkerboard I was calibrating the camera with was about 1% out in one dimension. That did not help! When I fixed that, matters improved considerably.
So check your camera calibration is perfect. I had to clip the edges of the image – as the calibration does not really reach all the way to the edge of the lens.

So, I think that if you have sufficient overlapping points, a good guess of the translation matrix to start with, then you can start optimising the threshold value to use. I’ve been looking for answers as to what the units of the threshold value are – but I can only currently guess.


Hi all,

If you take screenshots in Open3D, a camera matrix storing the transformation and intrinsics will also be saved in a json file. Do you think that would work?
You may easily load that file via param = o3d.io.read_pinhole_camera_parameters(json), and load the extrinsics via param.extrinsic