I have two images
image1 (object):
]3
Original image without marked keypoints:
image2 is a white picture (500×500)
In image1 and in image2 I have marked keypoints. I want to align image1 on image2 by keypoints. So the goal that both keypoints overlaps with stretching, scaling and transforming image2.
This are my keypoints (csv file). The coordinates are x and y for image1 in image1 and for image2 in image2.
object1_x,object1_y,image_x,image_y 0,0,80,137 286,0,409,42 286,198,416,390 174,198,331,384 158,116,291,119 0,97,111,311
How can I do this with opencv and python? So the result image should looks like this (without the red dots, the red dots are only for demonstration the keypoints):
Advertisement
Answer
The Concept
Extract sets of 3 indices from the first set of keypoints that will form triangles when indexed from both sets of keypoints. With the indices we can get corresponding triangles from both sets of keypoints, allowing us to build the warped image triangle by triangle (see Warp one triangle to another using OpenCV for more details):
image1.png
(with added points):
image2.png
(with added points):
Result (with added points)
The Code
import cv2 import numpy as np def triangles(points): points = np.where(points, points, 1) subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0))) subdiv.insert(list(points)) for pts in subdiv.getTriangleList().reshape(-1, 3, 2): yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts] def crop(img, pts): x, y, w, h = cv2.boundingRect(pts) img_cropped = img[y: y + h, x: x + w] pts[:, 0] -= x pts[:, 1] -= y return img_cropped, pts def warp(img1, img2, pts1, pts2): for indices in triangles(pts1): img1_cropped, triangle1 = crop(img1, pts1[indices]) img2_cropped, triangle2 = crop(img2, pts2[indices]) transform = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2)) img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101) mask = np.zeros_like(img2_cropped) cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0) img2_cropped *= 1 - mask img2_cropped += img2_warped * mask img1 = cv2.imread("image1.png") img2 = cv2.imread("image2.png") pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]]) pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]]) warp(img1, img2, pts1, pts2) for pt in pts2: cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1) cv2.imshow("Original", img1) cv2.imshow("Transformed", img2) cv2.waitKey(0) cv2.destroyAllWindows()
The Output
The Explanation
- Import the necessary libraries:
import cv2 import numpy as np
- Define a function,
triangles
, that will take in an array of coordinates,points
, and yield lists of 3 indices of the array for triangles that will cover the area of the original array of coordinates:
def triangles(points): points = np.where(points, points, 1) subdiv = cv2.Subdiv2D((*points.min(0), *points.max(0))) subdiv.insert(list(points)) for pts in subdiv.getTriangleList().reshape(-1, 3, 2): yield [np.where(np.all(points == pt, 1))[0][0] for pt in pts]
- Define a function,
crop
, that will take in an image array,img
, and an array of three coordinates,pts
. It will return a rectangular segment of the image just large enough to fit the triangle formed by the three point, and return the array of three coordinates transferred to the top-left corner of image:
def crop(img, pts): x, y, w, h = cv2.boundingRect(pts) img_cropped = img[y: y + h, x: x + w] pts[:, 0] -= x pts[:, 1] -= y return img_cropped, pts
- Define a function,
warp
, that will take in 2 image arrays,img1
andimg2
, and 2 arrays of coordinates,pts1
andpts2
. It will utilize thetriangles
function defined before iterate through the triangles from the first array of coordinates, thecrop
function defined before to crop both images at coordinates corresponding to the triangle indices and use thecv2.warpAffine()
method to warp the image at the current triangle of the iterations:
def warp(img1, img2, pts1, pts2): for indices in triangles(pts1): img1_cropped, triangle1 = crop(img1, pts1[indices]) img2_cropped, triangle2 = crop(img2, pts2[indices]) transform = cv2.getAffineTransform(np.float32(triangle1), np.float32(triangle2)) img2_warped = cv2.warpAffine(img1_cropped, transform, img2_cropped.shape[:2][::-1], None, cv2.INTER_LINEAR, cv2.BORDER_REFLECT_101) mask = np.zeros_like(img2_cropped) cv2.fillConvexPoly(mask, np.int32(triangle2), (1, 1, 1), 16, 0) img2_cropped *= 1 - mask img2_cropped += img2_warped * mask
- Read in your images. In your case,
img1
is the image we want to warp, andimg2
is the blank 500 x 500 image. Also, define 2 array of coordinates to be the keypoints of the images:
img1 = cv2.imread("image1.png") img2 = cv2.imread("image2.png") pts1 = np.array([[0, 0], [286, 0], [286, 198], [174, 198], [158, 116], [0, 97]]) pts2 = np.array([[80, 37], [409, 42], [416, 390], [331, 384], [291, 119], [111, 311]])
- Finally, use the
warp
function defined before to warpimg1
to have its keypoints overlap with the kewpoints ofimg2
and show the resulting image. I drew the points from the second array of coordinates onto the resulting warped image to make the warping process easier to visualize:
warp(img1, img2, pts1, pts2) for pt in pts2: cv2.circle(img2, tuple(pt), 15, (0, 0, 255), -1) cv2.imshow("Original", img1) cv2.imshow("Transformed", img2) cv2.waitKey(0) cv2.destroyAllWindows()