OpenCV (short for Open Source Computer Vision) is a library for computer vision, machine learning, and image processing. It can be used to identify patterns in an image, extract features and perform mathematical operations on it. The first step is to process the puzzle screenshot using OpenCV. Let's quickly review the basics of OpenCV image processing.
Install the Python OpenCV package
pip install opencv-python
How to upload an image
cv.imreadreads an image file and converts it to an OpenCV array. If the image cannot be read because the file may be missing or in a format that OpenCV cannot understand, an empty array is returned. OpenCV array can be converted to an image using cv.imshow function.
import cv2 as cv
import numpy as np# Reading an image
original = cv.imread(")
cv.imshow("original", original)
How to draw a line, circle, rectangle or text on the same image
Once we detect the grid, we need to recreate it using lines and place Qusing text. Let's see a short snippet to draw a line, a circle, a rectangle and a text in the above reading matrix.
# Drawing a line
line = cv.line(original, (original.shape(1)//2, original.shape(0)//2), (0,0) , (0,255,0), thickness=2)
cv.imshow("line", line)# Drawing other shapes
circle = cv.circle(line, (line.shape(1)//2, line.shape(0)//2), 50, (0,0,255), thickness=2)
rect = cv.rectangle(circle, (10,10), (circle.shape(1)//2, circle.shape(0)//2), (255,0,0), thickness=2)
text = cv.putText(rect, "Hi", (rect.shape(1)//2, rect.shape(0)//2), cv.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), thickness=2)
cv.imshow("all shapes", text)
How to detect contours
Contours are simply a shape that joins all points of similar color and intensity into a continuous boundary. They are useful for detecting shapes and analyzing the outline of objects. We will draw our puzzle grid by detecting the individual cells.
# Its best to convert image to grayscale
# and add a bit of blur for better contour detections
# since our image is mostly a grid we dont need blur# by default OpenCV reads images as BGR
# as opposed to traditional RGB
gray = cv.cvtConvert(original, cv.COLOR_BGR2GRAY)
contours, _ = cv.findContours(gray, cv.RETR_TREE, cv.CHAIN_APPROX_NONE)
By default, OpenCV reads images as BGRunlike the traditional RGB
Crop an image
So we can remove unnecessary areas from the screenshots and reduce noise, once we have detected our outlines.
# its essentially selecting the pixels we need from the entire image
cropped = original(0:original.shape(1)//2, 0:original.shape(0)//2)
cv.imshow("cropped", cropped)
First, we start by loading the image into memory and converting it to grayscale. This helps simplify contour detection, a general step that is always followed as it reduces image complexity. Next, we find contours, sort them and select the largest one. Typically, the first outline is the bound box of the original image, so we use the second largest outline to isolate the puzzle grid. Then we crop the image just to get the grid and nothing else. We find contours again, since the noise is now reduced, it will detect the grid better. We determine the number of cells within the grid and iterate over each cell, taking the average color and assigning a number of each color, giving us the 2D matrix of our puzzle.
# Read the input image and save the original
original = cv.imread(file_name)
cv.imwrite("solution/original.png", original)# Convert the image to grayscale
gray = cv.cvtColor(original, cv.COLOR_BGR2GRAY)
# Find contours in the grayscale image and sort them by area
contours, _ = cv.findContours(gray, cv.RETR_TREE, cv.CHAIN_APPROX_NONE)
contours = sorted(contours, key=cv.contourArea, reverse=True)
# Extract the bounding box of the puzzle grid (using the second largest contour)
x, y, w, h = cv.boundingRect(contours(1))
# Crop the grid area from the original image
grid = original(y:y+h, x:x+w)
cv.imwrite("solution/grid.png", grid)
# Convert the cropped grid to grayscale
gray = cv.cvtColor(grid, cv.COLOR_BGR2GRAY)
cv.imwrite("solution/gray-grid.png", gray)
# Find contours again in the cropped grayscale grid
contours, _ = cv.findContours(gray, cv.RETR_TREE, cv.CHAIN_APPROX_NONE)
contours = sorted(contours, key=cv.contourArea)
# Determine the total number of cells in the grid
total_cells = len(contours) - 2
grid_size = int(math.sqrt(total_cells))
# Check if the detected cells form a complete square grid
if total_cells != grid_size**2:
print("Unable to detect full grid! Aborting")
# Calculate individual cell dimensions
cell_width = w // grid_size
cell_height = h // grid_size
# Initialize color mappings and board representation
colors = ()
board = ()
color_index = 1
color_map = {}
reverse_color_map = {}
padding = 10
# Iterate through each cell in the grid
for i in range(grid_size):
row = ()
for j in range(grid_size):
# Calculate cell coordinates with padding
cell_x = j * cell_width
cell_y = i * cell_height
padding = 15
cell = grid(cell_y+padding:cell_y+cell_height-padding, cell_x+padding:cell_x+cell_width-padding)
# Get the average color of the cell
avg_color = cell.mean(axis=0).mean(axis=0)
avg_color = avg_color.astype(int)
avg_color = tuple(avg_color)
# Map the color to a unique index if not already mapped
if avg_color not in color_map:
color_map(avg_color) = str(color_index)
reverse_color_map(str(color_index)) = avg_color
color_index += 1
# Add the color index to the row
row.append(color_map(avg_color))
# Add the row to the board
board.append(row)
