Navigate your LinkedIn feed using Computer vision

In this article we will explore how you can control social media feed -not only Linkedin- or do a specific action on it with Computer vision.

I invite you to follow Programming Hero on Youtube, which inspired me to make this video.

To make this project, we first need a camera, IDE(Integrated development environment), and Python, you need to install some package on Python and you are ready to go!

Phase I | Packages :

What is openCV :

• "OpenCV-Python?is a library of Python bindings designed to solve computer vision problems."[1]
• More information about cv2 : https://pypi.org/project/opencv-python/
• You can open your terminal or IDE to install it with the command "pip install opencv-python"

What is Numpy ? :

• It's a Python library used for working with arrays, you can find more information here : https://numpy.org/
• You can install it by writing "pip install numpy"

What is Pyautogui ? :

We will use it to let Python scripts control the mouse and the keyboard to automate interactions, ideal for bots! ??

We've all had times like this, haven't we?...??

Image source ?? : https://www.dofus.com/fr/forum/1769-moderation/2311668-sujet-unique-bots-dofus-retro?page=12

• So when you install everything, start importing the package

import cv2
import numpy as np
import pyautogui

        

Phase II | Set up the camera:

• Next we need to create a variable to start the video

cap = cv2.VideoCapture(0)?        

• If you have multiple webcam on your computer, maybe you need to change the value from (0) to (1)
• Now we need to create a while loop to keep our script active.
• Don't forget to close it! If you select the window of the camera and press 'q' on your keyboard, you will close your windows (break the loop)

cap = cv2.VideoCapture(0)?


while True:
? ? ret, frame = cap.read()?
? ? cv2.imshow('frame',frame)
? ? if cv2.waitKey(10) == ord('q'):
? ? ? ? break


cap.release()
cv2.destroyAllWindows()        

Phase III | Detecting a specific object

• We now need to detect a specific object in our webcam, the ideal would be to detect a specific color.
• We are going to use black in this example( we will change it later because it's not a good practice when you wear black shirt and have a dark hair...)
• We will also take into consideration that it is not enough to take only the color "black", but its shades too, because the light can play a role in the color change of your object (in addition lighter or darker black)

Image source : https://gfycat.com/discover/lighting-and-shadows-gifs

• For black it will be (we will change it later)

lower_black = np.array([0, 0, 0]
upper_black = np.array([350,55,100]))        

• We also need to convert the color from BGR to HSV to detect shades and create a mask to only view the prefered color and its shades

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, lower_black, upper_black)        

• We will create another window to show the mask

 cv2.imshow('frame', frame)        

• As you can see, the mask detects only the shades of black, the white card is totally absent in the picture.

Phase IV | Get rid of the noise :

• Sometimes you can have noise in the detection, it can detect some shades of color that are not truly the one you chose.
• We need to define a contour in the mask, it will be an external borders, we selected the method of finding contour, we used "CHAIN_APPROX_SIMPLE".
• In this method we don't need all the points to define a contour of an object like in the "CHAIN_APPROX_NONE" technique, but only the end point of a line and simulate the line.
• You can see the difference between the 2 techniques; on the left it's CHAIN_APPROX_NONE with 734 points and on the right it's the CHAIN_APPROX_SIMPLE with only 4 points

Image source : https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html#:~:text=To%20draw%20the%20contours%2C%20cv,useful%20when%20drawing%20individual%20contour.

• Of course we will use the CHAIN_APPROX_SIMPLE technique ??

contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)        

• Let's look at those contours first! We need to draw them in the frame window by selecting our variable for contours (the way we want them).
• You need to add an index for it, you also need to define the color, it's defined by 3 numbers (RGB) for us it will be (0,0,0) (black; it's better in that case to take another color, we will change it later), finally we need to add the thickness of the line, I put 2

• Please don't wear black and choose a black color for detection with black contours, that's not a good practice...??
• We will change the outline to green ! ??

• Better right ?! But we have a lot of noises.
• We don't need a mask window anymore at this time, we have our contour.
• Now we need to loop through this outline, calculate and detect the area then print it.
• The small values are noises, we need to delete them from the capture.

? ? for x in contours:
? ? ? ? area = cv2.contourArea(x)
? ? ? ? print(x)        

• We will add condition, we will print area only if it's bigger than 1000, we will focus only on the big value to eliminate the noise.
• Note: For best detection I set yellow as the trigger color and the counter as green.You can change those values.

? ? for x in contours:
? ? ? ?area = cv2.contourArea(x)
? ? ? ?if area > 1000:
          cv2.drawContours(frame, contours, -1, (0,255,0),2)
        

• For better precision, we will draw the contour when we have the parameter of the rectangle (the corners)

x, y, w, h = cv2.boundingRect(x)
cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 3))
        

Phase V | Configuration of the command:

• If we want to scroll we need to move down the object, so first, we need to identify the initial position, "y" would refer to the initial position

y = 0        

• If the new position is less than the last one, we will use pyautogui to press "space" from the keyboard
• Every time we change the position we need to reset it as the initial one, for that we need to do

prev_y = y        

That's it you are done ??

• Here is the full code

import cv2
import numpy as np
import pyautogui


cap = cv2.VideoCapture(0)


yellow_lower = np.array([22, 93, 0])
yellow_upper = np.array([45, 255, 255])
prev_y = 0


while True:
? ? ret, frame = cap.read()
? ? hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
? ? mask = cv2.inRange(hsv, yellow_lower, yellow_upper)
? ? contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)


? ? for x in contours:
? ? ? ? area = cv2.contourArea(x)
? ? ? ? if area > 1000:
? ? ? ? ? ? x, y, w, h = cv2.boundingRect(x)
? ? ? ? ? ? cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
? ? ? ? ? ? if y < prev_y :
? ? ? ? ? ? ? ? pyautogui.press('space')


? ? ? ? ? ? prev_y = y

? ? cv2.imshow('frame', frame)
? ? if cv2.waitKey(10) == ord('q'):
? ? ? ? break


cap.release()
cv2.destroyAllWindows()        

You can use the original one from Programming Hero here: https://github.com/ProgrammingHero1/webcamfun/blob/master/main.py

Restrictions :

• Good Ram
• Good Camera
• Good light

Reference :

1 - GeeksforGeeks. (2019, August 2). Python OpenCV | cv2.imread() method. https://www.geeksforgeeks.org/python-opencv-cv2-imread-method/#:%7E:text=OpenCV%2DPython%20is%20a%20library,method%20returns%20an%20empty%20matrix.