Video Chat App with Face detection using Python and OpenCV

Hello everyone. In this article I am going to explain about my python based Video chat application. The Python modules used are socket, cv2, numpy, and threading.

First of all, What is a video. A video is a series of images which is displayed so fast that can human mind thinks them as continuously moving scenes. Image is again a collection of pixels with are stored as a list of lists in a numpy array. So to create a video chat app we have to capture and transfer images at very high speed. Both these task are done by OpenCV and Socket modules respectively. To manage and manipulate images we use nunpy and OpenCV modules. Threading module is used to carry on the sending and receiving of images parallelly.

What is Socket Programming?

Socket programming is a way of connecting two nodes on a network to communicate with each other. One socket(node) listens on a particular port at an IP, while the other socket reaches out to the other to form a connection. The server forms the listener socket while client reaches out to the server.

What is Threading?

While executing a program a single statement is executed at a time. But to create a bidirectional video chat app we have both receive and send the images at same time. This problem is solved by concept of threading. Here we can send different parts of code to be executed in different parts of CPU, by putting them in different threads. Thus executing them parallelly, saves time and improves the performance of program.

Let's jump to the code

I have created two files in my system. Both files will act as server as well as client.

server1.py

####### SERVER 1 ########
import socket
import cv2, numpy ,threading


ss = socket.socket(socket.AF_INET , socket.SOCK_STREAM)? #server socket for sending
cs = socket.socket(socket.AF_INET , socket.SOCK_STREAM)? #client socket for recieving


ss.bind(("192.168.99.1" , 2022))
ss.listen(5)


c_s ,addr = ss.accept()? ? ? ? ? ? ? ? ? ? ? #accept connection request to server socket
print ("Connected to - " ,addr)


cs.connect(("192.168.99.1" , 3033))? ? ? ? ? # send request to server2 for connection
cs.settimeout(1)??


# loading haarcascade model to detect face
model = cv2.CascadeClassifier(cv2.data.haarcascades +'haarcascade_frontalface_default.xml')
flag = -1? ? ? ? ? ?
cap = cv2.VideoCapture(0)? ?# start capture video form local camera


def send():? ? ? ? ? ? ? ? ??
? ? global flag , cap
? ? while(True):
? ? ? ? if ( flag == 0):?
? ? ? ? ? ? break
? ? ? ? try:
? ? ? ? ? ? ret,photo=cap.read()? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #read images
? ? ? ? ? ? b_img = cv2.imencode(".jpg" , photo)[1].tobytes() # first encode and then convert to bytes
? ? ? ? ? ? c_s.sendall(b_img)? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? # send images
? ? ? ? except:
? ? ? ? ? ? continue
? ? cap.release()? ? ? #release camera after the function is done sending?
? ? ss.close()? ? ? ? ?# close the server (sending ) socket?
?


def receive():
? ? global flag , cap , model
? ? count=0
? ? while(True):
? ? ? ? if (count>10): break
? ? ? ? try:
? ? ? ? ? ? mess = cs.recv(100000)? ? ? ? ? ? ? ? ? ? ? ? #recieve 1 lack bytes of data
? ? ? ? ? ? if (mess):
? ? ? ? ? ? ? ? nparr = numpy.frombuffer(mess, numpy.uint8)? # retreive the array form bytes
? ? ? ? ? ? ? ? img1 = cv2.imdecode(nparr, cv2.IMREAD_COLOR) # create image from array
? ? ? ? ? ? ? ? if (not (img1 is None) ):? ? ?
? ? ? ? ? ? ? ? ? ? try:
? ? ? ? ? ? ? ? ? ? ? ? face =model.detectMultiScale(img1)? # detect face in img1
? ? ? ? ? ? ? ? ? ? ? ? if (len(face)==0):
? ? ? ? ? ? ? ? ? ? ? ? ? ? pass
? ? ? ? ? ? ? ? ? ? ? ? else:
? ? ? ? ? ? ? ? ? ? ? ? ? ? x1=face[0][0]
? ? ? ? ? ? ? ? ? ? ? ? ? ? y1=face[0][1]
? ? ? ? ? ? ? ? ? ? ? ? ? ? x2=face[0][2]
? ? ? ? ? ? ? ? ? ? ? ? ? ? y2=face[0][3]
? ? ? ? ? ? ? ? ? ? ? ? img1 = cv2.rectangle(img1,(x1,y1),(x1+x2,y1+y2),[255,255,255],3)? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ??
? ? ? ? ? ? ? ? ? ? except:
? ? ? ? ? ? ? ? ? ? ? ? pass
? ? ? ? ? ? ? ? ? ? # adding the small image in top left corner
? ? ? ? ? ? ? ? ? ? ret,photo1=cap.read()
? ? ? ? ? ? ? ? ? ? resized = cv2.resize(photo1, (150,150), interpolation = cv2.INTER_AREA)
? ? ? ? ? ? ? ? ? ? img1[ 10:160 , 10:160? ] = resized
? ? ? ? ? ? ? ? ? ? img1 = cv2.rectangle(img1, (0,0), (170,170), (0,0,250), 3)
? ? ? ? ? ? ? ? ? ? # showing the final images? ? ?
? ? ? ? ? ? ? ? ? ? cv2.imshow("From server2",img1)
? ? ? ? ? ? ? ? ? ? if cv2.waitKey(10) == 13:? ? ? ?# if user press enter break the loop
? ? ? ? ? ? ? ? ? ? ? ? break
? ? ? ? except:
? ? ? ? ? ? count+=1
? ? ? ? ? ? continue
? ? ? ? ? ??
? ? flag = 0
? ? cv2.destroyAllWindows()? # destroy the image window
? ? cs.close()? ? ? ? ? ? ? ?# close client (reciever) socket
? ??
# Putting the send and recieve functions in different thread so they can run in parallel
t1 = threading.Thread(target = receive)? ?
t2 = threading.Thread(target = send)
t1.start()? ? ? ?
t2.start()#        

First I have imported all the required modules.

In the next two line sockets are created for sending the receiving respectively. since one socket can be used for one purpose only.

The bind() function is used to bind the given port with the ip, it takes a tuple of (ip , port no.) as argument. Next there is a accept() method which help to establish connections form another user. It keeps on waiting until it gets a connection.

The connect() method sends request to the given ip and port to connect as a client. You will find that in "server2.py" file the connect() method is placed above the accept() method. This is because accept() method holds the program until it gets any one connection. So the network of connection will be established in the given sequence -

• server2 will send connection request to server2.
• server1 will accept the request.
• After the accept() method of server1 is executed , connect() method of server1 will make a request to server2.
• At last accept() of server2 will accept the request of server1.

The settimeout() method is used to set a time of execution for the recv() function, so that the program might not get struck while the recv() is waiting for data from other program.

In the above line I am loading a Machine Learning model to detect faces. flag is variable which helps to shut one of the function since they are running in different threads. The VideoCapture() method is used to establish connection form the device's camera. Here argument 0 means internal camera and 1 imply external camera.

In "server2.py" I have given 1 in the argument as two program running on windows can't use same camera because of security reasons.

Now we have send() function. This function is used to send the images to other server using the ss (server socket). I have placed the code in infinite while loop so that fucntion continuously keeps on sending images. read() method is used to capture images.

I had to use the global keyword because send() function will be running in a different thread.

To send any data over network we have to convert it to bytes format. In the next line imenocde() function is used to convert image format into streaming data and assign it to memory cache, then the tobytes() function convert the data to bytes format.

The sendall() method will send data to the connected socket. when the function is about to end cap.release() will disconnect the camera from program and ss.close() will destroy the socket.

In the receive() function also I have placed all the code in infinite while loop. cs.recv() method is used to receive data form the cs (client socket). count variable is used as a counter it will increase it's value if the main code in try block fails and will help to end the whole program successfully and avoid getting struck in while loop.

frombuffer() function convert the bytes data into numpy array. then cv2.imdecode() function extracts the original image from the array. While transferring data over network some part may go corrupt that's why I kept most of code in different try-except bocks so that whole program may not get affected by a single corrupt images.

if() checks weather the img1 is not None. cv2.detectMultiScale() method returns the coordinates of the face in the image. cv2.rectangle() method draws a while rectangle on the given coordinates.

In the above snippet the images being send to other program is captured and added to the top left corner of the received images. resize() method is used to change the shape and size of an image.

At the end of receive() function, imshow() method is used to display the final image.

cv2.waitKey() function captures the user's response key form the keyboard, if any. It will break the loop if user press enter.

In the end of receive() function flag variable will be assigned 0,this will end the send() function if it is still running. cv2.destroyAllWindows() method will destroy the window created to display the images.

Note: If one of the user stops or terminates it's program then other user's code in receive function won't work properly and except block will run again and again , the program will get struck in that position. To sort out this I have placed a counter variable "count" in except block which will increase up to 10 and until then if the first user won't make the connection again the program will be terminated. (As happened in the end of demonstration video)

At the end of code I have placed the functions send in receive in t2 and t1 threads respectively. Then started them using the start() method. Now the two functions will continue to run without interrupting each other.

server2.py

####### SERVER 2 ########
import socket
import cv2, numpy ,threading


ss = socket.socket(socket.AF_INET , socket.SOCK_STREAM)? #server socket for sending
cs = socket.socket(socket.AF_INET , socket.SOCK_STREAM)? #client socket for recieving


ss.bind(("192.168.99.1" , 3033))
ss.listen(5)


cs.connect(("192.168.99.1" , 2022))? ? # send request to server2 for connection


c_s ,addr = ss.accept()? ? ? ? ? ? ? ? ? ? ? #accept connection request to server socket
print ("Connected to - " ,addr)? ? ? ??
cs.settimeout(1)??


# loading haarcascade model to detect face
model = cv2.CascadeClassifier(cv2.data.haarcascades +'haarcascade_frontalface_default.xml')
flag = -1? ? ? ? ? ?
cap = cv2.VideoCapture(1)? ?# start capture video form external camera


def send():? ? ? ? ? ? ? ? ??
? ? global flag , cap
? ? while(True):
? ? ? ? if ( flag == 0):?
? ? ? ? ? ? break
? ? ? ? try:
? ? ? ? ? ? ret,photo=cap.read()? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #read images
? ? ? ? ? ? b_img = cv2.imencode(".jpg" , photo)[1].tobytes() # first encode and then convert to bytes
? ? ? ? ? ? c_s.sendall(b_img)? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? # send images
? ? ? ? except:
? ? ? ? ? ? continue
? ? cap.release()? ? ? #release camera after the function is done sending?
? ? ss.close()? ? ? ? ?# close the server (sending ) socket?
?


def receive():
? ? global flag , cap , model
? ? count=0
? ? while(True):
? ? ? ? if (count>10): break
? ? ? ? try:
? ? ? ? ? ? mess = cs.recv(100000)? ? ? ? ? ? ? ? ? ? ? ? #recieve 1 lack bytes of data
? ? ? ? ? ? if (mess):
? ? ? ? ? ? ? ? nparr = numpy.frombuffer(mess, numpy.uint8)? # retreive the array form bytes
? ? ? ? ? ? ? ? img1 = cv2.imdecode(nparr, cv2.IMREAD_COLOR) # create image from array
? ? ? ? ? ? ? ? if (not (img1 is None) ):? ? ?
? ? ? ? ? ? ? ? ? ? try:
? ? ? ? ? ? ? ? ? ? ? ? face =model.detectMultiScale(img1)? # detect face in img1
? ? ? ? ? ? ? ? ? ? ? ? if (len(face)==0):
? ? ? ? ? ? ? ? ? ? ? ? ? ? pass
? ? ? ? ? ? ? ? ? ? ? ? else:
? ? ? ? ? ? ? ? ? ? ? ? ? ? x1=face[0][0]
? ? ? ? ? ? ? ? ? ? ? ? ? ? y1=face[0][1]
? ? ? ? ? ? ? ? ? ? ? ? ? ? x2=face[0][2]
? ? ? ? ? ? ? ? ? ? ? ? ? ? y2=face[0][3]
? ? ? ? ? ? ? ? ? ? ? ? img1 = cv2.rectangle(img1,(x1,y1),(x1+x2,y1+y2),[255,255,255],3)? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ??
? ? ? ? ? ? ? ? ? ? except:
? ? ? ? ? ? ? ? ? ? ? ? pass
? ? ? ? ? ? ? ? ? ? # adding the image image being sent in top left corner
? ? ? ? ? ? ? ? ? ? ret,photo1=cap.read()
? ? ? ? ? ? ? ? ? ? resized = cv2.resize(photo1, (150,150), interpolation = cv2.INTER_AREA)
? ? ? ? ? ? ? ? ? ? img1[ 10:160 , 10:160? ] = resized
? ? ? ? ? ? ? ? ? ? img1 = cv2.rectangle(img1, (0,0), (170,170), (0,0,250), 3)
? ? ? ? ? ? ? ? ? ??
? ? ? ? ? ? ? ? ? ? # showing the final images? ? ?
? ? ? ? ? ? ? ? ? ? cv2.imshow("From server1.ipynb",img1)
? ? ? ? ? ? ? ? ? ? if cv2.waitKey(10) == 13:? ? ? ?# if user press enter break the loop
? ? ? ? ? ? ? ? ? ? ? ? break
? ? ? ? except:
? ? ? ? ? ? count+=1
? ? ? ? ? ? continue
? ? ? ? ? ??
? ? flag = 0
? ? cv2.destroyAllWindows()? # destroy the image window
? ? cs.close()? ? ? ? ? ? ? ?# close client (reciever) socket
? ??
# Putting the send and recieve functions in different thread so they can run in parallel
t1 = threading.Thread(target = receive)? ?
t2 = threading.Thread(target = send)
t1.start()? ? ? ?
t2.start()#        

server2.py is same as server1.py except for the order of accepting and sending connection request as mentioned earlier in the article.

Here is a Demonstration video of the above code

GitHub link -

Thank You for Reading............