In this example we’ll talk about Network Programming with Python. By using Python 3.4 and its powerful socket module, at the end of this example we’ll have a messenger working.
First of all, let’s make it very clear. Sockets are behind any kind of network communications done by your computer. We all know that, when you type a url in your browser, it’s connecting to a server somewhere in the world which receives a request, returns a response and then the connection is closed. All of this is made through sockets, always. It doesn’t matter if your are communicating through HTTP, UDP or whatever, there is a socket behind every one of those protocols.
Let’s have a quick overview of what a socket is: A socket is one endpoint of a two-way communication link between two programs running on the network.
Fancy, huh? Well, it’s not that hard. You can see sockets as an interface to communicate with another application, whether it is on the same computer or not. It’s just something that, given a connection, sends or receives some data through it. As the definition says, to send data through a socket, there must be another socket waiting for it at the other end of the line.
Now, let’s differentiate two types of sockets here: client sockets and server sockets.
- Client sockets: When you need to connect to a server and make an exchange, you open a client socket. Keep in mind that client sockets should live for only one exchange, and then be closed. If we could see how our browsers work (we actually can, but won’t), we’ll see that when we download a page, there is a socket being opened for every resource the page needs to be downloaded. No socket will be used to download two stylesheets, the browser will open a socket for each one of them.
- Server sockets: In a server, to listen for connections from clients, we open a server socket and, instead of connecting to a host and port, we bind it to our host and some port in the machine. When a connection is received, it is not the server socket the one who receives the request and returns the response, the server socket only opens a client socket and connects it to the other end through a dynamically assigned port which will be recycled after the connection is closed.
To see the basics of how to work with sockets, you can see this example and then come back to start working with our messenger application.
1. A P2P Messenger
1.1. The Concept
An extremely simple messenger has two components, a sender and a receiver.
The sender is the one responsible of sending messages. In most cases, it’s a component which, every time the user wants to send a message, opens a client socket, connects it to the recipient of the message, actually sends the message and then closes the socket.
The receiver is the one responsible of receiving messages. It will be a component which binds a server socket to listen to connections and receive the data the client is sending.
These two components should run on separated threads, as both will have to run inside a while True, for the receiver will have to accept connections as long as the user wants, and the sender will have to read user’s input as long as the application is running.
1.2. The Code
So, what will we use to make this work? Well, as I said before, I’ll use Python 3.4 and only two native modules: socket and threading. I’ll give an overview of the threading module, to make the example richer and simpler, but if you don’t know something already this example would be a nice read for you.
We will have two classes: Receiver and Sender. Both of them will extend threading.Thread.
Let’s see the code of the Receiver first:
messenger.py
...
class Receiver(threading.Thread):
def __init__(self, my_host, my_port):
threading.Thread.__init__(self, name="messenger_receiver")
self.host = my_host
self.port = my_port
def run(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.host, self.port))
sock.listen(10)
while True:
connection, client_address = sock.accept()
try:
full_message = ""
while True:
data = connection.recv(16)
full_message = full_message + data.decode(ENCODING)
if not data:
print("{}: {}".format(client_address, full_message.strip()))
break
finally:
connection.shutdown(2)
connection.close()
...
As you can see, it’s a class that extends threading.Thread and defines two methods: __init__ and run.
The first one, __init__, is the method a class executes when instantiated. So, to instantiate this class we would run something like Receiver("localhost", 8881). This method will only receive the host and port, and assign them to properties of the class.
The second method, run, is the one where the magic happens. It creates a server socket, bind it to the address provided on the initialization and define a maximum number of parallel connections of 10. Then, it enters a while True to accept connections as long as the thread is alive (we won’t worry about gracefully quitting the application). Every time this socket accepts a connection, the accept method will return a socket, which will be used to receive and send information, and the address of the client (see? The server socket only creates more sockets, it does not actually send or receive any data itself). Once we get this socket, which we called connection, we receive data until the connection is closed (the inner while True), and when the socket stops receiving data, we print the message we received. Of course, when the exchange is done, we close our end of the connection.
Now, let’s have a look at the Sender class:
messenger.py
...
class Sender(threading.Thread):
def __init__(self, my_friends_host, my_friends_port):
threading.Thread.__init__(self, name="messenger_sender")
self.host = my_friends_host
self.port = my_friends_port
def run(self):
while True:
message = input("")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.host, self.port))
s.sendall(message.encode(ENCODING))
s.shutdown(2)
s.close()
...
Sender is also a class that extends threading.Thread, and as such, also defines __init__ and run.
The __init__ method has the same behaviour of Receiver’s, but the host and port are not the local ones to bind a server socket to, but the client’s one. We will connect to that endpoint.
In the run method, we also have a while True, that receives an input from the user, opens a client socket and sends the message through it. Of course, after the exchange, we close the socket.
Now, in a main method we will ask for the endpoints and start messaging:
messenger.py
...
def main():
my_host = input("which is my host? ")
my_port = int(input("which is my port? "))
receiver = Receiver(my_host, my_port)
my_friends_host = input("what is your friend's host? ")
my_friends_port = int(input("what is your friend's port?"))
sender = Sender(my_friends_host, my_friends_port)
treads = [receiver.start(), sender.start()]
...
There we are requesting for the endpoint to bind the server to and the endpoint of the recipient of the message to connect to it. We instantiate both Receiver and Sender and start them. We store the threads in an array in case we want to do some stuff with them.
There are a couple things to notice though. For example, we are not sending or receiving any response from the messages, the sender sends the message and closes the socket, does not expect anything back. Also, the receiver receives the message an closes the socket, if the client expects a response, it will wait forever (or for a really really long time). I don’t do it because it isn’t constructive to make the point, but we could use it to notify the sender that the message has arrived, and the user could see something like “your message has arrived”.
Also, we are not closing the server socket, which in modern operating systems is not a problem, as the resources will be released as the main process stops, but we should write something to quit the application gracefully and close the socket correctly.
So, this code, altogether, will look like this:
messenger.py
import socket
import threading
ENCODING = 'utf-8'
class Receiver(threading.Thread):
def __init__(self, my_host, my_port):
threading.Thread.__init__(self, name="messenger_receiver")
self.host = my_host
self.port = my_port
def listen(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.host, self.port))
sock.listen(10)
while True:
connection, client_address = sock.accept()
try:
full_message = ""
while True:
data = connection.recv(16)
full_message = full_message + data.decode(ENCODING)
if not data:
print("{}: {}".format(client_address, full_message.strip()))
break
finally:
connection.shutdown(2)
connection.close()
def run(self):
self.listen()
class Sender(threading.Thread):
def __init__(self, my_friends_host, my_friends_port):
threading.Thread.__init__(self, name="messenger_sender")
self.host = my_friends_host
self.port = my_friends_port
def run(self):
while True:
message = input("")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.host, self.port))
s.sendall(message.encode(ENCODING))
s.shutdown(2)
s.close()
def main():
my_host = input("which is my host? ")
my_port = int(input("which is my port? "))
receiver = Receiver(my_host, my_port)
my_friends_host = input("what is your friend's host? ")
my_friends_port = int(input("what is your friend's port?"))
sender = Sender(my_friends_host, my_friends_port)
treads = [receiver.start(), sender.start()]
if __name__ == '__main__':
main()
Now we execute it in two independent terminals ($> python3 messenger.py), in one we will only fill the “my host” and “my port” prompts, as we want a server socket to be available before starting the first client. In the second one we fill everything, and then go back to the first and fill the client data. One console will look like this:
which is my host? localhost
which is my port? 8881
what is your friend's host? localhost
what is your friend's port?8882
hi!
('127.0.0.1', 47893): hello! how are you?
I'm fine! I love this messenger app. It's so simple. I LOVE it.
('127.0.0.1', 47897): Yeah! It's awesome, and it's written in Python! I LOVE PYTHON!
And, the second one, will look like this:
which is my host? localhost
which is my port? 8882
what is your friend's host? localhost
what is your friend's port?8881
('127.0.0.1', 44624):
('127.0.0.1', 44625): hi!
hello! how are you?
('127.0.0.1', 44627): I'm fine! I love this messenger app. It's so simple. I LOVE it.
Yeah! It's awesome, and it's written in Python! I LOVE PYTHON!
The messages containing an endpoint, are the ones received, and the other ones are sent messages. Now there is something really interesting. We are printing the client endpoint when we receive a message, notice anything weird? Yeah, the ports are never the same. Client sockets are bound to ports that the system provides dynamically and then recycled once the connection is closed (as we explained before). Another thing you might notice is the first empty message… I pressed enter accidentally on the first console, haha.
This example was tested with localhost as I only have one computer connected to my network at home but, of course, if you have 2 machines connected to the same network, you can provide IPs instead of “localhost” and you’ll see the same results.
Here it is. A very simple P2P (Peer to Peer) messenger. We now have a pretty good idea of how to handle sockets and send data through a network, this example should be useful for any kind of connection you need to make between applications, but to make it more useful, we’ll write something a lot more interesting now.
2. Something Bigger
2.1. The Concept
We are going to make some modifications to this messenger to avoid using P2P, as it’s not a very popular protocol. Also, this modifications will allow us to write a chat room, instead of a conversation between two users.
Our chat room application will consist actually of two applications: a client and a server.
The server will receive connections from new users, save their information and receive messages to forward to all users present in the chat room. It will define a repository of users an a queue of messages. In one thread, we’ll have a server socket, and when a connection arrives, if it’s a connection request (some standard we’ll define, because we can) it will add a user to the repository, and if it’s a message it will parse and format the message and add it to the queue. On another thread, we’ll have a process that checks if the queue is not empty, and if it isn’t, it will pop a message out of it and send it to all the users (except the sender of course).
The client will be pretty similar to our P2P messenger, but we need to add support for a user name and fix the messages that don’t apply to the problem now.
2.2. The Code
Let’s first have a look at the repositories in our messenger server:
messenger/server/repository.py
class Queue:
def __init__(self):
self.queue = []
def add_message(self, sender, message):
self.queue.append({"sender": sender, "message": message})
def messages_waiting(self):
return len(self.queue) > 0
def pop_message(self):
return self.queue.pop()
class Users:
def __init__(self):
self.users = []
def add_user(self, name, host, port):
self.users.append({"name": name, "host": host, "port": int(port)})
def all_users(self):
return self.users
As you can see, it is pretty simple. Both repositories are containers of an array, and just expose convenient methods to access the arrays within them. Now, let’s jump to the server.
The server, as we said, has two components, Receiver and Sender, both of them extend threading.Thread.
The Receiver is a thread that starts a server socket, receives messages and resolves them (decides whether it’s a new user or an actual message):
messenger/server/messenger_server.py
...
class Receiver(threading.Thread):
def __init__(self, my_host, my_port, queue, users):
threading.Thread.__init__(self, name="messenger_receiver")
self.host = my_host
self.port = my_port
self.queue = queue
self.users = users
def resolve_message(self, message):
if "connect_request" in message:
print("connect request received")
arr = message.split("|")
if len(arr) == 4:
user_name = arr[1]
user_host = arr[2]
user_port = arr[3]
print("adding user: {}, {}, {}".format(user_name, user_host, user_port))
self.users.add_user(user_name, user_host, user_port)
else:
print("invalid connect request...")
elif "from:" in message:
print("received message... starting queueing process")
arr = message.split("|")
if len(arr) == 2:
sender_name = arr[0].replace("from:", "")
print("adding message from {} to queue".format(sender_name))
message_text = arr[1]
self.queue.add_message(sender_name, message_text)
def run(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.host, self.port))
sock.listen(10)
while True:
connection, client_address = sock.accept()
print("connection received from {}".format(str(client_address)))
try:
full_message = ""
while True:
data = connection.recv(16)
full_message = full_message + data.decode(ENCODING)
if not data:
print("received message: [{}]".format(full_message))
self.resolve_message(full_message.strip())
break
finally:
print("exchange finished. closing connection")
connection.shutdown(2)
connection.close()
...
Now, let’s talk in detail about what is going on here. As every class that extends threading.Thread, it defines __init__ and run.
On initialization this class will only receive an endpoint (host and port) to bind the server socket to and our queue and users repository.
On the run method, it starts the server sockets and enters a while True where it will accept connections and receive messages. When a message arrives it throws them to another method called resolve_message that will decide whether it’s a new user or an actual message that it has to put in our queue to forward it to all connected users. The decision is pretty simple, we defined a standard where a new user should send a message containing “connect_request|-user_name-|-client_host-|-client_port-“, and a message should contain “from:-user_name-|-message_text-“. Every client that meets this standards will be able to use our chat room without any problem.
Now, when do the messages actually get sent to other clients? Let’s see the Sender now:
messenger/server/messenger_server.py
...
class Sender(threading.Thread):
def __init__(self, queue, users):
threading.Thread.__init__(self, name="messenger_sender")
self.queue = queue
self.users = users
def send(self, message, host, port):
print("sending message {} to {}".format(message, str((host, port))))
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.connect((host, port))
s.sendall(message.encode(ENCODING))
finally:
print("message has been sent. closing connection")
s.shutdown(2)
s.close()
def run(self):
while True:
if self.queue.messages_waiting():
print("there are messages on queue. popping one")
message = self.queue.pop_message()
users = self.users.all_users()
for user in users:
if user.get("name") != message.get("sender"):
self.send("{}: {}".format(message.get("sender"), message.get("message")), user.get("host"), user.get("port"))
...
This class also extends threading.Thread. On initialization it will only require our queue and the user repository. In the run method, we just enter a while True and check if there is a message waiting in our queue. If there is any, it will pop one, get all connected users, and send the message it popped to every user that is not the sender. The send process is performed in a function called send that will open a socket, connect to the recipient endpoint and just send the text.
Then, on a main method…
messenger/server/messenger_server.py
def main():
queue = repository.Queue()
users = repository.Users()
host = input("host: ")
port = int(input("port: "))
receiver = Receiver(host, port, queue, users)
sender = Sender(queue, users)
threads = [receiver.start(), sender.start()]
…we initialize our queue and users repository, ask for the endpoint to bind the server to, and just start our two threads. This code altogether looks like:
messenger/server/messenger_server.py
import socket
import threading
import repository
ENCODING = 'utf-8'
class Receiver(threading.Thread):
def __init__(self, my_host, my_port, queue, users):
threading.Thread.__init__(self, name="messenger_receiver")
self.host = my_host
self.port = my_port
self.queue = queue
self.users = users
def resolve_message(self, message):
if "connect_request" in message:
print("connect request received")
arr = message.split("|")
if len(arr) == 4:
user_name = arr[1]
user_host = arr[2]
user_port = arr[3]
print("adding user: {}, {}, {}".format(user_name, user_host, user_port))
self.users.add_user(user_name, user_host, user_port)
else:
print("invalid connect request...")
elif "from:" in message:
print("received message... starting queueing process")
arr = message.split("|")
if len(arr) == 2:
sender_name = arr[0].replace("from:", "")
print("adding message from {} to queue".format(sender_name))
message_text = arr[1]
self.queue.add_message(sender_name, message_text)
def run(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.host, self.port))
sock.listen(10)
while True:
connection, client_address = sock.accept()
print("connection received from {}".format(str(client_address)))
try:
full_message = ""
while True:
data = connection.recv(16)
full_message = full_message + data.decode(ENCODING)
if not data:
print("received message: [{}]".format(full_message))
self.resolve_message(full_message.strip())
break
finally:
print("exchange finished. closing connection")
connection.shutdown(2)
connection.close()
class Sender(threading.Thread):
def __init__(self, queue, users):
threading.Thread.__init__(self, name="messenger_sender")
self.queue = queue
self.users = users
def send(self, message, host, port):
print("sending message {} to {}".format(message, str((host, port))))
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.connect((host, port))
s.sendall(message.encode(ENCODING))
finally:
print("message has been sent. closing connection")
s.shutdown(2)
s.close()
def run(self):
while True:
if self.queue.messages_waiting():
print("there are messages on queue. popping one")
message = self.queue.pop_message()
users = self.users.all_users()
for user in users:
if user.get("name") != message.get("sender"):
self.send("{}: {}".format(message.get("sender"), message.get("message")), user.get("host"), user.get("port"))
def main():
queue = repository.Queue()
users = repository.Users()
host = input("host: ")
port = int(input("port: "))
receiver = Receiver(host, port, queue, users)
sender = Sender(queue, users)
threads = [receiver.start(), sender.start()]
if __name__ == '__main__':
main()
Now, the code of our client is pretty much the same as it was for our P2P messenger. For a client of this server, it will be transparent whether it is a room chat or a one on one conversation, and actually the old P2P messenger, connected to the server will work just as we want it to, but there will be no info on who was the sender of the message. Let’s look how it looks now:
messenger/client/messenger_client.py
import socket
import threading
ENCODING = 'utf-8'
class Receiver(threading.Thread):
def __init__(self, my_host, my_port):
threading.Thread.__init__(self, name="messenger_receiver")
self.host = my_host
self.port = my_port
def listen(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.host, self.port))
sock.listen(10)
while True:
connection, client_address = sock.accept()
try:
full_message = ""
while True:
data = connection.recv(16)
full_message = full_message + data.decode(ENCODING)
if not data:
print(full_message.strip())
break
finally:
connection.shutdown(2)
connection.close()
def run(self):
self.listen()
class Sender(threading.Thread):
def __init__(self, server_host, server_port, user_name, local_host, local_port):
threading.Thread.__init__(self, name="messenger_sender")
self.host = server_host
self.port = server_port
self.user_name = user_name
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.host, self.port))
s.sendall("connect_request|{}|{}|{}".format(user_name, local_host, local_port).encode(ENCODING))
s.shutdown(2)
s.close()
def run(self):
while True:
message = input("")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.host, self.port))
s.sendall("from:{}|{}".format(self.user_name, message).encode(ENCODING))
s.shutdown(2)
s.close()
def main():
my_host = input("host: ")
my_port = int(input("port: "))
receiver = Receiver(my_host, my_port)
server_host = input("server host: ")
server_port = int(input("server port: "))
user_name = input("user name: ")
sender = Sender(server_host, server_port, user_name, my_host, my_port)
treads = [receiver.start(), sender.start()]
if __name__ == '__main__':
main()
Let’s see one by one the differences between our first example and the client of our chat room server:
- The prompts changed, as the endpoint we ask for is no longer your friend’s, but the server’s.
- We are now asking for a user name.
- The sender now receives both the endpoint from the remote server and the local one on initialization. Also, it receives the user name now.
- Also, on initialization, the
Senderwill send a message to the server for it to acknowledge the new user. - Messages are now formatted as our standard defines.
The rest looks the same, take your time reading the code, and make sure you understand every line. If you accomplish that goal, you will be able to use what you learnt here for any network related problem!
Let’s have a look to our application working. Let’s start the server on a console and two clients on two independent consoles. Below is the output after the two clients connect and send a couple messages:
server’s output
host: localhost
port: 8880
connection received from ('127.0.0.1', 44009)
received message: [connect_request|first client|localhost|8881]
connect request received
adding user: first client, localhost, 8881
exchange finished. closing connection
connection received from ('127.0.0.1', 44010)
received message: [connect_request|second client|localhost|8882]
connect request received
adding user: second client, localhost, 8882
exchange finished. closing connection
connection received from ('127.0.0.1', 44011)
received message: [from:second client|HI!]
received message... starting queueing process
adding message from second client to queue
exchange finished. closing connection
there are messages on queue. popping one
sending message second client: HI! to ('localhost', 8881)
message has been sent. closing connection
connection received from ('127.0.0.1', 44013)
received message: [from:first client|Hi! Love this chat room!]
received message... starting queueing process
adding message from first client to queue
exchange finished. closing connection
there are messages on queue. popping one
sending message first client: Hi! Love this chat room! to ('localhost', 8882)
message has been sent. closing connection
connection received from ('127.0.0.1', 44015)
received message: [from:second client|Yeah! It's so simple. ]
received message... starting queueing process
adding message from second client to queue
exchange finished. closing connection
there are messages on queue. popping one
sending message second client: Yeah! It's so simple. to ('localhost', 8881)
message has been sent. closing connection
connection received from ('127.0.0.1', 44017)
received message: [from:first client|Python is the best!]
received message... starting queueing process
adding message from first client to queue
exchange finished. closing connection
there are messages on queue. popping one
sending message first client: Python is the best! to ('localhost', 8882)
message has been sent. closing connection
first client’s output
host: localhost port: 8881 server host: localhost server port: 8880 user name: first client second client: HI! Hi! Love this chat room! second client: Yeah! It's so simple. Python is the best!
second client’s output
host: localhost port: 8882 server host: localhost server port: 8880 user name: second client HI! first client: Hi! Love this chat room! Yeah! It's so simple. first client: Python is the best!
Yes, that’s right, our chat room is working. Hurray!
3. Download the Code Project
This was a basic example on Network Programming with Python.
You can download the full source code of this example here: python-network-programming
I am coming across an error, Exception in thread messenger_receiver:
Traceback (most recent call last):
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
self.run()
File “pp.py”, line 33, in run
self.listen()
File “pp.py”, line 29, in listen
connection.shutdown(2)
OSError: [Errno 57] Socket is not connected