Flask and Socket.IO

Tags python flask socketio
Posted by Logan

Under some scenarios, we would like to push a message from an HTTP server to clients. For example, in a group messaging application, whenever a user sends a message to the server, the server has to push such message to everyone. Since 2001, several techniques have been proposed. Eventually, WebSocket has been developed and standardized in 2011. WebSocket is a full-duplex protocol that supports bidirectional communications between servers and clients. It is supported by all modern browsers.

On top of WebSocket, Socket.IO is a Javascript library that abstracts the protocol details. Socket.IO provides an event-driven interface and serializes JSON data automatically. Socket.IO even implements other pushing technologies. If a browser does not support WebSocket, Socket.IO will fall back to long polling protocol. These characteristics make Socket.IO quite appealing to web developers.

In this post, I would like to demonstrate how to build a chat room application named LiveChat with Flask (server side) and Socket.IO (client side).

Prerequisite

The LiveChat application requires 3 Python packages: Flask, Flask-SocketIO, and Eventlet. Flask is a well-known Python web framework. Flask-SocketIO implements the Socket.IO protocol and provides Socket.IO APIs for Flask applications. Eventlet is an efficient event-based networking library that are used by Flask-SocketIO.

To install these packages, create a requirements.txt with:

Flask==0.10.1
Flask-SocketIO==1.2
eventlet==0.17.4

And then, run pip install:

$ pip intall -r requirements.txt

Sending and Receiving Messages

Let's start with sending and receiving messages. In the initial simplistic design, the user will send a message to server and the server will relay the message to all users. Whenever a user receives a message, the Javascript client renders the messsage in the HTML document.

The protocol for the initial simplistic design consists of two events:

  • send-msg -- A client sends a message to the server. Its payload is a string which stands for the message to be sent.
  • show-msg -- The server broadcast a message to all clients. Its payload is a string which should be rendered.

Now, let's look at the code.

First, templates/index.html contains the HTML for the user interface. There is a <form> for users to enter their messages. In addition, there are two <script> tags, which include socket.io.min.js and livechat.js.

<!DOCTYPE html>

<html charset="utf-8">
    <head>
        <title>LiveChat</title>
        <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/1.3.6/socket.io.min.js"></script>
        <script type="text/javascript" src="/static/livechat.js"></script>
    </head>

    <body>
        <form id="inputForm">
            <input id="input" type="text" />
            <input type="submit" value="Send" />
        </form>
        <div id="stream"></div>
    </body>
</html>

Second, static/livechat.js contains the client-side Javascript code:

;(function () {
    'use strict';
    var socket, domInput, domStream;

    function createMessageDOM(text) {
        var line;
        line = document.createElement('p');
        line.appendChild(document.createTextNode(text));
        return line;
    }

    function showMsg(msg) {
        domStream.insertBefore(createMessageDOM(msg), domStream.firstChild);
    }

    function sendMessage(msg) {
        socket.emit('send-msg', msg);
    }

    function onSubmit(evt) {
        if (domInput && domInput.value != '') {
            sendMessage(domInput.value);
            domInput.value = '';
        }
        evt.preventDefault();
        return false;
    }

    function initSocketIO() {
        socket = io.connect('//' + document.domain + ':' + location.port);
        socket.on('show-msg', showMsg);
    }

    function onDOMContentLoaded(evt) {
        var domInputForm = document.getElementById('inputForm')
        domInputForm.addEventListener('submit', onSubmit, false);
        domInput = document.getElementById('input');
        domStream = document.getElementById('stream');

        initSocketIO();
    }

    document.addEventListener('DOMContentLoaded', onDOMContentLoaded, false);
})();

To sum up, livechat.js will:

  • Connect to the server with io.connect().
  • Register a Socket.IO event listener with socket.on(). In the code snippet, the function showMsg() is registered as the event handler for show-msg event.
  • If a show-msg event arrives, showMsg() will create a DOM element and add it to the document with domStream.insertBefore().
  • Register a DOM event listener with domInputForm.addEventListener() to capture the submit event.
  • If a user clicks the Send button, the event handler onSubmit() will emit a send-msg event with socket.emit('send-msg', ...).

Third, livechat.py contains the server-side Python code:

#!/usr/bin/env python

from flask import Flask, render_template
from flask_socketio import SocketIO, emit
import os

app = Flask(__name__)
app.secret_key = os.urandom(48)
app.debug = True

socketio = SocketIO(app)

@app.route('/')
def index():
    return render_template('index.html')

@socketio.on('send-msg')
def handle_message(msg):
    emit('show-msg', msg, broadcast=True)

if __name__ == '__main__':
    socketio.run(app)

To sum up, livechat.py performs following tasks:

  • index() will handle the HTTP request to the path /. It will return templates/index.html when there is an HTTP request from a client.
  • handle_message() is decorated by @socketio.on() decorator. With this decorator, handle_message() will be called when a client emits the send-msg event.
  • handle_message() will call flask_socketio.emit() with broadcast=True to emit show-msg events to all users.

Now, let's run our initial implementation:

$ python livechat.py
* Restarting with stat
* Debugger is active!
* Debugger PIN: 000-000-000
(24559) wsgi starting up on http://127.0.0.1:5000/

Open the browser and visit http://127.0.0.1:5000/. You can also open two tabs and check whether they can send the messages to each other.

Message History

In the initial design, users will only receive the messages that are sent after they have joinned. In this section, we would like to extend our LiveChat application so that the server can send the message history to newly joinned users.

Two events are added to the protocol:

  • request-all-msgs -- A client requests for message history. This event will be emitted when the connection is established.
  • show-all-msgs -- In response to the request-all-msgs event, the server will send the message history to the client with the show-all-msgs event. Its payload is an array of strings which stands for the message history.

On the client side, two extra event handlers are registered. One is for the connect event and the other is for the show-all-msgs event. After the connection is established, the connect event handler will be called. It will send a request-all-msgs event to the server to request for message history. After the server replies, the show-all-msgs event handler will be called and show the messages with showAllMsgs():

;(function () {
    // ... Omitted ...

    function showMsg(msg) {
        domStream.insertBefore(createMessageDOM(msg), domStream.firstChild);
    }

    function showAllMsgs(msgs) {
        var i, domStreamNew;
        for (i = 0; i < msgs.length; ++i) {
            showMsg(msgs[i]);
        }
    }

    // ... Omitted ...

    function initSocketIO() {
        socket = io.connect('//' + document.domain + ':' + location.port);
        socket
        .on('show-msg', showMsg)
        .on('show-all-msgs', showAllMsgs)  // Added
        .on('connect', function() {  // Added
            socket.emit('request-all-msgs');
        });
    }

    // ... Omitted ...
})();

On server side, a _msgs global variable is added to keep all messages in the history. Besides, an event handler for request-all-msgs is registered. When the server receives a request-all-msgs event, handle_sync() will send a show-all-msgs event along with the _msgs list to the origin:

# ... Omitted ...

_msgs = []  # Added

@app.route('/')
def index():
    return render_template('index.html')

@socketio.on('send-msg')
def handle_message(msg):
    _msgs.append(msg)  # Added
    emit('show-msg', msg, broadcast=True)

@socketio.on('request-all-msgs')  # Added
def handle_sync():
    emit('show-all-msgs', _msgs)

if __name__ == '__main__':
    socketio.run(app)

Now, a newly joined user will see the dialogue before the user's entrance.

Using Database

In the previous section, the message history was kept in a global variable. However, all messages will be lost if the server is restarted. To keep the messages, the messages should be saved to a database. In this section, I would like to rewrite the server-side code to utilize SQLite database.

First, open schema.sql and write following SQL statements, which will create a livechat table:

DROP TABLE IF EXISTS livechat;

CREATE TABLE livechat (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    text TEXT NOT NULL
);

Second, several modifications are required for livechat.py:

  • flask.g is imported to keep application variables for an appcontext.
  • get_db() will try to get the database handle from flask.g. If it is not available, then it will open the database with _connect_db(). The returned database handle will be assigned to flask.g as well.
  • close_db() is decorated with @app.teardown_appcontext() so that the database connection can be closed before shutting down the application.
  • init_db() is a utility function to initialize the database. It reuses get_db() and close_db() by wrapping the code with with app.app_context(). It will open the database and execute the SQL statements in schema.sql.
  • handle_message() will save the message with an INSERT INTO statement.
  • handle_sync() will get all messages with a SELECT statement.

Here is the code listing for livechat.py:

#!/usr/bin/env python

from flask import Flask, render_template, g  # Modified
from flask_socketio import SocketIO, emit
import os
import sqlite3  # Added

app = Flask(__name__)
app.secret_key = os.urandom(48)
app.debug = True

socketio = SocketIO(app)

DATABASE = 'livechat.sqlite'

def _connect_db():
    return sqlite3.connect(DATABASE)

def get_db():
    db = getattr(g, '_database', None)
    if db is None:
        db = g._database = _connect_db()
    return db

@app.teardown_appcontext
def close_db(exception):
    db = getattr(g, '_database', None)
    if db is not None:
        db.close()

def init_db():
    with app.app_context():
        with app.open_resource('schema.sql', mode='r') as f:
            db = get_db()
            db.cursor().executescript(f.read())
            db.commit()

@app.route('/')
def index():
    return render_template('index.html')

@socketio.on('send-msg')
def handle_message(msg):
    db = get_db()  # Modified
    db.cursor().execute('INSERT INTO livechat(text) VALUES (?);', (msg,))
    db.commit()
    emit('show-msg', msg, broadcast=True)

@socketio.on('request-all-msgs')
def handle_sync():
    cursor = get_db().cursor()  # Modified
    cursor.execute('SELECT text FROM livechat ORDER BY id ASC;');
    emit('show-all-msgs', list(cursor.fetchall()))

if __name__ == '__main__':
    socketio.run(app)

Now, initialize the SQLite database with:

python -c 'from livechat import init_db; init_db()'

With these changes, we can keep all messages in the database. The messages will no longer disappear after we restart the application.

Conclusion

In this post, we learned how to write a small chat room application with Flask-SocketIO. In the example, we registered event handlers with the .on() method and send events with the .emit() method. We also learned to keep appcontext variables in flask.g and decorate the teardown callbacks with @app.teardown_appcontext. All of the code can be found at my GitHub repositoy @loganchien/livechat.

Notes

Polling (keep sending requests from clients) is an old trick to retrieve the updates from the server, but polling will unnecessarily waste the network bandwidth. Several push technologies (sometimes referred as Comet) have been developed. For example, long polling is a variant of traditional polling. Under long polling model, clients will send the request and wait for the response from the server. The server will defer the response until the message is available. However, most techniques developed in early days have some drawbacks and usually rely on the implementation details of browsers. Fortunately, most use cases can be replaced by WebSocket now.