Twisted is a framework designed to be very flexible and let you write powerful servers. The cost of this flexibility is a few layers in the way to writing your server.
At the base, the place where you actually implement the protocol
parsing and handling, is the protocol handler class. This class will usually be
decended
from twisted.protocols.protocol.Protocol
.
Most protocol handlers inherit either
from this class or from one of its convenience children. An instance of the
protocol class will be instantiated per-connection, on demand, and it will go
away when the connection is finished. This means that persistent configuration
is not saved in the Protocol
.
The persistent configuration is kept in a Factory class, which usually
inherits from twisted.protocol.protocol.Factory. The default factory class
just instantiates each Protocol, and then sets on it an attribute called
factory
which points to itself. This lets every
Protocol
access, and possibly modify,
the persistent configuration.
It is frequently needed to serve the same thing on two or more different ports or network addresses. This is why the Factory does not listen to connections, and in fact does not know anything about the network. This is the job of a Port class. Usually, the class twisted.internet.tcp.Port will be used, but there are several alternatives like twisted.internet.udp.Port and twisted.internet.ssl.Port.
Because of this multi-faceted aspect to writing servers, it is useful to have some place where all this things are bound together - using a specific Protocol class, with a specially initialized factory and a specific network interface on a specific port. For that, the twisted.tap package exists, which records "common server deployments". It creates a TAP (Twisted Application Pickle) which keeps the configuration information for a specific server.
So, in conclusion - most often, you will just have to write your Protocol class (which might get complicated, and you may want to write auxiliary classes for it) and then write a twisted.tap submodule which programmatically explains how to use this class.
This document will explain each step of the way.
As mentioned above, this, along with auxiliary classes and functions, is where most of the code is. A Twisted protocol handles data in an asynchronous manner. What this means is that the protocol never waits for an event, but rather responds to events as they arrive from the network.
Here is a simple example:
from twisted.protocols.protocol import Protocol class Echo(Protocol): def dataReceived(self, data): self.transport.write(data)
This is one of the simplest protocols. It simply writes back whatever is written to it. There are many events it does not respond to. Here is an example of a Protocol responding to another event:
from twisted.protocols.protocol import Protocol class QOTD(Protocol): def connectionMade(self): self.transport.write("An apple a day keeps the doctor away\r\n") self.transport.loseConnction()
This protocol responds to the initial connection with a well known quote, and then terminates the connection.
The connectionMade event is usually where set up of the connection object happens, as well as any initial greetings (as in the QOTD protocol above, which is actually based on RFC 865). The connectionLost event is where tearing down of any Protocol-specific objects is done. Here is an example:
from twisted.protocols.protocol import Protocol class Echo(Protocol): def connectionMade(self): self.factory.numProtocols = self.factory.numProtocols+1 if self.factory.numProtocols>100: self.transport.write("Too many connections, try later") self.transport.loseConnection() def connectionLost(self): self.factory.numProtocols = self.factory.numProtocols-1 def dataReceived(self, data): self.transport.write(data)
Here connectionMade
and connectionLost
cooperate to keep a count of the
active protocols in the factory. connectionMade
immediately
closes the connection if there are too many active protocols.
In this section, I will explain how to test your protocol easily. You'll still need to read the rest of the document to run a production-grade Twisted server, though.
Here is code that will run the QOTD server discussed earlier
from twisted.protocols.protocol import Protocol, Factory from twisted.internet.main import Application class QOTD(Protocol): def connectionMade(self): self.transport.write("An apple a day keeps the doctor away\r\n") self.transport.loseConnction() # Next lines are magic: factory = Factory() factory.protocol = QOTD app = Application("QOTD") # 8007 is the port you want to run under. Choose something >1024 app.listenTCP(8007, factory) app.run()
Don't worry about the last 6 magic lines -- you will understand what they do later in the document.
Many protocols build upon similar lower-level abstraction. The most popular in internet protocols is being line-based. Lines are usually terminated with a CR-LF combinations.
However, quite a few protocols are mixed - they have line-based sections and then raw data sections. Examples include HTTP/1.1 and the Freenet protocol.
For those cases, there is the LineReceiver
protocol.
This protocol dispatches to two different event handlers -
lineReceived
and rawDataReceived
. By default,
only lineReceived
will be called, once for each line. However,
if setRawMode
is called, the protocol will call
rawDataReceived
until setLineMode
is called again.
Here is an example for a simple use of the line receiver:
from twisted.protocols.basic import LineReceiver class Answer(LineReceiver): answers = {'How are you?': 'Fine', None : 'I don't know what you mean'} def lineReceived(self, line): if self.answers.has_key(line): self.sendLine(self.answers[line]) else: self.sendLine(self.answers[None])
Note that the delimiter is not part of the line.
Several other, less popular, helpers exist, such as a netstring based protocol and a prefixed-message-length protocol.
Many Twisted protocol handlers need to write a state machine to record the state they are at. Here are some pieces of advice which help to write state machines:
As mentioned before, usually the class
twisted.protocols.protocol.Protocol
Factory
.
For a factory which simply instantiates instances of a specific protocol
class, simply instantiate Factory
, and sets its "protocol"
attribute:
from twisted.protocols.protocol import Factory from twisted.protocols.wire import Echo myFactory = Factory() myFactory.protocol = Echo
If there is a need to easily construct factories for a specific configuration, a factory function is often useful:
from twisted.protocols.protocol import Factory, Protocol class QOTD(Protocol): def connectionMade(self): self.transport.write(self.factory.quote+'\r\n') self.transport.loseConnection() def makeQOTDFactory(quote=None): factory = Factory() factory.protocol = QOTD factory.quote = quote or 'An apple a day keeps the doctor away' return factory
A Factory has two methods to perform application-specific building up and tearing down (since a Factory is frequently persisted, it is often not appropriate to do them in __init__ or __del__, and would frequently be too early or too late).
Here is an example of a factory which allows its Protocols to write to a special log-file:
from twisted.protocols.protocol import Factory from twisted.protocols.basic import LineReceiver class LoggingProtocol(LineReceiver): def lineReceived(self, line): self.factory.fp.write(line+'\n') class LogfileFactory(Factory): protocol = LoggingProtocol def __init__(self, fileName): self.file = fileName def startFactory(self): self.fp = open(file, 'a') def stopFactory(self): self.fp.close()
So, you know what factories are, and want to run the QOTD with configurable quote server, do you? No problems, here is an example.
from twisted.protocols.protocol import Factory, Protocol from twisted.internet.main import Application class QOTD(Protocol): def connectionMade(self): self.transport.write(self.factory.quote+'\r\n') self.transport.loseConnection() def makeQOTDFactory(quote=None): factory = Factory() factory.protocol = QOTD factory.quote = quote or 'An apple a day keeps the doctor away' return factory app = Application("QOTD-with-factory") app.listenTCP(8007, makeQOTDFactory("configurable quote")) app.run()
And that's it!
While more ports can be written, it is a less frequent operation
and so will not be discussed here. If there is a need to write another
Port, follow the example in twisted.internet.tcp.Port.
Most code uses twisted.internet.tcp.Port
s. These are
initialized with a factory and a port number:
from twisted.internet.tcp import Port from twisted.protocols.protocol import Factory from twisted.protocols.basic import Echo factory = Factory() factory.protocol = Echo port = Port(7, factory)
Usually, code will not need to handle Port
s directly.
twisted.tap
As seen, many times the interplays between a specific Protocol class
and a specific Factory class depend on quite a few attributes which all
need to be set correctly. The twisted.tap
package allows
a programmer to document it programmatically, while giving a naive user
an easy way to generate configurations.
Let us assume that a module, twisted.quote
, included
the following code, which has already been given as example:
from twisted.protocols.protocol import Factory, Protocol class QOTD(Protocol): def connectionMade(self): self.transport.write(self.factory.quote+'\r\n') self.transport.loseConnection() def makeQOTDFactory(quote=None): factory = Factory() factory.protocol = QOTD factory.quote = quote or 'An apple a day keeps the doctor away' return factory
We want a user to be able to generate a quote server easily. He should be able to give a quote, and optionally a port number. If no port number is given, the server should listen on port 17, the one mandated by the RFC.
We will have the configuration-creator accept the options -q
or --quote
to set the quote, and -p
or
--port
to set the ports.
Here is an example (put this code in twisted/tap/quote.py):
from twisted.python import usage from twisted.quote import makeQOTDFactory class Options(usage.Options): synopsis = "Usage: mktap quote [options]" optStrings = [["port", "p", 17], ["quote", "q", None]] longdesc = "Quote server" def updateApplication(app, config): factory = makeQOTDFactory(config.quote) app.listenTCP(int(config.port), factory)
% mktap quote --quote "Twisted is coolness personified" % su # twistd -f quote.tap # exit %You need to run
twistd
as root, because it needs to bind
to a privileged port. Don't worry -- it'll shed privileges and suid to
the user you ran mktap
as soon as possible. It is not very
hard to configure it to run as any other user:
% mktap --uid 33 --gid 33 quote --quote "Twisted is coolness personified" % su # twistd -f quote.tap # exit %Will run it as
www-data
on Debian systems, for example.