System Interface Specification

Author: Szymon Acedański
Signed-off-by: not yet

Table of Contents

  1. Table of Contents
  2. Introduction
  3. Decomposition into layers
  4. Core Layer
    1. The SystemInterface class
    2. Supported data types
      1. Passing SIO2 objects
      2. Passing large data blobs and files
    3. Passing errors
    4. The SystemInterfaceContext class
    5. Introspection API
  5. Protocol Layer
    1. XML-RPC Protocol
      1. Handling blobs and files
    2. Other protocols
  6. Transport Layer
    1. Three incarnations of HTTP
  7. Final notes
    1. Multithreading

Introduction

The SIO2 consists of two large architectural parts: PySIO, which includes core business intelligence, and everything else, outside of PySIO, i.e. user interface modules. This document describes mechanisms of communication between PySIO and the other components.

These external components are not meant to be limited only to these provided by the SIO2 team. It is very important to have the System Interface designed in such a way, which promotes writing new components, which makes integrating PySIO into existing solutions easier. This also means, that System Interface should be conveniently accessible from a wide range of programming languages, using various protocols.

Decomposition into layers

The design of System Interface is to be split into three layers:

  • Core Layer
  • Protocol Layer
  • Transport Layer

The design of Core Layer states, how a PySIO programmer can expose desired functionality to external users. The external users communicate with PySIO by invoking methods of internal PySIO objects, using some specified protocol. Protocol Layer design tells about these RPC protocols. Finally, method invocation requests, formulated using a specified protocol, must be carried to the PySIO server using some Transport (network) protocol.

Core Layer

This section will show, from a perspective of the core programmer, how to expose some methods of a class to the external user.

At first, let's introduce a concept of class interface. A class interface is just a group of methods implementing a consistent piece of functionality which may be later exposed to user by implementing them in one or more classes. This is just like in Java and some other programming languages.

It is also similar from the programming point of view. A class interface in Python will be a class, with the following properties:

  • Its name starts from I_, e.g. I_IntrospectionIface. By convention the name should end with Iface.
  • The name must be unique within the entire SIO2 project.
  • It has no base class.
  • If has no fields, no class nor static methods, no constructor etc., just plain instance methods, named with the following pattern <class name>_<method name>, e.g. I_IntrospectionIface_listMethods.
  • All methods are "pure virtual", i.e. they raise NotImplementedError.
  • All methods are documented (using docstrings). This documentation will be in some way made available even through the system interface.
  • All methods take at least two arguments: usual self and a mandatory context, which will be an instance of SystemInterfaceContext, defined later.
  • Additional method arguments are allowed, default argument values are allowed. Argument names are not exposed externally, they are identified by their positions.
  • Data types, which may be expected in arguments, as well as returned by methods, are specified below in section Supported data types.
  • The interface must be registered with the system interface core, by calling sysiface.TheSystemInterface.registerClassIface(<class interface>).

Below, there is an example of a definition of a class interface: 

class I_IntrospectionIface:
    def I_IntrospectionIface_listInterfaces(self, context):
        """
        Return a list of all interfaces exposed by this object.
        """
        raise NotImplementedError

    def I_IntrospectionIface_listMethods(self, context, iface_name):
        """
        Return a list of all methods exposed by the given interface.
        """
        raise NotImplementedError

    def I_IntrospectionIface_methodDocumentation(self, context,
            iface_name, method_name):
        """
        Return a documentation string for a given method of the
        given interface.
        """
        raise NotImplementedError

sysiface.TheSystemInterface.registerClassIface(I_IntrospectionIface)

Having such a class interface defined, providing the functionality is just implementing the defined methods (in a derived class). This is usually done in one of the following ways:

  1. By directly inheriting the class interface. Using the interface defined above, this would look like the following. 
    class Contest(OrmEntity, I_IntrospectionIface):
    [... other methods etc. ...]
    
    def I_IntrospectionIface_listInterfaces(self, context):
        return ['I_IntrospectionIface']

    def I_IntrospectionIface_listMethods(self, context, iface_name):
        if iface_name == 'I_IntrospectionIface':
            return ['listInterfaces', 'listMethods',
                'methodDocumentation']
        else:
            raise <some error>

    def I_IntrospectionIface_methodDocumentation(self, context,
            iface_name, method_name):
        if iface_name == 'I_IntrospectionIface':
            # This is not the safest way, just an example.
            return getattr(I_IntrospectionIface, method_name).__doc__
        else:
            raise <some error>
  2. By implementing a mixin. Like this: 
    class IntrospectionMixin(I_IntrospectionIface):
    def I_IntrospectionIface_listInterfaces(self, context):
        # Look at self, go through the inheritance tree (using
        # __bases__ attribute) and return all classes, whose
        # names begin with 'I_'.

    def I_IntrospectionIface_listMethods(self, context, iface_name):
        # Look to the global interfaces store, as provided by the
        # sysiface module, get the desired interface and return its
        # methods.

    def I_IntrospectionIface_methodDocumentation(self, context,
            iface_name, method_name):
        # Get the desired interface from the sysiface module,
        # look for the specified method and return its docstring.
    Such a mixin is then directly used when implementing a regular SIO2 class: 
    class Contest(OrmEntity, IntrospectionMixin):
    ...

Of course, a SIO2 class can implement (or have mixed in) any number of class interfaces. It is also OK to have some interface methods redefined at any level of inheritance.

The SystemInterface class

The already mentioned sysiface module should provide a class called SystemInterface and a single global instance of it, TheSystemInterface.

The class should implement the following methods: 

class SystemInterface(object):
    def __init__(self, addressing_root):
        """
        Constructor. Initializes the SystemInterface instance with the
        provided addressing root (being an instance of RootResolver)
        """

    def registerClassIface(self, iface):
        """
        Add the specified iface class to the store of all class
        interfaces.
        """

    def classIfaceByName(self, name):
        """
        Find the specified interface, identified by its name (including
        'I_').

        Raise KeyError if not found.
        """

    def allClassIfaces(self):
        """
        Return an iterable of all registered class interfaces.
        """

    def invokeMethod(self, context, object_addr, iface_name, method_name,
            *arguments):
        """
        Invoke the specified method of the specified object, in the
        passed context.
        """
        # This is the main functionality of the SystemInterface class.
        # This method should be called by the upper layers.

The most notable above is the invokeMethod method. All the method calls from outside world will end up in this method. Is should perform the following actions when called:

  1. Begin a new transaction on the default database connection.
  2. Resolve the object address.
  3. Check if the specified interface is presented by the resolved object.
  4. Call the specified method.
  5. Commit the transaction if everything was completed successfully.
  6. Restart the transaction if sqlal.exception.SqlNotTransacted has been raised.

Please note that the SIO2 Core should never call invokeMethod nor any I_* methods. They are dedicated for external calls only, and should generally just wrap simple methods, which indeed can be called from inside the SIO2 Core.

Supported data types

The following Python data types can be expected in parameters and returned:

  • integers, given they are 32bit signed integers
  • floats
  • 8-bit strings (but not unicode strings)
  • lists, given that all values are of the same, supported type
  • tuples, given that all components are supported types, maybe of different types
  • dictionaries, given that
    • values are supported data types
    • keys are strings

Please note that None is not supported.

Higher layer may interchangeably provide lists or tuples (or some other indexable things) as arguments to methods.

Passing SIO2 objects

It should be noted, that the supported data types does not include native SIO2 objects (like Contest etc.). Nevertheless the objects can be passed indirectly by passing their addresses (see Object Addressing). It is also feasible to return object addresses, where an object should be returned.

Passing large data blobs and files

There are two more types, which should be understood by I_* methods:

  • sysiface.types.StoredFile -- representing an uploaded file, already stored (temporarily) in our file storage. Objects of this type may be passed from higher layers to invokeMethod. The following attributes are defined:
    • fileName
    • mimeType
  • sysiface.types.OutputStream -- encapsulating a file-like object, which should be sent back to the client. Such an object may be returned from the invokeMethod method. It has a single attribute:
    • file -- the file-like object.

Passing errors

At this layer, errors are signaled by throwing exceptions. There are no specific exceptions for use in system interface methods. It is up to higher system interface layers to marshal all the possible exceptions back to the callers.

The SystemInterfaceContext class

This class's role is to provide some additional information about the context, in which the method is being invoked. The two notable pieces of information is the timestamp of method invocation, and some authorization information. Some internal state is also needed.

Therefore the SystemInterfaceContext class must provide at least these attributes:

  • timestamp -- represented as a datetime.datetime object containing the moment, when the method invocation request has been actually received by the upper layers
  • auth -- authentication information, its specific format is out of scope of this document
  • addressingRoot -- the instance of Resolver (see Object Addressing) used as a root of the objects tree, filled inside invokeMethod
  • sysiface -- current instance of SystemInterface class, filled inside invokeMethod

It is allowed to make derived classes from this one, although it is not recommended. It may be the case, that some of the methods exposed by SIO2 core require wider context, although it must be noted, that higher level protocols may not provide this wider context. Do it at your own risk. :)

Introspection API

The system interface core will also provide the introspection mechanism, by exporting the I_IntrospectionIface, exactly the same as we used as an example. Every object, which can be addressed and reached using any instance of SystemInterface should provide this interface.

Of course, this won't be the job of the implementors or particular classes. The mixin approach will be used and an appropriate introspection mixin will be implemented as a part of sysiface module.

The introspection functionality can also be provided to the user using some other means than by using the system interface core. For instance, XML-RPC has its own defined introspection API, and our introspection API should also be mapped (by XML-RPC Protocol Layer) into what is specified in the XML-RPC Introspection API specification as well.

Protocol Layer

The core layer provides the notable method (in the SystemInterface class): invokeMethod(context, object_addr, iface_name, method_name, *arguments). The goal of the Protocol Layer is to get the values of the parameters of this method from the user, and to call it. Sounds simple.

But how can we get these parameters? Ok, they will be sent to us using e.g. network, but this is the problem of the Transport Layer. We just get a piece of data. We provide the following interface, so that the Transport Layer will be able to feed us with these data: 

class SystemInterfaceProtocolHandler(object):
    def parseFile(self, context, fileobj):
        """
        Read a request from the passed file-like object, unpack it,
        invoke the specified method, pack the request back to the
        protocol-specific form and return the resulting string.
        Also exceptions must be packed at this layer.
        """

    def parseString(self, context, s):
        """
        Read a request from the passed string, unpack it,
        invoke the specified method, pack the request back to the
        protocol-specific form and return the resulting string.
        Also exceptions must be packed at this layer.
        """

We also have a constructor, as we need to know an instance of SystemInterface, with the famous invokeMethod method. 

    def __init__(self, sys_iface):
        """
        Constructor. Initializes the handler, assigning the
        passed instance of SystemInterface for handling requests.
        """

It is also worth pointing out, that it is not our responsibility to construct the context, although sometimes it may be required to fill a part of it here, at this level. It depends on particular protocols used. For instance, user authorization can be done on HTTP level, above us, but in some other situation, the higher level may not provide any way for user authorization. Then this will have to be solved at this level.

That's the entire abstract base class. Now some notes about the particular protocols we want to have implemented.

XML-RPC Protocol

Nowadays this is a very common RPC protocol, supported in a very wide range of programming languages. We have it also in Python, so it won't be very hard to use it.

There caller will have two ways of executing an exposed method on an object:

  1. By calling an exported method called INVOKE, expecting exactly the same parameters as the invokeMethod method of SystemInterface.
  2. By calling a method named <interface name>.<method name> with the first argument being the address of the object, which implements such a method, and the rest of arguments being method-specific.

The second way is designed to make it sometimes easier to integrate XML-RPC access to SIO2 with some object/interface oriented programming languages.

Our XML-RPC protocol handler should also provide introspection API, by implementing at least system.listMethods (which lists all methods of all registered class interfaces) and system.methodHelp methods (see http://scripts.incutio.com/xmlrpc/introspection.html), matching the second way of invoking methods.

Handling blobs and files

XML-RPC protocol is not designed for carrying large amounts of data. For completeness, we include support for passing sysiface.types.StoredFile objects through XML-RPC.

The provided mechanism is ugly, but should be enough for testing, as well as for some production use. Let's illustrate it using an example. Consider an RPC call: 

SomeInterface.someMethod(addr, arg1, arg2, arg3)

or alternatively 

INVOKE(addr, 'SomeInterface', 'someMethod', arg1, arg2, arg3)

Assume, that arg2 should be a StoredFile. To achieve this, the following calls should be requested using XML-RPC: 

SomeInterface.someMethod('FILE:2:text/plain', addr, arg1, <data as string>, arg3)

or 

INVOKE('FILE:4:text/plain', addr, 'SomeInterface', 'someMethod', arg1, <data as string>, arg3

The number after FILE: is the position of the argument, which is to be replaced by a StoredFile filled with the passed string. The next part is the MIME type.

It is possible to specify multiple FILE: arguments.

sysiface.types.OutputStream instances are passed back as strings.

Other protocols

We will not implement any other protocol handlers, although the interface is designed so that it will be possible to add other protocols without too much work.

Here are some notable candidates:

Transport Layer

The transport layer means involves using some generic request-response based protocol to carry the XML-RPC (or other protocol-layer request) from the user to us and the response back.

There is no specific interface designed for a class implementing transport layer functionality. The expected action is that transport layer at some point calls an appropriate method of an appropriate object of protocol layer.

It is also the responsibility of the transport layer to construct an instance of SystemInterfaceContext and initialize it with as much information as possible at this point.

Three incarnations of HTTP

We will provide an implementation of the transport layer using HTTP. This will come in three flavors.

  1. A mod_proxy module for Apache HTTP sever.
  2. A generic CGI script, pluggable to any CGI-capable web server.
  3. A standalone web server.

It is acceptable that in this third case the transport layer and protocol layer are coupled together by using SimpleXMLRPCServer or even better DocXMLRRPCServer.

Final notes

Multithreading

All specified methods of core layer and protocol layer must be thread-safe (i.e. callable from multiple threads even on the same object).