Thespian Director Guide

As a review, loadable Actor sources provide a method for hot-updates of the Actor source code and possibly running multiple versions of the sources in parallel. This has a number of advantages:

• The sources only need to be loaded into the leading ActorSystem; all other ActorSystems will update from the leading system automatically on an as-needed basis. This also alleviates the need to restart the ActorSystem on the remote nodes each time the source is updated.
• Multiple versions of the sources can be running simultaneously. One use case for this is allowing the older sources to continue running the requests submitted to that source while directing all new requests to the newer version. The older sources can be unloaded at a later time after all of the in-progress requests have been completed.
• Loaded sources are validated by the Source Authority, which can use cryptographic signature validation of the sources, increasing the overall security of the system.

The Thespian Director is easily invoked from the command line via the Python module specification syntax:

$ python -m thespian.director [verbose] [command [args...]]

When invoked with no command, the Director will output help information to stdout but take no other action. This can be used to quickly determine what the needed command is or the overall settings are.

The Director takes several primary commands, and each has a different set of arguments. The "help" command will print the same information as when specifying no command, but following the help command with the name of another command will print help information specific to that second command.

When invoked, the Director will typically look for various local files specifying configuration settings and loadable sources. It is customary to setup a single directory to contain all of these files, and the Director uses the THESPIAN_DIRECTOR_DIR environment variable to locate this directory; if not set, the directory defaults to "/opt/thespian/director" under Unix or "C:thespian\director" under Windows.

One of the primary purposes of the Director is to support the ability to dynamically load new Actor sources into a running Thespian ActorSystem (or a Convention of ActorSystems). In order to safely load these sources and provide appropriate security, Thespian will not use any loaded source unless the registered Source Authority indicates that the source has been validated.

The Director provides a Source Authority actor that will register as the validator of these packages. The Source Authority will ensure that the loaded source package was generated by the Director itself (see the next section) and that the source itself is signed by a cryptographic key that validates the provenance of the source and that the source has not been modified after it was signed. To do this, the Source Authority will read any public key files (named "*_tls.key") found in the THESPIAN_DIRECTOR_DIR location and use these to verify the loaded sources. There may be multiple key files and these key files are read dynamically each time a source is loaded; as long as a source can be validated by one of the key files, it will be accepted.

The Director's Source Authority is started automatically when the Thespian ActorSystem is started by the Director. Other than ensuring the correct key files are available in the Director's operational directory there is no configuration or ongoing management needed for the Source Authority.

In order to create a loadable source, the varous Python files must be packaged into a ZIP file, and that ZIP file should be prepared for validation by the Source Authority. This preparation process will modify the ZIP file, making it invalid in such a way that it can only be restored and used if the Source Authority successfully validates the loaded file.

The preparation process will then sign the sources with a locally-specified private key file. The public key corresponding to this private key must be placed in the Directors operational directory on each system where the loaded sources may be utilized (each ActorSystem in a convention will independently validate the loaded sources when utilized). The private key should not be distributed in this manner: any entity possessing the private key can create a loadable source that will be accepted by the Source Authority. The private key should only be accessible to the system where the loadable sources are built or generated.

One of the Director's capabilities is being able to startup the Thespian ActorSystem on the local host to be able to run the Actor applications. In order to properly start the ActorSystem, the Director needs the set of configuration parameters that should be used for that startup.

The configuration elements used for the ActorSystem are read from files in the THESPIAN_DIRECTOR_DIR directory. These configuration elements are not in a "config.ini" style file, nor are they JSON, YAML, or any other encoding. Instead, each configuration value is maintained in a separate file whose contents can simply be read and directly used as a Python value. This is done to make it easy to update these configuration files via automated means: no existing configuration format must be read and partially manipulated: simply re-write an entire file with the new configuration value.

The configuration files and their usage are described by running the help command for the Director:

$ python -m thespian.director help

Here is an example that specifies a remote convention leader, a local set of capabilities, a logging configuration, and a logging directory.

$ ls $THESPIAN_DIRECTOR_DIR
convleader.cfg othercaps.cfg thesplog.cfg thesplogd.cfg

$ cat convleader.cfg
192.168.32.19

$ cat othercaps.cfg
{ 'Worker': True, 'Slot': 19, 'nproc': 2 }

$ cat thesplog.cfg
{ 'version': 1,
  'formatters': {
    'normal': {
      'format': '%(asctime)s,%(msecs)d %(actorAddress)s/PID:%(process)d %(name)s %(levelname)s %(message)s',
      'datefmt': '%Y-%m-%d %H:%M:%S'
    }
  },
  'root': {'level': 20, 'handlers': ['foo_log_handler']},
  'loggers': {
    'subsysA': {'level':30, 'propagate': 0, 'handlers': ['foo_log_handler']}
  },
  'handlers': {
    'foo_log_handler': {
       'filename': 'fooapp.log',
       'class': 'logging.handlers.TimedRotatingFileHandler', 
       'formatter': 'normal', 
       'level': 20, 'backupCount':3, 
       'filters': ['isActorLog'], 
       'when': 'midnight'
    }
  },
  'filters': {'isActorLog': {'()': 'thespian.director.ActorAddressLogFilter'}}
}

$ cat thesplogd.cfg
/tmp/log

$

The Director can be used to start the local Thespian ActorSystem either on-demand or at boot time. The startup will use the configuration parameters defined in the configuration files described above, and the startup will have no effect if an ActorSystem is already running.

To manually start:

$ python -m thespian.director start

To configure Thespian to be started at boot time:

$ python -m thespian.director bootstart

Note that the latter command will also start Thespian immediately as well unless the word "nostart" appears on the line after the bootstart command. There may also be a name argument after the bootstart (and before the "nostart" argument) which will specify the name to assign to the system service associated with the Thespian ActorSystem started at boot time; the default is "thespian.director".

The boot configuration automatically detects the current system's startup manager and writes the appropriate files to the appropriate locations. Currently supported system managers are: systemd, initctl, and Windows Boot Manager.

A running Thespian Actor System can be stopped with the shutdown Director command:

$ python -m thespian.director shutdown

This will cause all Actors to receive an ActorExitRequest and the Actor System itself to shutdown, just as if the ActorSystem().shutdown() method had been called.

The source files generated by the 'gensrc' command (described above) can be loaded into the current Thespian ActorSystem and run. The Director can be used to load individual sources on-demand, or load the latest version of each source in every group.

To load a single source, specify the name of either the tli file or the specific tls file to load:

$ ls /opt/thespian/director
foo-201710261938.tls foo-201710261902.tls foo.tli    sourcekey_tls.key
other-1.3.5          other-1.4            other-2.0  other.tli

$ python -m thespian.director load foo-201710261938.tls

$ python -m thespian.director load foo

Both of the load commands above are equivalent and will load the indicated tls source into the Thespian ActorSystem. The load command will automatically start the Thespian ActorSystem if it is not currently running.

The set of currently loaded and running sources can be retrieved with the list command:

$ python -m thespian.director list

Each loaded source is described by its group name and also by its hash value. The hash value is the same hash value returned by the ActorSystem().loadSource() function and represents the hash of the loaded source.

The sources can be unloaded as well:

$ python -m thespian.director unload {HASH}

The source associated with the specified hash value is unloaded.

To find out what sources are available to be loaded, use the avail Director command:

$ python -m thespian.director avail
THESPIAN_DIRECTOR_DIR=/opt/thespian/director
foo: /opt/thespian/director/foo-20170261938.tls  4d41d823c3702e9cf7b8b5a650429d7e
   + /opt/thespian/director/foo-20170261902.tls  2a9fcdc7589812b4596f317e85b76042
other: /opt/thespian/director/other-2.0.tls    1bd9df7222c035855babefdfff8f9e29
     + /opt/thespian/director/other-1.4.tls    8406cf8f4596f9cfcae33c7f463bfc1f
     + /opt/thespian/director/other-1.3.5.tls  d9550b9af01692d6a4154cc54cd8d9aa

$

The above shows sample output for the directory contents previously shown. It lists each loadable source by group, sorted by version within the group with the most recent version first and the oldest version last. Each line also specifies the hash associated with the loadable source.

To remove a loadable source from consideration for loading, simply delete it from the THESPIAN_DIRECTOR_DIR directory.

As well as the command-line interface, a Python program can interact with the Director in two ways:

1. Exchanging messages to the Director Actor (via ActorSystem.ask() or another Actor).
2. Via the DirectorControl object.

In the first form, the source for the director can be examined: there are documentation strings describing all the messages that can be exchanged with the Director and the associated responses. The Director Actor is registered under the "Director" global name, so an example of retrieving information about all loaded sources is as follows:

import thespian.actors
asys = ActorSystem()
director_addr = asys.createActor('thespian.director', globalName='Director')
resp = asys.ask(director_addr, { "DirectorOp": "RetrieveAll"})
print(str(resp))

The second alternative is to create an instance of the DirectorControl object. Once created, there is a cmd_COMMAND method for each COMMAND that the Director supports on the command-line, along with arguments corresponding to the arguments supplied to the command-line version. There is also docstring help for each method.

$ THESPIAN_DIRECTOR_DIR=/opt/thespian/director python3
>>> from thespian.director import DirectorControl
>>> dc = DirectorControl()
>>> dc.cmd_avail()
THESPIAN_DIRECTOR_DIR=/opt/thespian/director
foo: /opt/thespian/director/foo-20170261938.tls  4d41d823c3702e9cf7b8b5a650429d7e
   + /opt/thespian/director/foo-20170261902.tls  2a9fcdc7589812b4596f317e85b76042
other: /opt/thespian/director/other-2.0.tls    1bd9df7222c035855babefdfff8f9e29
     + /opt/thespian/director/other-1.4.tls    8406cf8f4596f9cfcae33c7f463bfc1f
     + /opt/thespian/director/other-1.3.5.tls  d9550b9af01692d6a4154cc54cd8d9aa
0
>>> help(dc.cmd_avail)
... docstring information ...

The final 0 in the above output is the return value from the command method: command methods will return 0 on success or non-zero on error.

There is an additional method: the DirectorControl.get_role_address('ROLENAME') method, which does not print any output but returns a dictionary with information about the currently active Actor associated with the specified Role (as specified in the TLI file).

$ THESPIAN_DIRECTOR_DIR=/opt/thespian/director python3
>>> from thespian.director import DirectorControl
>>> dc = DirectorControl()
>>> r = dc.get_role_address('MainFoo')
>>> print(r)
{'DirectorResponse': 'RoleAddress',
 'ActorAddress': <thespian.actors.ActorAddress object at 0x7f266ct1d1d0>,
 'Group': 'foo',
 'ActorClass': 'foo.fooactor',
 'SourceHash': '4d41d823c3702e9cf7b8b5a650429d7e',
 'Role': 'MainFoo',
 'Success': True}
>>> r = dc.get_role_address('unknown role name')
RoleAddress ERROR: {'DirectorResponse':'RoleAddress',
                    'ActorAddress': None, 'Role': 'un',
                    'SourceHash': None, 'Group': None,
                    'ActorClass': None, 'Success':False}
>>> print(r)
None
>>> r = dc.get_role_address('unknown role name', silent=True)
>>> print(r)
None
>>>

This method can be used to easily obtain these role-identified actor addresses to initiate communications with them from an application. Printing of errors can be suppressed by passing the argument silent=True, and a group argument can also be specified to limit the role search to the corresponding group (otherwise all groups are searched for a role match).

Although the loadActorSource() functionality works quite well, there are a few scenarios that can cause difficulties:

1. Loaded sources containing or relying on object libraries. The loadActorSource() functionality affects the Python module lookup, but it does not affect object file lookups that would need to be performed for native object libraries. In addition, object libraries are specific to their build environment and may not work in all target environments. If the loaded source simply refers to an externally available library (e.g. PyOpenSSL python package refers to /usr/lib/libssl.so) then this may work if the proper object library has been installed on the target system by other means (i.e. not via the loadActorSource() call.
2. Sources containing other PEP302 path importers may not work correctly when used in conjuction with the loadActorSource() PEP302 importer. As an example, the loadActorSource() importer prefixes loaded modules with the Source Hash associated with those modules to ensure uniqueness. This is not always compatible with other PEP302 importers.

   1. The six package (providing Python2/Python3 compatibility) uses a PEP302 importer to manage moved packages. This functionality of the six package is broken by the loadActorSource() Source Hash prefixing. To avoid this loadActorSource() processing handles six as a special case and does not prefix six with any Source Hash.

      The assumption is that six is a mature and unchanging package and a global version is acceptable for all loaded sources (and that a working six importer is preferrable to a non-working importer).

3. The importlib package has a subtle flaw that is not compatible with the Source Hash prefixing performed by loadActorSource(). To work around this problem Thespian provides an updated version of importlib; use " import thespian.importlib as importlib " instead of just " import importlib " in Actor sources that will be loaded via loadActorSource().
4. Loaded sources should expect that __file__ or other internals can be set to None.
5. It is not possible to send internally-defined objects between sources (see Communication across package boundaries below).
6. The loadable source can only contain pure Python code, including all dependencies. If there are dependencies that have associated C object extensions or other binary-specific components, those must be pre-installed and accessed via the normal fashion and not included as a dependency in the loadable source package.

The Python logging library has the ability to configure callable objects to provide various formatter capabilities. Since the thesplog.cfg file must define only a Python dictionary and not executable code, that file cannot contain any callable objects.

If a callable object was needed for logging, it would need to be installed on the target systems (along with Thespian) prior to starting up Thespian. As long as the callable target is a valid import path for the context in which Thespian is started, it can then be used.

By default, Thespian uses the pickle module to send messages from one actor to another. This requires that the unpickling operation in the target Actor's environment be able to reconstitute the pickled object by name.

When using loadable sources, all names are prefixed with the hash value of the source they have come from, thereby creating a private namespace for each loaded source. Actors within the same loaded source will share the same hash and can use internally created class objects to communicate, but that they will not be able to do this when communicating with Actors in other loaded sources (or the external environment). This means that all messages exchanged with Actors across a loaded source boundary must be regular, built-in Python objects (strings, numbers, booleans, tuples, lists, dictionaries, etc.), or a standardized format such as JSON.

If there is an attempt to send an internal source-loaded object across this boundary, a deserialization error will be written to the logfile for the destination actor, but it will not even be able to generate a PoisonMessage response because it cannot parse the message to determine the sender.

When using the ActorSystem().createActor() method, the Actor to be created must be specified by the fully module-qualified string name of the Actor instead of the class reference.

The following two will not work:

import foo
fail1 = ActorSystem().createActor(foo.fooActor.FooActor)

from foo.fooActor import FooActor
fail2 = ActorSystem().createActor(FooActor)

however, the following will work correctly:

good_addr = ActorSystem().createActor('foo.fooActor.FooActor', ...)