service.ipcdatastore

Add On Settings Page


_images/ipc-settings.png

There are several configurable settings for the addon:

  1. The name to be used to address the object being shared. This is an arbitrary string.
  2. The socket host name.
  3. The socket port.
  4. Whether to start the server at startup.
  5. A simple test to assess if the server is working correctly which runs on clicking.
  6. Demo: Whether or not to place data regarding the currently playing video in the datastore automatically.
  7. Demo: Whether or not to show the above data as a notification when a video begins playing.
  8. A more detailed testing suite for the datastore object’s methods which runs when clicked.






Warning

Changes to any of the settings requires you to click ‘OK’ and exit back out of settings and then disable and re-enable the addon in order for the changes to take effect.

Note

When the more detailed testing suite is run, several exceptions will appear in the kodi.log (xbmc.log) file. This is expected behavior. Exceptions are part of the testing.

Note

When using the class IPCClientX (described below) if the addon id is specified, an attempt will be made to retrieve the object name, host name and port from that addon’s settings.xml. Thus, if addon_id='service.ipcdatastore' is provided, then these settings will be read in, even if the code is running in another addon.


Simple use case of IPCClientX

IPCClientX is really a class that wraps some core functionality of a shared dictionary object. Objects are stored in the dictionary under a key consisting of the object’s “name” and “author”. This was implemented in this manner to prevent naming conflicts if serveral different addons are placing objects in the datastore.

The name is required at the time that the data is stored using ipcclientx.IPCClientX.set(). If the author is not provided and the code is running under an addon whose id is available, that will be used. No addon id will be available when the code is running from a ‘RunScript’ statement and an exception will be raised. Each object is time-stamped as it is stored.

1
2
3
4
5
6
7
 # Depending where this code is run from the location of the import may differ

 from resources.lib.ipcclientx import IPCClientX

 client = IPCClientX(addon_id='service.ipcdatastore')
 if client.server_available():
     x = client.set('x', 20, author='service.ipcdatastore')

On line 5, the client is instantiated using the addon_id syntax option. This cause the client to read the name, host and port settings from the addon whose id is provided. This allows any other addon to discover the server’s settings at the time of instantiation instead of having to hard code it.

Line 6 first checks to see if the server is available using the inherited method from IPCClient from script.module.ipc. This isn’t truly necessary, but provides a graceful means of failure if the server is not up.

Line 7 then stores the object using the set method, in this case, an integer using the name ‘x’. The author is explicitly given using a keyword (optional).

Warning

Do not use the tilde (~) in any data names or author names. This will cause errors if you decide to make any of the data persistent between Kodi sessions.


Retrieving the data is much as expected except that there are two additional optional keyword parameters:
  1. requestor
  2. return_tuple

The name of the data requestor may be provided, which also defaults to the addon name if not provided. This is used in the caching mechanism (see below).

If return_tuple is False (default) the returned object is of the same ‘type’ as when it was stored. If return_tuple if True, the return is a namedtuple with three attributes: value, ts and cached. The value is the actual object, ts is the timestamp (float) and cached is a boolean indicating if a cached value was used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
 # Again, depending where this code is run from the location of the import may differ

 import xbmc
 from resources.lib.ipcclientx import IPCClientX

 client = IPCClientX(addon_id='service.ipcdatastore')
 if client.server_available():
     x = client.get('x', author='service.ipcdatastore', requestor='testclient')
     xbmc.log("service.ipcdatastore variable 'x' = {0}".format(x))
 else:
     xbmc.log("service.ipcdatastore server unavailable")

 if client.server_available():
     nt = client.get('x', author='service.ipcdatastore', requestor='testclient', return_tuple=True)
     x = nt.value
     ts = nt.ts
     cached = nt.cached

Line 8 retrieves the data using get as described above, while lines 14-17 illustrate the use of the namedtuple. The value of cached is provided mostly for testing purposes. With the caching algorithm used, there should be no circumstances where a cached object has become ‘stale’. See below.

The full set of methods for IPCClientX available are in the documentation below.

Caching mechanism

Each time data is retrieved, the requestor and the timestamp of the data requested are stored on the server side. The data is also stored on the client side. Also, at the time of each request, the server looks to see if the data has already been provided to the current requestor and if so, if the current data timestamp is the same as what was provided to the requestor during the last request. If it is, a one-byte message is returned instead of the object instructing the client to look for the data in it’s own local cache. If for some reason, the data is NOT in the local cache, another request is sent with a tag instructing the server to provide the data regardless.

This was implemented for performance purposes to minimize the amount of the data sent ‘over-the-wire’. In addition when there is asynchronous data being provided and consumed, it will allow a consuming client to wait in a request loop without transferring the full data set with each request, for instance, if the client is waiting for new data. As might be expected, the impact of caching in this manner is small for small object sizes.

Data persistence

Individual data objects can be tagged for persistence betweeen data sessions as follows:

client.set('x', 20, author='service.ipcdatastore', persist=True)

During each set event for persistent data, a backup of the data is stored in a file and the data is tagged for persistence. If Kodi exits gracefully, at the time of shutdown, all data with persistence tags are written to disk in bulk as well. Upon startup, the server looks to see if Kodi exited normally and, if so, the data is read in from the bulk file. If not, the directory is searched for any backup files and those are loaded instead. This provides at least some protection should Kodi crash, however, do not rely upon this system for critical data restoration.


Class methods from service.ipcdatastore

class ipcclientx.IPCClientX(addon_id='', name='kodi-IPC', host='localhost', port=9099, datatype='pickle')

Bases: ipc.ipcclient.IPCClient

Subclasses IPCClient from script.module.ipc and extends functionality for a datastore object

Parameters:
  • addon_id (str) – Optional keyword. If specified, this supersedes the name/port/host. IPCClient checks settings.xml for the info.
  • name (str) – Optional keyword. Arbitrary name for the object being used, must match the name used by server
  • port (int) – Optional keyword. Port matching server port
  • datatype (str) – Optional keyword. Type of data transport being used options: pickle, serpent, json, marshall. Must match server.

There are two useful public attributes than can be changed after instantiation:

raise_exception:
When set to True, will raise exceptions that can be caught rather than
failing silently, which is the default behavior.
num_of_server_retries:
If the client fails to connect to the server, the number of retries before
failing finally.
set(name, value, author=None, persist=False)

Sets a value on the server. Automatically adds the addon name as the author. The value is any valid object that can be accepted by the chosen datatype (see above). If the class attribute raise_exception is True, will raise an exception with failure.

Parameters:
  • name (Any object type compatible with the datatype transport) – Required. The variable name
  • valueRequired. The value of the variable
  • author (str) – Optional keyword. The originator of the data which along with the variable name is used as the primary key for the backend dictionary for storing the item.
  • persist (bool) – Flag data to be saved between Kodi sessions
Returns:

True for success, False for failure

Return type:

bool

get(name, author=None, requestor=None, return_tuple=False)

Retrieves data from the server based on author and variable name, optionally returns a namedtuple which also includes time stamp (float) and a bool representing whether or not the item came from the local cache. There is a caching function implemented such that the server tracks addon requests for data - if the most recent version has already been received by the client, a message is sent to the client to look in it’s cache for the data. There is a fallback such that if the data is NOT in the cache, the server then provides the data. Each piece of data that is received is locally cached for this purpose.

If returning a namedtuple, the parameters can be accessed as follows:

nt = client.get('x', author='me', requestor='me', return_tuple=True)
x = nt.value
x_timestamp = nt.ts
x_was_cached = nt.cached
Parameters:
  • name (str) – Required. The variable name
  • author (str) – Optional keyword. The author of the data. All of the data is indexed by author and variable name in order to reduce variable name overlap. If not supplied, the addon name is used.
  • requestor (str) – Optional keyword. The name of the requestor of the data. Defaults to the addon name if found. Used for caching.
  • return_tuple (bool) – Optional keyword. Whether to return the stored object or a named tuple containing the object, the timestamp and a bool indicating that the object came from the local cache. The named tuple returns the value, ts and cached.
Returns:

Either the value(object) assigned to ‘name’ or a namedtuple containing the value, ts and/or if the item came from the local cache.

Return type:

object or namedtuple defined as: ('Data', ['value', 'ts', 'cached'])

delete(name, author=None, return_tuple=False)

Deletes an item from the datastore and returns the deleted item’s value. Returns None if not found or raises an exception if the IPCClientX.raise_exceptions attribute is set to True. The return item is optionally returned as a namedtuple (see get()).

Parameters:
  • name (str) – Required. The name of the variable to be deleted.
  • authorOptional keyword. The author of the data. Defaults to the addon id.
  • return_tuple (bool) – Optional keyword. Flag to return data as namedtuple as in ipcclientx.IPCClientX.get()
Returns:

Return type:

object or namedtuple, None on failure

get_data_list(author=None)

Retrieves either a dict or list containing the variables names stored on the server.

Parameters:author (str or None) – Optional keyword. The author of the data or object
Returns:A dictionary containing the authors as key and their variable names as a list. If author specified, returns a list with the variable names. Returns None on failure or raises an exception.
Return type:dict or list
clearall()

Clears all of the data on the server. Use with caution if multiple users are storing data.

Returns:True on success, False on failure
Return type:bool
clearcache()

Clears both the local cache and the server cache for the given addon calling using the addon name.

Returns:True on success, False on failure.
Return type:bool
savedata(author=None)

Saves all of the data for a given author as a pickle object in the addon_data directory for service.ipcdatastore. Plan is to implement a way to do this automatically on server shutdown.

Parameters:author (str or None) – Optional keyword. Defaults to addon id.
Returns:True on success, False on failure
Return type:bool
restoredata(author=None)

Restores data on the server from a previously saved pickle (see savedata())

Parameters:author (str) – Optional keyword. The author whose data is to be restored. Defaults to addon id.
Returns:True on success, False on failure
Return type:bool
delete_data(author=None)

Delete data on the server in the form of a previously saved pickle (see savedata())

Parameters:author (str) – Optional keyword. The author whose data is to be restored. Defaults to addon id.
Returns:True on success, False on failure
Return type:bool
add_persistence(varname, author=None)

Adds a persistence tag to a pre-existing stored object and saves data to backup.

Parameters:
  • varname (str) –
  • author (str) –
Returns:

True on success, False on failure

Return type:

bool

remove_persistence(varname, author=None)

Removes the persistence tag from a stored object and deletes it from backup.

Parameters:
  • varname (str) –
  • author (str) –
Returns:

True on success, False on failure

Return type:

bool


Custom Exceptions

exception ipcclientxerrors.IPCClientError

Bases: exceptions.Exception

Base class for other exception classes

exception ipcclientxerrors.VarNotFoundError

Bases: ipcclientxerrors.IPCClientError

Raised when the author/variable name combination is not found on the server

Variables:message – Error message containing the author and variable name, if set
updatemessage(varname, author)

Allows updating .message if at the time the exception is instantiated, the variable name and author are not available (typical case).

exception ipcclientxerrors.ServerReconnectFailedError(uri, tb)

Bases: ipcclientxerrors.IPCClientError

Raised when a previous connection could not be reopened

Parameters:
  • uri – The uri to display in the message
  • tb – The string traceback
exception ipcclientxerrors.ServerUnavailableError(uri, tb)

Bases: ipcclientxerrors.IPCClientError

Parameters:
  • uri – The uri to display in the message
  • tb – The string traceback
exception ipcclientxerrors.ObjectNotSerializableError

Bases: ipcclientxerrors.IPCClientError

Raised when an object to be used is not seriablizable with the current serializer

updatemessage(obj)

Allows for message updating when the object is unavailable at the time of instantiation :param obj: the object that failed serialization

exception ipcclientxerrors.UseCachedCopyError

Bases: ipcclientxerrors.IPCClientError

Raised when the server instructs the client to use data from the cache

exception ipcclientxerrors.SaveFailedError

Bases: ipcclientxerrors.IPCClientError

Raised when a pickle save fails on the server

exception ipcclientxerrors.RestoreFailedError

Bases: ipcclientxerrors.IPCClientError

Raised when a pickle restore fails on the server

exception ipcclientxerrors.UnknownError(msg, tb)

Bases: ipcclientxerrors.IPCClientError

Error otherwise not defined

exception ipcclientxerrors.NoError

Bases: ipcclientxerrors.IPCClientError

Default case when no errors have occurred. Used to keep interface with IPCClientX.__getwrapper consistent.


Classes used for Datastore - The actual object that is Proxied

class datastore.DataObjectBase

Bases: object

Base class for DataObject and DataObjectX

class datastore.DataObject(dox)

Bases: datastore.DataObjectBase

Class of all objects returned by server. Includes object and timestamp.

Requires instance of datastore.DataObjectX during instantiation: that is the class actually stored in the datastore dict.

Parameters:dox (DataObjectX()) – Required. See above.
class datastore.DataObjectX(value, persist=False)

Bases: datastore.DataObjectBase

Class used to store objects in the datastore. Extends datastore.DataOnject with a dict of requestors

Parameters:value (pickleable obj) – The object to be stored
class datastore.DataObjects(persist_dir=None)

Bases: object

The actual datastore object whose methods are exposed via pyro4.proxy

If you desire to allow data to persist between Kodi sessions, the directory to store persistent data is needed at the time of instantiation in order to restore any saved data, if any exists.

Parameters:persist_dir (str) – the directory where the persistent data is stored
setautosave(val)

If True will save data when object deleted or .close() called

Parameters:val (bool) – True or False
set(name, value, author, persist=False)
Parameters:
Returns:

Nothing

get(requestor, name, author, force=False)
Parameters:
  • requestor (str) –
  • name (str) –
  • author (str) –
  • force (bool) –
Returns:

Either a dataoject or a one byte message code

Return type:

datastore.DataObject or one character str

delete(name, author)
Parameters:
  • name (str) –
  • author (str) –
Returns:

Either a dataoject or a one byte message code

Return type:

datastore.DataObject or one character str

get_data_list(author=None)
Parameters:author (str) –
Returns:
Return type:dict with author(s) as key(s)
clearall()
Returns:Nothing
savedata(author, fn)
Parameters:
  • author (str) –
  • fn (str) –
Returns:

True on success, False on failure

Return type:

bool

restoredata(author, fn)
Parameters:
  • author (str) –
  • fn (str) –
Returns:

True on success, False on failure

Return type:

bool

clearcache(requestor)
Parameters:requestor (str) –
Returns:Nothing
close()

Explicitly write persistent data to a file in anticipation of shutting down.

add_persistence(varname, author)

Adds a persistence tag to a pre-existing stored object and saves data to backup.

Parameters:
  • varname (str) –
  • author (str) –
Returns:

True on success, False on failure

Return type:

bool

remove_persistence(varname, author)

Removes the persistence tag from a stored object and deletes it from backup.

Parameters:
  • varname (str) –
  • author (str) –
Returns:

True on success, False on failure

Return type:

bool