1. Introduction

This document is intended for developers who are planning on integrating Techila Distributed Computing Engine (TDCE) to existing applications. If you are unfamiliar with the terminology or the operating principles of the TDCE system, please see Introduction to Techila Distributed Computing Engine.

The structure of this document is as follows.

Techila Distributed Computing Engine Integration contains an overview of the TDCE integration process and lists different approaches that can be used to add TDCE functionality to an application.

Creating a Redistributable Package from Techila SDK files contains a description on how to create a redistributable package by using the files in the Techila SDK. Creating a redistributable package will essentially consist of removing unnecessary files from the Techila SDK and modifying the techila_settings.ini configuration file to match the settings in your TDCE environment.

Including the Redistributable Package with your Application illustrates how including the redistributable TDCE package will change the directory structure of your application when it is installed by an End-User. Including the redistributable TDCE package will enable the End-User to access to the TDCE environment, without having to install the Techila SDK.

Creating a Runtime Bundle from a Locally Installed Application contains a description of Runtime Bundles, which can be used to transfer files required on the Techila Workers. Using a Runtime Bundle is recommended in situations where your application requires a multitude of files or other configuration parameters (e.g. such as setting environment variables) when executed on the Techila Workers.

Appendix contains programming language specific lists of files that need to be included when creating a redistributable TDCE package.

2. Techila Distributed Computing Engine Integration

Integrating Techila Distributed Computing Engine (TDCE) to an existing application will enable the computationally intensive part of the application to be performed in the TDCE environment. Typically the integration process will contain similar steps as described in Introduction to Techila Distributed Computing Engine.

These steps typically include the following steps:

  1. Modifying your application by isolating the computationally intensive section to a separate entity (e.g. an executable binary (.exe) or an executable jar file (.jar)). This step will be required because the computationally intensive part will need to transferred and executed on the Techila Workers.

  2. Adding the TDCE API function calls to the locally executable section of your application. These API functions will be used to distribute the computationally intensive part of the application to the TDCE environment.

  3. Creating a redistributable package from the Techila SDK. This redistributable package will need to be included and distributed with the application to enable End-Users to access the TDCE environment.

For more detailed information on how to distribute computations from applications developed with different programming languages, please see the programming language specific how-to guides located here.

2.1. Integration approaches

There are various ways of integrating the TDCE API functions to an application. The image below demonstrates some of the key points, which should be considered when choosing the best integration approach. The integration approaches illustrated in the flowchart are described in the following chapters.

image005
Figure 1. Techila Distributed Computing Engine API functions can be integrated to an application using several different approaches. Key determining factors include having access to the applications source code and whether or not the application supports plug-in modules.

2.1.1. Plug-in Integration

Plug-ins are software modules that can be used to add functionality and features to existing software applications in a modular way, without touching the core of the application. If the application supports plug-in modules, you have the option to implement a TDCE-specific plug-in, which contains all the TDCE API function calls.

image006
Figure 2. Plug-in integrations can typically be used in situations where the application supports plug-in modules. End-Users can continue using the original user interface of the ISV application.

2.1.2. Seamless Integration

The Seamless Integration approach can be used in situations where you have access to the applications source code and can make modifications to the code. This is often the case with in-house applications or open source projects. When using the Seamless Integration approach, the TDCE API functions will be embedded directly into the application source code. This means that the End-User can continue using the ISV application exactly as before and can benefit of the TDCE environment without having to pay any notice to it.

image007
Figure 3. Seamless integrations will require changes to the applications source code. End-Users can continue using the original user interface of the ISV application.

2.1.3. Black Box Integration

The Black Box Integration approach can be used in situations where the software is available as a precompiled binary and where there is no access to the source code of the application and where there is no plug-in interface available.

When using the Black Box approach, the command line interface (assuming the application has one) can be used to launch the application on the Techila Workers with different input arguments or with a collection of input files to perform parametric sweep type-of operations. A common example of this is encoding a large number of media files (input files) using an advanced encoder (precompiled application binary). Please note that when using the Black Box approach, the End-User will use the TDCE API to create the computational Project.

image008
Figure 4. Black Box integrations can be used in situations where the application provides a command line interface. The user interface used by End-Users will be provided by the Techila Distributed Computing Engine API functions such as the peach function.

3. Creating a Redistributable Package from Techila SDK files

The Techila SDK is software development kit provided by Techila Technologies, which includes API`s for several commonly used programming languages. The files included in the Techila SDK can also be used to create a redistributable package, which can included in applications that you plan on distributing to your End-Users. This redistributable package will enable End-Users to access the Techila Distributed Computing Engine (TDCE) computing environment by using the integrated TDCE API functions that you have added to your application.

Please note that when creating a redistributable package from the Techila SDK files, the relative path of the files and directory names will need to match the ones used the Techila SDK. The image below illustrates directories that will typically need to be included when creating a redistributable TDCE package.

image009
Figure 5. A redistributable package that enables access to a Techila Distributed Computing Engine environment can be created by using the files included in the Techila SDK. The relative paths of the directories and the names of directories need to match the ones used in the Techila SDK. Depending on the programming language you are using, you might only need to include some (or none) of the programming language specific directories under the lib directory.

Required files and folders will depend on the programming language used in your application. Please refer to Appendix 1: Programming language specific redistributable Techila packages for lists of files and directories required for several commonly used programming languages.

3.1. Modifying the techila_settings.ini Configuration File

Typically you will also need to include the techila_settings.ini configuration file in the redistributable TDCE package. This configuration file will need to be configured to correspond to the settings in your TDCE environment. The parameters that you will need to configure typically include (at least) the following parameters:

  • hostname=<The network address of your Techila server>

  • keystore=<The path of the End-Users personal keystore>

Settings that are compatible with multiple End-Users can typically be achieved by using macro notations (values enclosed in curly brackets, e.g. {user.home}). More information on configuring the techila_settings.ini file can be found in Getting Started.

Please note that in order for the End-Users to access the Techila environment, they will need to have access to End-User Key.

4. Including the Redistributable Package with your Application

After creating a redistributable package from the files included in the Techila SDK, you will need to include the redistributable package with your application. This will ensure that End-User can access the Techila Distributed Computing Engine (TDCE) environment without having to install any additional files.

image010
Figure 6. Applications that include a redistributable package will enable the end-user to create computational Projects in the Techila Distributed Computing Engine environment. The redistributable package will need to contain the necessary files listed for the used programming language.

When designing the deployment process for your TDCE enabled application, best practices include storing the techila_settings.ini file in the End-Users home directory or other location that can be used to store both temporary and permanent data. This is because the default settings in the techila_settings.ini file are configured to create directories for log information and temporary files under the directory that contains the techila_settings.ini file.

5. Creating a Runtime Bundle from a Locally Installed Application

In order to execute the computationally intensive part of the application (the Techila Worker Code) on Techila Workers, all necessary files required during runtime execution will need to be transferred and made available on the Techila Worker. These files can include dynamically linked libraries (.DLL`s), shared objects (.SO`s), class files and JAR archives.

Files that are required on all of the Techila Workers during runtime execution can either be transferred in Runtime Bundles or as additional data files common to all Jobs.

If the executable application requires multiple libraries and/ or other files during execution on the Techila Workers, the most efficient method will typically be to package all necessary files in a Runtime Bundle. This Runtime Bundle can then be imported to the computational Project to easily construct the runtime environment on each Techila Worker. Having a Runtime Bundle available on the Techila Server will often enable a faster launch of the Techila Distributed Computing Engine (TDCE) processes because the need of data transfer between the End-User’s computer and the TDCE environment can be minimized.

The image below illustrates how files and folders located on a local computer can be transferred to Techila Workers by using a Runtime Bundle. Runtime Bundles can be created using the Bundle Creator Tool or the createBundle command, which is available on the Techila Command Line Interface. Both tools are included in the Techila SDK.

image011
Figure 7. Runtime Bundles can be used to transfer components required to execute the application on the Techila Workers. When the Runtime Bundle is extracted on the Techila Workers, the directory structure of the extracted files will match the one used when the Bundle was created.

When creating a Runtime Bundle, it is also possible to specify values for environment variables and other resources that might be needed to construct the runtime environment on the Techila Workers. These resources can also be, for example, used to make executable files included in the Bundle available for use during computational Jobs or to modify values of environment variables.

After a Runtime Bundle has been created and it has been transferred to the Techila Server, the Bundle can be imported into Projects. When the Bundle is imported in a Project, the Bundle will be automatically transferred to all participating Techila Workers. After the Bundle has been transferred to a Techila Worker, it will be extracted and the folder structure will be restored, assuming file copying has been disabled.

6. Appendix

6.1. Appendix 1: Programming language specific redistributable Techila packages

6.1.1. Java

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with Java.

Programming language used: Java

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

techila_settings.ini

Note
The network address of the Techila Server and the location of the End-Users keystore file will need to be configured in the techila_settings.ini file. Note! If you use the init Java API function (or other initialization functions not requiring the file), you do not need to include the techila_settings.ini file in the redistributable Techila package.

6.1.2. C#

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with C#.

Programming language used: C#

Files to be included from the lib directory

  • TechilaManagement.dll

  • TechilaExecutor.exe

Files to be included from the techila directory

techila_settings.ini

Note
The network address of the Techila Server and the location of the End-Users keystore file will need to be configured in the techila_settings.ini file.

6.1.3. C/C++

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with C/C++.

Programming language used: C/C++

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

  • Techila32.dll

  • Techila64.dll

  • libTechila32.so

  • libTechila64.so

  • Microsoft.VC90.CRT.manifest

  • msvcr90.dll

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini

Note
The network address of the Techila Server and the location of the End-Users keystore file will need to be configured in the techila_settings.ini file. Note! If you use the function initWithoutFile function (or other initialization functions not requiring the file) in the Local Control Code for initializing a session with the Techila Server, you do not need to include the techila_settings.ini file in the redistributable package.

6.1.4. Command Line Interface

The following table can be used as reference when creating a redistributable Techila package for an application which will use the operating systems command line interface to distribute the computational workload.

Programming language used: Command Prompt/Shell

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini

6.1.5. MATLAB

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with MATLAB.

Programming language used: MATLAB

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

  • dnsjava.txt

Files to be included from the techila directory

  • techila_settings.ini

The following directory (all files)

  • lib\Matlab1

Note
When creating the redistributable package, a directory called Matlab directory needs to be available in the lib directory. All files in the lib\Matlab directory in the Techila SDK must also be available in the redistributable Techila package.

6.1.6. R

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with R.

Programming language used: R

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini

The following directory (all files)

  • lib\R

Note
When creating the redistributable package, a directory called R directory needs to be available in the lib directory. All files in the lib\R directory in the Techila SDK must also be available in the redistributable Techila package.

6.1.7. Python

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with Python.

Programming language used:Python

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

  • Techila32.dll

  • Techila64.dll

  • libTechila32.so

  • libTechila64.so

  • Microsoft.VC90.CRT.manifest

  • msvcr90.dll

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini

The following directory (all files)

  • lib\python

6.1.8. Perl

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with Perl.

Programming language used: Perl

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini

The following directory (all files)

  • lib\perl

Note
When creating the redistributable package, a directory called perl directory needs to be available in the lib directory. All files in the lib\perl directory in the Techila SDK must also be available in the redistributable Techila package.

6.1.9. FORTRAN

The following table can be used as reference when creating a redistributable Techila package for an application which is implemented with FORTRAN.

Programming language used:FORTRAN

Files to be included from the lib directory

  • bcmail.jar

  • bcprov.jar

  • bctsp.jar

  • commons-codec-1.3.jar

  • dnsjava.jar

  • techila.jar

  • jcommon-1.0.15.jar

  • jfreechart-1.0.12.jar

  • xmlrpc-2.0.jar

  • Techila32.dll

  • Techila64.dll

  • libTechila32.so

  • libTechila64.so

  • Microsoft.VC90.CRT.manifest

  • msvcr90.dll

Files to be included from the doc directory

  • Techila-SDK-EULA.txt

  • licence-LGPL.txt

  • LICENSE-2.0

  • LICENSE.html

  • LICENSE.txt

Files to be included from the techila directory

  • techila_settings.ini