1. Growing test suite can show a bottleneck

The key to building quality into software is making sure the engineers can get fast feedback on the impact of changes. Many teams have created a pipeline where every change runs a build that creates packages, runs automated tests, and builds reports to provide instant feedback to the developers.

When the number of tests increases, and the test cases grow in complexity, even automated testing may create a bottleneck to the pipeline. This bottleneck can be removed by using parallel testing.

intro

2. Parallel computing that connects to the workflow

Techila Distributed Computing Engine (TDCE) is an interactive high-performance computing solution. TDCE comes with open APIs that support efficient integration of TDCE to the workflow for parallel testing. The solution:

  • supports interactive run of automated tests that provides instant feedback to the developers,

  • provides easy APIs for fast integration with an existing pipeline,

  • runs in the user’s own cloud account and supports dynamic scalability for a faster test turn-around time,

  • simplifies the use of computing infrastructure by offering end-to-end security, automatic system management, and built-in error tolerance.

3. Case - Parallel testing of control software for power electronics

3.1. Problem - Developers must wait hours for feedback

A team that develops control software for power electronics uses Python, MATLAB and C in their R&D. The team’s Continuous Integration (CI) process uses a Jenkins environment, and testing that is automated using Robot Framework. The goals are to:

  • build quality into software,

  • cut down R&D costs, including infrastructure and software,

  • reduce the need for expensive physical prototypes, and

  • improve the productivity of R&D of new and custom products.

The test suite that the team uses has grown in size and complexity. The team needed a solution that can improve the throughput of the testing phase, and scale to the requirements of current and future business.

When a developer commits a change into the repository, the Jenkins environment starts the execution of tests that are implemented using the Robot Framework. The total run time of the tests was over 11 hours and growing. This is a problem for the productivity and efficiency of the team’s software R&D.

3.2. Solution - Techila Distributed Computing Engine: Developers get feedback in 3 minutes

Techila Distributed Computing Engine (TDCE) is available as a pay-as-you-go solution in Google cloud platform marketplace (GCP Marketplace), which made it easy for the team to try. TDCE is also available with an enterprise license that can be cost-efficient in sustained use.

The team launched TDCE in Google cloud, and downloaded a copy of Techila SDK from their Techila Configuration Wizard user interface in the Google cloud. The Techila SDK is a collection of programming language-specific libraries that integrate applications to the Techila environment. The Techila SDK (which is available in the Configuration Wizard) was pre-configured for use with their Google account, and therefore easy to take into use.

The team’s Robot script is written in Python. The original script runs the tests one Robot file at a time. Each Robot file contains one or several test cases. The original total run time of the automated tests is 11 hours. TDCE’s Python API makes it easy to reprogram the script to execute the tests in parallel - either by Robot files or test cases. When run using TDCE, the run time will be cut from 11 hours to 3 minutes. A developer does not need to wait for feedback, which benefits building quality into software.

4. Why Techila Distributed Computing Engine?

System management can cause inefficient overheads for testing. A user of Techila Distributed Computing Engine (TDCE) does not need to worry about managing the software configuration of the test infrastructure. The automatic system management and built-in error tolerance make TDCE an ideal solution for an automated pipeline.

The technology of TDCE configures the system automatically to support execution of the codes that are built in Python, MATLAB, and C. The self-configuration capabilities of TDCE improves the efficiency and helps to avoid software version issues. Therefore, TDCE is an efficient solution in a continuously evolving R&D environment.

In this customer’s case, the MATLAB codes are simulation models. Some of the test cases include computationally heavy simulation. One way to reduce simulation time is to use simplified models and rely on a reduced accuracy of simulation, this means trading accuracy and accepting the risk of cutting corners.

In addition to parallel execution of the test cases, the scalability provided by TDCE enables simulation of the models at the required high accuracy. TDCE also comes with the Techila Toolbox for MATLAB that is an efficient tool for the optimization of the simulation time without cutting corners in accuracy.

Many traditional high-performance computing solutions require a user to submit his/ her job to a queue and return later to check the results. Unlike these, TDCE supports interactive computing in the following ways:

  • TDCE does not require waiting in a queue,

  • an engineer can follow the progress of testing on his own computer,

  • a user can view intermediate test reports instantly when partial results have completed,

  • the engineer can get instant feedback already during the test cycle.

Many traditional schedulers use pool-based resource management. The scheduler of TDCE can scale dynamically and use any available compute resources. A user can even add more Virtual Machines to the system dynamically during a test run. More resources translate to more throughput. The throughput can be adjusted simply using TDCE’s graphical Configuration Wizard user interface, or the infrastructure management can be automated using the API of TDCE.

The original time to run the automated tests was 11 hours. TDCE made it possible to run the tests in just 3 minutes. When the test suite grows, the run time can be kept constant simply by adding more Virtual Machines to the system. TDCE orchestrates the workload automatically to utilize also the new compute resources.

5. Summary - Interactive parallel runs enable fast feedback on the impact of changes

Techila Distributed Computing Engine provides abstraction and simplifies the use of the computing infrastructure. The benefits include

  • The developers get an interactive view on the impact of changes, and their effect on the software quality.

  • The throughput of the automated testing can be adjusted on demand, without having to touch the test scripts.

  • A growing test suite or the need for increased accuracy in simulation-driven tests can be compensated simply by adding more computing resources.

  • The customer can integrate their own cloud account securely with their existing test IT, and offload test cases to the cloud for parallel processing.

When the team uses Techila Distributed Computing Engine, even when the number of tests increases, and the test cases grow in complexity, the turn-around time of the test phase can be optimized. The pipeline will:

  • run efficiently,

  • the developers will get fast feedback on the impact of changes,

  • the team can continue building quality into software.