How to manage large data amounts?

How to manage large data amounts?

Postby techila support » 2016-03-14 14:53:30

When using Techila in a Google Cloud Platform environment, you can use Google Cloud Storage for storing the input and output data. The process flow below explains how you can use the Google Cloud Storage to manage data when performing Techila computations.

1. Create the Google Cloud Storage bucket. The easiest way to create bucket is to use the Google Cloud Console here: https://console.cloud.google.com/storage
2. Transfer input data from your computer to the Cloud Storage bucket you created. You can transfer the data e.g. by using CloudBerry, Google Cloud SDK or by using the Google Cloud Console.
3. Modify your Worker Code so that each Job retrieves the correct file from the Cloud Storage Bucket. One of the easiest ways is to perform a system command in the Worker Code, which retrieves the file. The example system command shown below could be used to retrieve a file called 'input1.mat' from bucket 'mybucket1456' and store it in the current working directory on the Worker:

gsutil cp gs://mybucket1456/input1.mat .

In your actual use case, you will need to replace 'input1.mat' with the actual input file name needed by the Job. This means you will most likely need to do some string manipulation so that each Job retrieves a different file.

4. Modify your Worker Code so that each Job stores the correct result file to the Cloud Storage Bucket. The example system command shown below would transfer all files starting with the 'resultfile' string to 'mybucket1456'.

gsutil cp resultfile* gs://mybucket1456/

5. Start Techila Workers by using the Techila Configuration Wizard.
6. Create the computational Project.
7. Retrieve results from the Google Cloud Storage bucket using one of the methods mentioned before (CloudBerry, Google Cloud SDK or by using the Google Cloud Console.)
Techila MATLAB documentation available here:

http://www.techilatechnologies.com/help ... ngine.html
techila support
Techila Staff
Techila Staff
 
Posts: 51
Joined: 2015-12-21 10:19:47

Return to Google Compute Engine

cron