Shared data in a SOA scenario

Shared data in a SOA scenario

Postby e151761 » 2013-11-18 19:26:26

I have a SOA-type Python code. My plan is to distribute the services using Techila to achieve a better throughput. Currently, my services use some shared data in my main program. What would be the easiest way to implement the shared data when I go to Techila? Because my service objects will be in a distributed environment, I guess my main program can't give them a reference to a shared data, right?
e151761
 
Posts: 25
Joined: 2013-05-23 07:05:33

Re: Shared data in a SOA scenario

Postby kari » 2013-11-19 09:54:49

You could use a SQL server to host the shared data.

Or if the data won't change, then you could distribute the data to the workers. This can be done by using bundles or downloading the data from a shared source to the workers.
kari
Techila Staff
Techila Staff
 
Posts: 50
Joined: 2012-07-18 11:14:09

Re: Shared data in a SOA scenario

Postby Marko » 2013-11-27 15:03:47

As Kari mentioned, one option is to use a database for storing variables that need to be shared. I have attached a simple example that shows how you can connect to a mysql database from computational Jobs and modify the contents of a table in the database. The attached 'accounts_mysql.sql' file shows the commands that can be used to create a table called 'accounts' and populate the table with data. This table will be modified during the example.

Please note that running the example requires that the 'MySQLdb' Python package.

Please also note that the database connection will be opened from the Techila Workers to your database server, so please check that your firewall and database configurations have been configured to allow the connection.

You should be able to run the example using the steps below:

1. Create the 'accounts' table using the commands in the attached 'accounts_mysql.sql'.
2. Modify row 9 in the 'worker_updatedb_mysql.py' file to contain your database server connection details.
3. Install 'MySQLdb' package on your computer
4. Launch python
5. Execute following commands in Python to run the example:

execfile("run_db_demo_mysql.py")
run_db_demo_mysql()
Attachments
accounts_mysql.sql
Database initialization commands
(1.97 KiB) Downloaded 523 times
run_db_demo_mysql.py
Local Control Code
(581 Bytes) Downloaded 517 times
worker_updatedb_mysql.py
Worker Code
(1.2 KiB) Downloaded 536 times
Marko
Techila Staff
Techila Staff
 
Posts: 10
Joined: 2013-09-11 11:19:10

Re: Shared data in a SOA scenario

Postby Igor Savinkin » 2019-01-23 11:03:18

Marko, I've followed your description and it works to invoke workers, yet it does not connect to localhost... :|

Part of worker's code:
Code: Select all
db = MySQLdb.connect(db = 'correction_dict', host='localhost', user='xxx', passwd='xxx')


Error:
Code: Select all
Native execution returned with exitcode: 1 (0x1), starter value: 124.
STDOUT:
Error 2003: Can't connect to MySQL server on ''localhost'' (10061)
...


Why is there the error connecting to my localhost of my PC(Windows) from the techila distributed enviroment?

Note, when executing locally the code works perfect to connect to db:
Code: Select all
db = MySQLdb.connect(db = 'correction_dict', host='localhost', user='xxx', passwd='xxx')
Igor Savinkin
 
Posts: 2
Joined: 2019-01-21 15:07:47

Re: Shared data in a SOA scenario

Postby Marko » 2019-01-23 11:20:01

Hi Igor,

Thanks for reaching out. When the code is executed on the Techila Worker, the host='localhost' parameter will point to the Techila Worker virtual machine, it does not point to your computer.

To successfully establish a database connection from the Techila Worker to a database that is running on your own computer, you will need to use the public IP address of your computer as the value of the host parameter. Also all the firewalls need to be configured to allow the connection.

Another option would be to get the data from the database locally, before pushing computations to Techila. You could then transfer the input data via parameter/files to the Techila Workers. This would mean that you would not need to establish any database connections from the Techila Worker. But without knowing the use case details, this might not be a valid approach.
Marko
Techila Staff
Techila Staff
 
Posts: 10
Joined: 2013-09-11 11:19:10

Re: Shared data in a SOA scenario

Postby Igor Savinkin » 2019-01-23 11:58:07

Marko, thanks for a fast reply.
Project
My project consists of editing MS Word files by finding errors in it and replacing them with correspondent corrections.
A pre-compiled Error-correction DB consists of over 2.3M entries... and be increasing.
Sure, the project is intended to be an online service letting users to upload file and get it back corrected. I've turned to the Techila Distributed Computing (TDC) since the processing of huge files fails to be done (over-timed) as a single process flow.
DB usage
As far as the DB usage the best would be (I think) to host the Error-correction DB (DB) at the Google Servers and let the Correction service to request data from it. If you have any such a pattern, I'd be very thankful.

I can host DB at my PC but this is just an initial attempt/version. The best would be to put all the code and DB at the cloud (Google Compute Engine or Google cloud Platfom).
Any suggestions?
Igor Savinkin
 
Posts: 2
Joined: 2019-01-21 15:07:47

Re: Shared data in a SOA scenario

Postby Marko » 2019-01-23 13:07:29

Thanks for the background info, it is always interesting to hear how TDCE is being used to solve different types of problems.

Hosting the database in Google might be a good way to make the database accessible to Techila Workers. Google has a service for hosting databases:

https://cloud.google.com/sql/

As a first step, I think it might be a good idea to setup a small database for testing purposes and verify that the Techila Workers can access it ok.

After verifying that the database accesses work as intended, you could start to add more functionality to the system, such as multiple Techila Workers / actual Correction service operations.
Marko
Techila Staff
Techila Staff
 
Posts: 10
Joined: 2013-09-11 11:19:10


Return to Python

cron