Close

Get in Touch

Call Us on 1300 727 952
Find us

First Floor, 159 Victoria Pde
Collingwood, VIC 3066
(Google Map)

info@salsadigital.com.au

1300 727 952 
or
+61 3 9910 4099

 

Get in touch

Close

Scalable Worker Environments in the Cloud

Worker environments for processing application data are a common scenario for any cloud-based application. Depending on your requirements, Amazon Web Services offer services that may be applicable.

Rob A 13 January 2014

This is an introduction to a White Paper written by our resident expert, AWS Solutions Architect Rob.

Worker environments for processing application data are a common scenario for any cloud-based application. Depending on your requirements, Amazon Web Services offer services that may be applicable. The Amazon Simple Workflow Service (SWF) allows you to configure multi-stage processes that can be distributed over a series of worker nodes. Similarly, the new Amazon Kinesis service allows for real-time processing of large scale data streams by splitting it over your workers.

For simpler distributed task execution, a custom solution using the Amazon Auto Scaling service on top of your EC2 instances may be more cost effective. Our latest whitepaper outlines our approach to this problem for a recent project using Auto Scaling Groups, CloudWatch and the Amazon Simple Queuing Service, and highlights things you need to be aware of, including:

  • Scaling in your worker environment can be destructive,
  • Built-in CloudWatch metrics may not be accurate enough,
  • External dependencies increase complexity and points of failure.

 

We made each of the worker nodes in the Auto Scaling Group responsible for themselves including electing a master and self termination. The entire solution is self contained except for the SQS Queue and task schedule.

You can read more in the full white paper on our website and visit our AWS website.

Subscribe to the Salsa Newsletter

Subscribe to the Salsa newsletter

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×