Oropo Executor

Have you ever had a lot of data to process ? In such a moment after a while of processing we realize that it will take ages to complete. It would be faster if we could use two or three or even more computers. Lets use some computers - you think it is a lot of configuration ? You are wrong. With Oropo it's easy. Let's see.

It's difficult to talk about processing without an example. Let's discuss a problem of processing large number of pictures. First approach for solving this problem is to process pictures sequentially on one computer. Second approach is to process pictures parallelly on many computers.

The problem is to process 10000 pictures. Each picture is in hight quality, the goal is to create a smaller version of each one. There is a library libjpeg that provides suitable programs.

Script signature for processing single picture:

Sample script in bash:

#!/bin/bash

QUALITY=30

if [ $# -ne 1 ]; then
	echo "arguments" 1>&2
	exit 1;
fi

FILE_PATH=$1

djpeg $FILE_PATH | cjpeg -quality $QUALITY 

All pictures can be processed by invoking script make_smaller.sh for each picture.

#!/bin/bash

MAKE_SMALLER=$PWD/make_smaller.sh
IMGS_DIR=$PWD/imgs
TARGET_DIR=$PWD/imgs_smaller

for file in $IMGS_DIR/*; do
	bash $MAKE_SMALLER $file > $TARGET_DIR/${file##*/}
done

We can process all picture using Oropo Executor system. Tasks for processing pictures will be added to a queue and processed parallelly on many computers. Each picture will be processed with script make_smaller.sh.

#!/bin/bash

MAKE_SMALLER=$PWD/make_smaller.sh
IMGS_DIR=$PWD/imgs

for file in $IMGS_DIR/*; do
	oropo-system-pusher -p "string:bash" -p "path:$MAKE_SMALLER" -p "path:$file"
done

Processing results can be found in /var/lib/oropo/response/*/0 files.

In previous sections two approaches for processing pictures were presented. First approach uses single computer for processing. Second approach uses many computers for processing. Complexity of both solutions deployment is almost the same. With second approach processing will be completed faster.