Oropo

General

Oropo is a system for task distribution and processing in a computer cluster.

It is designed to process arbitrary tasks.

System is proof against:

• network failures
• computer failures

Oropo is also a framework for building services for this system.

Documentation

Architecture

System works in a computer cluster.
This computer cluster consists of many nodes (computers).
There is one special CENTRAL node in the system.
This CENTRAL node manages the whole system.
Other REGULAR nodes are used for task processing.

Requests for task processing are pushed to the CENTRAL node.
Results are stored in the filesystem in the CENTRAL node.

Task processing flow

• Requests for task processing are sent to the system
• System distributes requests and processes tasks in the cluster
• Tasks processing results are saved and can be picked up

Documents

• Oropo Executor User Manual

  Manual describes how to use Oropo Executor. Problem of processing large number of pictures is used to demonstrate the system.

  • single page html
  • multi page html
  • pdf

• Oropo Service Number Assignment Procedure

  Document describes procedure for number assignment to oropo service. Document also contains description of already assigned numbers.

  • single page html
  • multi page html
  • pdf

Example

Oropo Executor

Program oropo-executor is a service for program execution.

With oropo-executor on remote nodes oropo-system can distribute tasks for program execution.

Installation

on CENTRAL node

• add following entry to /etc/apt/sources.list
  • deb http://students.mimuw.edu.pl/~ms209495/oropo/debian sid main
• apt-get update
• apt-get install oropo-system

on REGULAR nodes

• add following entry to /etc/apt/sources.list
  • deb http://students.mimuw.edu.pl/~ms209495/oropo/debian sid main
• apt-get update
• apt-get install oropo-executor

Adding permissions

on CENTRAL node

• adduser `whoami` oropo

Adding hosts

on CENTRAL node

• oropo-monitor-ctl --id_prefix oropomonitor --add remote1_ip_address
• oropo-monitor-ctl --id_prefix oropomonitor --add remote2_ip_address
• oropo-monitor-ctl --id_prefix oropomonitor --add remote3_ip_address

Pushing requests

on CENTRAL node

• oropo-system-pusher -p "string:sha1sum" -p "path:/tmp/file_to_calculate_sum"
• oropo-system-pusher --timeout 300 -p "string:ffmpeg" -p "string:-y" -p "string:-i" -p "path:/tmp/file.avi" -p "string:-f" -p "string:ogg" -p "string:/dev/stdout"
• oropo-system-pusher -p "string:bash" -p "path:/tmp/script.sh"

As a result of this command task_id is printed.

Picking up results

Result for a task (its standard output) can be found in response directory (default: /var/lib/oropo/response/).

• cat /var/lib/oropo/response/task0_id/0
• mplayer /var/lib/oropo/response/task1_id/0
• cat /var/lib/oropo/response/task2_id/0

Downloads

• Debian Packages

  • /etc/apt/sources.list entry
    • deb http://students.mimuw.edu.pl/~ms209495/oropo/debian sid main

• Debian Sources

  • /etc/apt/sources.list entry
    • deb-src http://students.mimuw.edu.pl/~ms209495/oropo/debian sid main

• Repository public key

License

Program is licensed under BSD License (3-clause license).

Copyright (c) 2009, 2010, Maciej Smolenski (jezdnia@gmail.com) All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the Oropo nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

This program uses libraries:

• Boost Library

  Licensed under Boost Software License - Version 1.0.

• Glibc Library

  Licensed under GNU LESSER GENERAL PUBLIC LICENSE - Version 3.

  This program uses Sun RPC implementation from this library

  The Sun RPC support (from rpcsrc-4.0) is covered by the following license:

  Copyright (C) 1984, Sun Microsystems, Inc.

  Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of Sun Microsystems, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

  Sun RPC is a product of Sun Microsystems, Inc. and is provided for unrestricted use provided that this legend is included on all tape media and as a part of the software program in whole or part. Users may copy or modify Sun RPC without charge, but are not authorized to license or distribute it to anyone else except as part of a product or program developed by the user.

  SUN RPC IS PROVIDED AS IS WITH NO WARRANTIES OF ANY KIND INCLUDING THE WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.

  Sun RPC is provided with no support and without any obligation on the part of Sun Microsystems, Inc. to assist in its use, correction, modification or enhancement.

  SUN MICROSYSTEMS, INC. SHALL HAVE NO LIABILITY WITH RESPECT TO THE INFRINGEMENT OF COPYRIGHTS, TRADE SECRETS OR ANY PATENTS BY SUN RPC OR ANY PART THEREOF.

  In no event will Sun Microsystems, Inc. be liable for any lost revenue or profits or other special, indirect and consequential damages, even if Sun has been advised of the possibility of such damages.