Multi Tier PHP Application Using RabbitMQ, Web & Worker Architecture

Large Web application needs to have some strategy to ensure web run smoothly. It is also intended to avoid Server down on busy hours.

Imagine thousand to million hits your Server simultaneously. Application could be down even with great hardware specs.
In Hardware point of view, you can do Load Balancing, Traffic Manager, Scale out by adding more Virtual Machine or Scale in by getting higher specs i.e CPU, Memory, HD.
The infrastructure setup is even easier using popular Cloud Portal available widely in market i.e Ms Windows Azure, Amazon Web Service, Google Cloud Platform.
You don’t need to have physical servers on your room. Cloud Portal has menu to create VMs and there are several CPU, Memory, Storage specs options. More high the specs then more costly.
Also you can do Load balancing, Auto/Manual Scaling, Traffic Manager with just some clicks on the Portal.

However considering only H/W and infrastructure sometimes is not enough.
You don’t want to be forced urgently to add more VM or increase H/W specs just because actually you are not managing your application or software properly.

Modern enterprise application use Multi Tier architecture. They use Web and Worker tier for handling users request. On back end side there are database and storage tier.
Web tier contains user interface i.e html body, event handler, etc. Worker is a background process that continuously run and usually used to process jobs which require more computational resources asynchronously .
The main purpose to have web & worker tier is to avoid requests burden application too busy and further more causing your server not responding.

Lets say we have an incident example. There is a web app which has reporting feature sent to user in PDF format. Many clients request for reports simultaneously and those requests are directly processed by database server on same time.
Also after data is retrieved, the Web tier creating many PDF based on requests and send them to clients by email.
Slowness will occur and thus Server either web tier or database will not responding. Not to mention if web tier and database is in one VM only.
You need to stop requests by killing them or just restart the server.

This case is a good example for explaining the need of Worker. So Web tier will sent user request by writing a message to Message Queue instead of sent requests directly to back end or database tier.
Message Queue is just text stored in a Message broker and it has FIFO (First In First Out) for write & read those messages. In this scenario, Web Tier write a request as message to Message Queue
Worker instance that run continuously on background reads client request or message from Message Queue and creating PDF report. The PDF report creation will be sequential based on FIFO of message queue.
So only one report processed in a time for one Worker instance. Lets say you want more report generated in time then you can generate another Worker instance. So you have two or more Workers.
You can manage how many workers you can have based on Server performance.

After Report is generated either that Worker which auto send the report or you may create another Worker to do it.
So first Worker that generate report will write a message to Queue when report is done. Second Worker read a message and send the report.

Below is the schema of multi tier application with Web & Worker Tier.

It is a common scenario and also as a tutorial example in internet since it is understandable with ease.

However this blog post source code is not presenting above schema.
This time we assume for having a Photo Web App and it has more than thousands hits every day like Instagram. This example will retrieve list of photos of followed persons / owner.
Since the Web app hits is quite heavy then we need some strategy using multi tier application.

Basic method to retrieve an image is to download it from database via Web tier and show it to browser. We won’t do that way. We will use Worker to retrieve photos from DB and store them to file storage.
Web tier will show those photos to client browser. The communication between Web & Worker tier use Message queue i.e RabbitMQ.

Please look the schema below:

The difference for this case is the Message Queue has ‘reply-to’ feature. So every request will be replied through Message Queue to ensure Web tier (client) get response immediately after photos are generated.

Many Message Queue available on internet but I use RabbitMQ because of easiness to install and easy to code also.
If you have not familiar yet with RabbitMQ then I suggest you to go to this Url: https://www.rabbitmq.com/getstarted.html and install it to your PC/laptop/dev server.
There is also nice step by step tutorials so you can go through it.

First, we create a simple table for demonstration purpose only. I use MySQL.
The table name is blobs and it contains just three columns which are id (int), imgname (varchar(50)), img (LongBlob) to store photos.
Also I upload some photos to that table.

Here is Worker source code worker_imgdb.php.
This worker get photos from MySql table based on photo’s Owner and stored on local hard disk.
You may store photos to your file server or external storage.
This worker also apply basic_consume() method to read a message so worker can process a request and basic_publish() method to write/reply a message indicates photos is generated and already stored on storage.

include(__DIR__ . '/config.php');
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();

$channel->queue_declare('rpc_queue', false, false, false, false);

function get_images($owner) {
    $host = 'localhost';
	$user = 'root';
	$pass = '';

	mysql_connect($host, $user, $pass);
	mysql_select_db('test');

	$result=mysql_query("SELECT imgname, img FROM blobs where imgowner='".$owner."' order by id ");
	$filename="";
	while($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
		$filename = $filename.$row['imgname'].";";
		$file = $row['imgname'];
		file_put_contents('./images/'.$file, $row['img']);
	}
	mysql_close();
	return $filename;
}

echo " [x] Awaiting RPC requests\n";
$callback = function($req) {
    $owner = $req->body;
	
    echo " [.] get_images(", $owner, ")\n";

    $msg = new AMQPMessage(
        (string) get_images($owner),
        array('correlation_id' => $req->get('correlation_id'))
        );

    $req->delivery_info['channel']->basic_publish(
        $msg, '', $req->get('reply_to'));
    $req->delivery_info['channel']->basic_ack(
        $req->delivery_info['delivery_tag']);
};

$channel->basic_qos(null, 1, null);
$channel->basic_consume('rpc_queue', '', false, false, false, false, $callback);

while(count($channel->callbacks)) {
    $channel->wait();
}

$channel->close();
$connection->close();

Run above script from command prompt with this syntax ‘php worker_imgdb.php‘.

This worker runs continuously until you break it. If you want to add more worker then type again the ‘php worker_imgdb.php‘ syntax thus you will have two workers run.

Below is script for Web tier demo_getimg.php.
This web tier is an user interface which has one textbox and button to get Owner photos.

<html>
<head>
<meta charset="UTF-8">
<title></title>     
<script type="text/javascript" src="jquery-1.6.2.min.js"></script>
<script type="text/javascript">
$(document).ready(function() {
$("#btnGetImg").click(function() {
	$.ajax( {
		async: false,
		url: "client_imgdb.php",
		type: "post",
		data: $("#frm1").serialize(),
		success: function(response) {	
			var begin = response.indexOf("Begin>>>");
			var end = response.indexOf("End>>>");
			var str = response.substr(begin + 8, end-begin-8);
			var res = str.split(";");
			var strhtml = "";
			for (var i=0; i<res.length; i++) {
				if (res[i]!=="") {
					strhtml = strhtml + "<img src='images/" + res[i] + "' /><br><br>"
				
				}
			}
			$("#div_img").html(strhtml);
		}		
		
	});
});

});

</script>
</head>
<body>
<form id='frm1'>
Input Owner: &nbsp;<input type='text' id='owner' name='owner'> <br>
<input type='button' id='btnGetImg' value="Owner's Images"/>
</form>
<div id='div_img'>
</div>
</body>
</html>

There is client_imgdb.php file on ajax call above. This PHP file client_imgdb.php is intended to sent request to Message Queue from Web Tier and standing by for a reply.

include(__DIR__ . '/config.php');
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

class GetImageClient {
    private $connection;
    private $channel;
    private $callback_queue;
    private $response;
    private $corr_id;

    public function __construct() {
        $this->connection = new AMQPStreamConnection(
            'localhost', 5672, 'guest', 'guest');
        $this->channel = $this->connection->channel();
        list($this->callback_queue, ,) = $this->channel->queue_declare(
            "", false, false, true, false);
        $this->channel->basic_consume(
            $this->callback_queue, '', false, false, false, false,
            array($this, 'on_response'));
    }
    public function on_response($rep) {
        if($rep->get('correlation_id') == $this->corr_id) {
            $this->response = $rep->body;
        }
    }

    public function call($owner) {
        $this->response = null;
        $this->corr_id = uniqid();

        $msg = new AMQPMessage(
            (string) $owner,
            array('correlation_id' => $this->corr_id,
                  'reply_to' => $this->callback_queue)
            );
        $this->channel->basic_publish($msg, '', 'rpc_queue');
        while(!$this->response) {
            $this->channel->wait();
        }
        return $this->response;
    }
};

$image_rpc = new GetImageClient();
$owner = $_REQUEST["owner"];
$response = $image_rpc->call($owner);
echo " Begin>>>", $response, "End>>>\n";

If client get reply then the script will be done and also image is displayed.

This is simple example only. You can do better then that. According to information on this URL https://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-celery-and-rabbitmq.html Instagram also use Workers and RabbitMq as Message broker.

Below is a screen shot of demo_getimg.php.
Type the owner of images and click button then images will be displayed. Also I give the screen shot just for fun 🙂

Cheers,
Agung Gugiaji

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s