Torsten's Blog Space

HPC Cloud Bursting to Windows Azure… Published!

torstenlangner — Tue, 03 Jan 2012 18:50:23 +0000

Happy New Year!

Just happy that the book is available on Amazon now. Will publish the related code samples in a few days…

Someone told me that HPC can save lives… so, let’s be pragmatic: all gains will be donated

HPC Cloud Bursting to Windows Azure

Code Samples:

Download from my Skydrive

Content:

Cloud Bursting – Introduction. 9
Internal & External Cloud Concepts. 10
Installation and Configuration of HPC Server 2008 R2 On-Premise. 13
Architecture of HPC Server 2008 R2. 14
Typical HPC Architecture #1: Automotive Industries = HPC. 16
Typical HPC Architecture #2: Financial Services Industry (FSI) = Grid. 18
Installation of High Available head nodes. 19
Prepare a Lab Environment with Hyper-V. 20
Cluster Configuration – To Do List 33
Create an On-Premise Node Template. 37
Create Node Templates. 38
Cluster Management 50
Configure Azure. 52
Add Azure Nodes to Your Head Node. 56
Deploy Standard Azure Nodes. 57
Deploy Virtual Machines as Azure Nodes. 61
Azure Connect: Connect your Extneral with the Internal Cloud. 68
Azure HPC Scheduler 70
Core Development of SOA Based Applications. 71
Prepare the Development Environment 71
Prepare the Cluster Environment 72
SOA: Architectural Overview.. 73
How Does Message Submission Work?. 74
Analysing the Difference Between Interactive and Durable Sessions. 75
Durable or Interactive?. 75
Before Developing the First Application. 77
Develop the First Interactive Application. 78
The Service. 78
The Interactive Client 78
Run the Application. 90
Debug the Application. 91
Develop the First Durable Application. 92
The Service. 92
The Durable Client 92
Run & Debug the Application. 96
Understand the Core Classes & Concepts. 96
Parameters to Create a Session. 96
Parameters to Attach to an Existing Session. 97
Working With Broker Clients in a Shared Session. 100
Real Life Scenario: 2 Independent Clients Need To Share a Session. 103
Client Based Failover – Resubmitting Tasks. 115
Exception Handling. 116
Add Tracing Logic. 120
Catch the Exit Event of a Service Instance. 125
Understand the Behavior of SOA Services. 126
Full Speed with “Hot” Sessions: Minimize the Send & Response Time. 134
Flush: Minimize the Send & Response Time of Durable Sessions. 139
Understand the Different Scheduler Policies. 140
Queued Mode. 142
Queued Mode | Preemption. 143
Queued Mode | No Preemption. 144
Queued Mode | Graceful Preemption. 145
Queued Mode | Immediate Preemption. 147
Balanced Mode. 149
Lend & Borrow Through Resource Pools. 153
Concepts of Stateful Computation. 158
The State Management 159
On-Premise Based State Management 161
Azure Blob Based State Management 162
Azure AppFabric Cache Based State Management 165
The Stateful Service implementation. 169
Reduce the Total Computation Time. 172
Concepts of Common Data. 173
On-Premise Hosted Common Data. 174
Azure Hosted Common Data. 178
Deploy the Application to the Cluster 181
HPC SOA Configuration Files. 183
SOA Service Versioning. 190
Deploy On-Premise. 191
Deploy in Azure. 191
Deploy both in Azure and On-Premise. 193
Remote UDFs with Excel 2010 – On-Premise and Azure Based. 193
Start the C# Coding. 194
The UDF Part 194
Configure Excel-DNA. 196
Use the UDF in Excel 2010 – Local UDF Run. 196
Use the UDF in Excel 2010 – Remote UDF Run. 197
Run Excel 2010 VBA Based Apps – On-Premise and Azure Based. 199
Core Development of Batch Based HPC Applications. 199
Create the Windows Azure Package & Upload It 200
Deploy the Package Content Locally. 201
Run the Mixed Workload. 201

Bring Excel 2010 to Speed: Remote UDFs with Excel 2010 and HPC Server 2008 R2

torstenlangner — Mon, 10 Jan 2011 14:04:44 +0000

A typical sceanrio in the Financial Services world is the following: A user wants to calculate some heavy stuff. In most of the cases it’s thousands of Monte Carlo scenarios. He uses Excel to read real-time data from feeds like Reuters or Bloomberg and uses a function to calculate portfolio values.

The multi-core features of Excel 2010 might help to scale the distribute workload and reduce the total calculation time. But in many cases it is not enough. Assuming a single simulation takes 1 second to calculate and assuming further you need 1,000 different simulations to calculate a portfolio then the whole calculation won’t finish before 1,000 / cores available seconds.

User Defined Functions are a key technology of Excel extensibility. UDFs allow users to create their own mathematical functions and use them in formulas just like they use Excel’s built-in functions sum, avg etc. With the introduction of Excel 2010 and HPC Server 2008 R2 users are now able to distribute UDF workload accross hundreds of physical servers inside an HPC cluster.

UDF programming is usually done in C++. I’m not a C++ expert. A customer in the banking space pointed me to an open source project called Excel DNA that he uses to implement portfolio evaluation UDFs in .NET code. Please have a look at Govert’s blog for updates regarding his project.

I’ll use the Excel DNA project now to demonstrate the functionality of remote UDFs, Excel 2010 and HPC Server 2008 R2:

Start the C# Coding

The sample application I’m going to develop now contains 2 parts: a math part and the UDF part. The application will allow users to forecast stock prices based on historical prices. Assuming the Microsoft stock price was 23$, 23.45$, 24.12$, 23.23$ etc. in the past days, the application will predict the stock price by using a magic glass bowl

The .NET code will we implemented in a simple class library.

The Math Part

The math part of the application was not written by a Quant. It was written by me – and I am the worst mathematican you can find on earth. I’m creating some code that will put some workload on the CPU now.

The core part of the application is a forecast tool that tries to predict tomorrow’s stock price based on a double array of historic input prices.


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace MySuperCoolRemoteUDF.MathTools
{
 public class MCForecastTool
 {
  ...
  public MCForecastTool(List Prices)
  {
    ...
    Initialize();
  }

  public double PedictNextPrice()
  {
    double absolutePriceChange = lastPrice - beforeLastPrice;

    double rnd1 = RandomNumberGenerator.NextSystem();
    double rnd2 = RandomNumberGenerator.NextSystem();

    double propUpTomorrow = 1d - IndicatorState.ProbUp;
    double propDownTomorrow = IndicatorState.ProbUp;

    short tomorrowsDirection = (short)0;
    short direction = absolutePriceChange < 0 ? (short)-1 : (short)1;
    ...
    double priceChange = CalculateCurrentAVG();

    double lambda = 1d / priceChange;

    double rnd3 = RandomNumberGenerator.NextExponential(lambda);
    double priceTomorrow = ...;

    return priceTomorrow;
  }

  ...
 }
}

The UDF Part

After implementing the mathematical part it is now time to create a UDF using .NET and Excel-DNA. To enable an integration add the Excel-DNA integration .dll to the current Visual Studio project.

Excel-DNA will publish all public static methods as UDFs in Excel that are tagged as ExcelFunction (line 11). Remote UDFs need to set the IsClusterSafe-attribute to true. The UDF named NextPricePlease will accept two parameters: the number of Monte Carlo runs (iterations) and the historical prices that arrive from cell values as a multi-dimensional object array (HistoricalPrices). That’s pretty it!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using MySuperCoolRemoteUDF.MathTools;
using ExcelDna.Integration;
namespace MySuperCoolRemoteUDF
{
 public class MonteCarloRunner
 {
   [ExcelFunction(Description = "Glass bowls the next stock price",
Category = "My UDFs", IsClusterSafe=true)]
   public static double NextPricePlease(double Iterations, object[,] HistoricalPrices)
   {
     List Prices = new List(HistoricalPrices.Length);
     // read the Excel Cells:
     foreach (object HistoricalPriceObj in HistoricalPrices)
     {
        double Price = (double) HistoricalPriceObj;
        Prices.Add(Price);
     }
     int IterationsInt = (int)Iterations;
     List MCToolList = new List(IterationsInt);
     // Create N-many forecast tools:
     for (int i = 0; i < IterationsInt; i++)
        MCToolList.Add(new MCForecastTool(Prices));
     double tempSum = 0d;
     // Predict N-many forecast prices
     foreach (MCForecastTool MCFT in MCToolList)
         tempSum += MCFT.PedictNextPrice();
     // Tomorrow's price is the mean of all predictions:
     double NextPrice = tempSum / (double)IterationsInt;
     return NextPrice;
   }
}
}

Configure Excel-DNA

After compiling the C# code it’s time to configure Excel-DNA in the right way. Since the code was compiled to a *.dll we just need to crate an XML file with a unique name: MySuperCoolRemoteUDF.dna. Further we need to copy the ExcelDna.xll as MySuperCoolRemoteUDF.dll to the same directory (see picture). A more detailed Excel-DNA how-to is discussed in Govert’s forum.

Use the UDF in Excel 2010

Local UDF Run

By default every UDF will be executed locally. A user must explicitely activate remote UDF execution on an HPC Server 2008 R2 cluster. Let’s use a local run first to create a new Excel 2010 workbook. As you can see in the screenshot below the spreadsheet will contain historical prices of 20 stocks. These prices could be real- or near-time prices but in our sample they are randomized. Starting from Now we will use the SuperCoolRemoteUDF to predict tomorrows price of every stock. The day after tomorrow will be predicted based on the historical prices from row B to row L plus the predicted price of row M. Thus we have dependencies between every cell starting from row N to row Q.

The local computation of one cell (like M8) takes approximately 6.8 seconds using 250,000 iterations. To calculate every cell row M, N, O, P and Q in parallel we need at least a 20 core machine. Assuming a typical 4 core workstation the whole matrix will be calculated within 20 / 4 * 5 * 6,8 = 170 seconds.

Remote UDF Run

Using an HPC Server 2008 R2 cluster we can scale the computational power to thousands of cores. If there were no dependencies between the cells we would be able to calculate every cell in parallel within the cluster. But since there are dependencies between the rows M to Q we can only calculate row by row.

To enable a parallel run we just need to allow user-defined XLL functions to run on a cluster. That’s it. The spreadsheet will immediately re-calculate its values once they update on a cluster. This reduces the pure computational time from 170 seconds to 20 / 20 * 5 * 6,8 = 35 seconds. Approximately 3-5 seconds need to be added to create a session and send and receive the message requests.

Run Batch Workload on a Mixed Infrastructure (Windows Azure Worker Nodes & On-Premise HPC Server 2008 R2 Compute Nodes)

torstenlangner — Mon, 10 Jan 2011 12:33:32 +0000

With the introduction of SP1 of HPC Server 2008 R2 it is possible to run workload on Windows Azure. If you want to be able to off-burst your batch application (= extending the infrastructure from classic on-premise servers to cloud based Azure worker nodes) you need to set specific environment variables within the on-premise server infrastructure.

Here is a sample of how to upload a batch based application (azuretestbatch.exe) to Azure workder nodes and run a parametric sweep workload across the full cluster (on-premise compute nodes + Azure worker nodes):

Create the Windows Azure Package & Upload It

Use the hpcpack create command to create a *.zip file. Once the package got created, upload it to the Windows Azure storage by using the hpcpack upload command. Specify the node template of your Windows Azure worker nodes. Additionally specify a relative path – in our sample its named test. This forces the hpc sync command which we call later to unzip the package content to the folder named test:

C:\Users\tlangner>hpcpack create AzureTestPackage.zip
C:\temp\folder1

C:\Users\tlangner>hpcpack upload AzureTestPackage.zip
/nodetemplate:AzureNodeTemplate /relativepath:test

You can test your upload by calling the hpcpack view command. As you can see in the lines below the targeted physical directory on the Windows Azure worker node disks is %CCP_PACKAGE_ROOT%\test. The environment variable %CCP_PACKAGE_ROOT% only exists on Windows Azure worker nodes.

C:\Users\tlangner>hpcpack view azuretestpackage.zip
/nodetemplate:AzureNodeTemplate
Connecting to head node: head
Querying for Windows Azure Worker Role node template "AzureNodeTemplate"
Windows Azure Worker Role node template found.
Retrieving Azure account name and key.
Found account: hpc*** and key: ********
Package Name:           azuretestpackage.zip
Uploaded:               06.01.2011 10:43:44
Description:            Node
Template:               AzureNodeTemplate
Target Azure Dir:       %CCP_PACKAGE_ROOT%\test

After the hpcpack upload command has finished, it is time to deploy the package content to the Windows Azure worker nodes (ie. local disks of the cloud machines). This is done by calling the hpcsync command:

C:\Users\tlangner>hpcsync /nodetemplate:AzureNodeTemplate

Deploy the Package Content Locally

In order to run the batch workload on the “classic” compute nodes, too, it is helpful to deploy the package using the same environment variables as they are are available in Windows Azure. In this sample we want the CCP_PACKAGE_ROOT variable link to the local C:\TEMP folder of every machine in the cluster:

C:\Users\tlangner>cluscfg setenvs "CCP_PACKAGE_ROOT=C:\TEMP"

Then, let’s copy the zip package content to the target directory of every “classic” compute node in the cluster. The content will be deployed to the C:\TEMP\test directory:

C:\Users\tlangner>clusrun /nodegroup:computenodes xcopy
\\head\temp\AzureTestBatch\AzureTestBatch\bin\Debug\*.*
^%CCP_PACKAGE_ROOT%^\test

Run the Mixed Workload

After deploying the package content to the whole cluster it’s time to run the batch job:

C:\Users\tlangner>job submit /nodegroup:X
/parametric:1-1000 /workdir:^%CCP_PACKAGE_ROOT%^\test
azuretestbatch.exe 5000