Academia.NET

When it rains, it pours --- articles on parallel processing that is

joe-hummel — Sat, 11 Oct 2008 04:47:00 GMT

Wow, my reading pile just got a lot bigger today. First, the latest (Oct 2008) issue of MSDN contains 4 articles related to parallel processing:

"Design considerations for parallel programming"
"Improved Support for Parallelism in the next version of Visual Studio"
"Solving 11 Likely Problems in your Multithreaded Code"
"Build Concurrent Apps From Simple F# Expressions"

I'm currently reading #4, since I'm a believer that functional programming may be one of the ways we make parallel processing safer, scalable, and generally more approachable. I don't see much in F# yet, it seems right now to be just another way to gain access to PFx and MPI.NET. But the potential is definitely there.

If that isn't enough, the latest issue (Oct 2008) of ACM's Queue magazine ("Architecting Tomorrow's Computing") presents a broader view of the concurrency problem, with articles on

"Real-world Concurrency"
"Erlang for Concurrent Programming"
"Parallel Programming with Transactional Memory"

Erlang is another functional language, but with a proven process-based approach to concurrency and parallel processing. I'm just learning about Erlang now, I really really need to implement some apps and collect some performance numbers. Erlang's process-oriented concurrency model was originally designed for system building and robustness, so the question is how it fares for HPC. Transactional memory is an interesting approach --- in short, you don't worry about accessing shared memory, you just go for it and check when you're done to see if a conflict has occurred. If not, you commit, otherwise you rollback and try again.

Anyway, happy reading!

32-bit SDK for HPC Server 2008 fails --- "The procedure entry point GetProcessIdOfThread could not be located..."

joe-hummel — Sat, 11 Oct 2008 02:21:00 GMT

The RTM for HPC Server 2008 was recently released, and I've been happily running HPC apps on my clusters. I wanted to run some demos for my students, so I installed the SDK for HPC Server 2008 on the classroom machine (a 32-bit Windows XP box), and launched as usual with mpiexec:

mpiexec -n 4 MyMPIApp.exe

A dialog box immediately appeared with the error message "The procedure entry point GetProcessIdOfThread could not be located in the dynamic link library KERNEL32.dll". I closed the dialog, and the console window then displayed this error message:

ReadFile() failed, error 109
Error: unable to start the local smpd manager

A firewall problem? Disabled the firewall, same error. Reboot? Same error. Installed MPI.NET and ran an MPI.NET app; same error. So it seems to be a problem with the 32-bit SDK for HPC Server. I did find a temporary solution: uninstall the SDK, download and install the v1 SDK for Compute Cluster Server, available here. Same MPI apps ran fine with the older SDK. I've submitted a bug report to Microsoft, I'll post when we figure out what's going on.

RTM for HPC Server 2008 is now available

joe-hummel — Thu, 25 Sep 2008 20:28:00 GMT

In case you missed it, RTM for Windows HPC Server 2008 is now available here. Don't forget to grab the SDK for HPC Server, which allows you to program MPI, as well as access a cluster programmatically via the HPC Server API.

There are also a number of tutorials and how-to articles available on Microsoft's developer resources page, two of which we wrote and blogged about earlier: "Classic HPC Development using Visual C++", and "From Sequential to Parallel Code". The former is for more experienced HPC C++ developers, and uses OpenMP, MPI and MPI.NET to develop an image processing application. The latter is oriented towards newcomers to HPC and HPC Server 2008, and shows how to move step-by-step from a sequential, desktop GUI application (managed C++ or C#) to a parallel, cluster-based application. A variety of parallelization techniques are presented, including parametric sweep, explicit threads, OpenMP, TPL, and HPC Server's new SOA-based approach. Both tutorials include extensive lab exercises, solutions, and demos.

Cheers!

Learning Parallel Programming --- from shared-memory multi-threading to distributed-memory multi-processing

joe-hummel — Tue, 19 Aug 2008 03:29:00 GMT

We're working with Microsoft to develop several tutorials introducing developers to high-performance computing, parallel programming, and Windows HPC Server 2008. Our first tutorial (for experienced HPC developers, "Exploiting Data Parallelism with OpenMP, MPI and HPC Server 2008"), is available here. We are working on our second tutorial, "From Sequential to Parallel Development", which is nearing completion. This is more introductory in nature, taking the traditional "sequential" developer step-by-step into the world of parallel programming. The tutorial is extensive, currently 50 pages in length, complete with source code, lab exercises, and solutions in both VC++ and C#.

We start with a GUI-based Mandelbrot application, and show various ways to parallelize the application via multi-threading --- explicitly with .NET threads, then implicitly with OpenMP and the TPL (Task Parallel Library in the upcoming PFx, or Parallel Extensions to .NET). We then architect a client-server version of the Mandelbrot app, introduce Windows HPC Server 2008, and show how to parallelize the app across a cluster. This includes a discussion of many topics, including:

Multi-threading
Multi-processing
BackgroundWorker and Thread classes
OpenMP
TPL
MPI
MPI.NET
programming against the HPC Server 2008 API
automating application deployment and job submission

The goal is to show developers how to take a sequential, GUI-based application and parallelize this app across both multi-core and cluster-based hardware. Here's the current draft of the tutorial document: SeqToParallelHPC.pdf. The current download including source code and lab exercises: SeqToParallelHPC.zip. Any and all feedback greatly appreciated. Enjoy!

Exploiting Data Parallelism with OpenMP, MPI and Microsoft HPC Server 2008

joe-hummel — Thu, 19 Jun 2008 15:35:00 GMT

We're working with Microsoft to develop some tutorials for the HPC Server 2008 SDK. We just finished one of the tutorials, representing a classic HPC development scenario: “Exploiting Data Parallelism with OpenMP, MPI and Microsoft HPC Server 2008”. The scenario revolves around the parallelization of an image processing application, and includes both lab exercises and solutions. The tutorial is extensive, 74 pages covering a multitude of topics in the HPC space:

OpenMP
MPI
MPI.NET
MPI tracing with ETW and Jumpshot
MPI debugging --- locally on your workstation and remotely on the cluster
Submitting workloads to an HPC Server 2008 cluster
Troubleshooting submission problems
HPC Server 2008 cluster setup

Here's the tutorial document: ClassicHPCDevC++.pdf. The complete download: ClassicHPCDevC++.zip. Enjoy!

March CTP of HPC Server 2008 just released

joe-hummel — Thu, 13 Mar 2008 16:54:00 GMT

Microsoft has just released a CTP of its upcoming HPC Server 2008 product (the next version of Windows Compute Cluster Server 2003). The MPI library has undergone a series of performance enhancements (e.g. Network Direct for more efficient message communication), and the scheduler now provides an easier and more interactive way to execute kernels of work (based on the scheduler's new SOA capabilities for job submission and execution). I'm just starting to kick the tires, existing MPI and OpenMP apps runs just fine on the new version. Next up is to start playing with the new scheduler...

The CTP also includes numerous other improvements, including better setup and administration support. To download, click here to join the beta program.

incorrect MSMPE 64-bit library installed as part of SDK for Compute Cluster Pack

joe-hummel — Thu, 21 Feb 2008 04:06:00 GMT

During the first run of our Windows Compute Cluster Server class last week in Irvine, we discovered that when you install the SDK for Compute Cluster Pack, the wrong version of msmpe.lib is installed. In particular, the 64-bit msmpe.lib file, which gets installed into C:\Program Files\Microsoft Compute Cluster Pack\Lib\amd64, is actually the 32-bit library. So when you compile in 64-bit mode and link against the 64-bit msmpe.lib, you get unresolved symbols for the MPI functions: MPI_Init, MPI_Send, MPI_Recv, MPI_Finalize, etc. Microsoft has built a correct version of the library, which I've made available here for download.

Some background... The SDK for Compute Cluster Pack is the MPI programming libraries for Windows Compute Cluster Server. There are 2 libraries each for 32-bit and 64-bit. For normal MPI programming, you link against msmpi.lib. For MPI programming with MPE-based event tracing, you link against msmpe.lib. The 32-bit msmpe.lib file is fine, the 64-bit msmpe.lib file is incorrect.

Microsoft HPC Server 2008

joe-hummel — Mon, 19 Nov 2007 14:18:00 GMT

At Supercomputing last week, Microsoft announced the next version of its HPC product: HPC Server 2008. The current version (”Windows Compute Cluster Server” or WCCS) is built upon Server 2003, while this next version is built upon Server 2008. I'll be blogging about v2 in the coming weeks, for now you can find additional information here. In particular, there's technical information available as a Word .docx, and you can download beta1 if you want to play.

Speaking of playing... I mentioned in an earlier post that you can build your own cluster with very little hardware, e.g. I'm using 2 dual-core Mac Minis. Phil Pennington just posted how to setup a cluster on your laptop / desktop using virtualization, very cool. Just add lots of RAM, and hopefully you have a few cores free on your CPU :-)

Supercomputing 2007: "Data Streaming"

joe-hummel — Fri, 16 Nov 2007 08:24:00 GMT

Michael Wolfe, PhD, is a well-known compiler writer and parallelization guru, and currently works for The Portland Group. He gave a vendor talk at Supercomputing this week (Reno, NV) on what he calls “Data Streaming”. He started by showing a timeline of CPU clock rates from 1997 ... 2007. There was a clear trend: from 1997 .. 2002, the clock rates followed Moore's law and grew by a factor of 10 --- from 300MHz to 3GHz. But from 2002 to 2006, the rates barely grew at all (3.8GHz), and in 2007 the clock rates actually dropped back to 3GHz. So we appear to have hit the wall in terms of CPU clock rates. As you all know, what chip manufacturers are doing now is going multi-core to offer increased performance.

But forget about multi-core for a moment. Let's look at just one core running at 3GHz. In the good 'ole days of compiler optimization, we optimized by reducing instruction count / instruction cycles. We have become pretty good at that, to the point that we can't feed the CPU with data fast enough --- Mike did some back-of-the-envelope calculations that I won't repeat here, but basically showed that optimized, compute-intensive code needs 16-24GB/Sec bandwidth to memory. CPUs today don't have anywhere near that bandwidth to RAM.

Okay, so add another core. Michael showed this very simple picture of a two-headed straw: “Imagine you're at your favorite dinner sharing a milkshake with your significant other” (my words). Both of you are pulling through the same base --- i.e. with two cores you still have only 1 pipe to memory. This problem is going to get bad very quickly, since the # of cores will grow exponentially (quad-cores are getting common, 8-cores by next year, etc.). But the # of pins to memory will not grow exponentially, though it will grow (AMD's new Barcelona quad-core chip has 2 pipes to memory). Think of the multi-headed straw as you ponder the purchase of your next quad-core box...

The answer is to (a) throw more and more cache at the CPU (L1, L2, L3), and (b) start programming & optimizing for the cache --- “to the point that you consider the cost of executing an instruction to be $0.00” (Michael's words). So “Data Streaming” is the idea that instead of always agressively doing computation, you instead aggressively fill the cache with data you will need, and then do computation once the data arrives. If you have 2 cores, then core #1 can be reading data while core #2 is computing; when core #2 runs out of data, it starts reading data while core #1 executes. This sounds crazy --- to have an execution core basically spend its time waiting for data --- but it turns out that you get much better performance (in compute-intensive programs) than having both cores go full-speed ahead executing instructions, and essentially both waiting for data. Michael and a colleague did this for a spec benchmark, and got around 1.5-1.7 speedup on a 2-core CPU (vs. essentially no speedup when the code was optimized for instruction counts).

Here's a simple example of what you can do today to optimize for better cache performance. First, imagine a collection of objects or structs. Second, suppose you have to run through the collection, touching a field of each object / struct:

for (int i = 0; i < N; i++)
a[i].field++;

Every time you read from memory, the hardware reads a cache line --- 8 or 16 bytes (at least). By using an array of objects / structs, what you just did was pollute the cache with things you are never going to use (the other fields of the struct), to get at the one field you will use. The solution is to move back to the old days of parallel arrays, i.e. using separate arrays to hold the various fields. The loop then becomes:

for (int i = 0; i < N; i++)
afield[i]++;

Now when you read a cache line, you'll use those other values in the cache line in future iterations. The next step is to add prefetch instructions (which VS supports) to get ahead:

for (int i = 0; i < N; i++) {
prefetch( afield[i+8] ); // or something, this is very HW dependent
afield[i]++;
}

This is starting to approximate the “Data Streaming” idea, in that instead of doing computation, we think first about streaming in the data we're going to need. The point of Michael's talk is that we need to start thinking about this more deeply. I suspect Michael has some good ideas about how the compiler can help us.

And here's a related item: the TLB (translation lookaside buffer), which is responsible for translating addresses from virtual memory to physical memory, is another potential area of concern. The TLB is in essence another cache, and is getting killed as we go multi-core --- if the cores are running different instruction streams, the TLB miss rate is growing and this means more trips to RAM, and this means more waiting. Rough rule of thumb: the cost of each memory level is a factor of 10, so CPU to L1 is 10 cycles, CPU to L2 is 100, CPU to L3 is 1,000, and CPU to RAM is 10,000 cycles. So you can see that waiting 10,000 cycles to read something from RAM is a BAD thing for performance.

The bottom line: getting good performance from today's multi-core chips is not as easy as just creating multiple threads and handing them off to the operating system. Fun times!

Microsoft Compute Cluster Server

joe-hummel — Thu, 15 Nov 2007 08:36:00 GMT

My PhD is in parallelizing compilers, back in the 1990's when supercomputers were hot. Then the field cooled off, as the machines were too costly, the programming models were too difficult, and the auto-parallelizing compilers failed to live up to (unrealistic) expectations. But the field has come roaring back to life the last couple years, especially with cheap high-speed interconnects and multi-core CPUs. Microsoft has a number of technologies in the HPC (high performance computing) arena: Parallel LINQ, Task Parallel Library, F#, and Compute Cluster Server.

Microsoft Compute Cluster Server, or CCS, is Microsoft's foray into the HPC cluster arena --- imagine a rack of blades with quad-core CPUs and a high-speed interconnect. CCS was released in 2006, runs on Windows Server 2003 64-bit, and is currently at v1.0 with SP1; v2 was just announced at Supercomputing this week in Reno, and will run on Windows Server 2008 64-bit.

I'm really excited to be working in the field again, finally putting all that PhD work to use :-) Right now I'm part of a team here at Pluralsight developing a 5-day course on CCS: what it offers, how it works, and its supported programming models (basically OpenMP and MPI). Our first course offering is slated for the last week of November, 26-30. For more info on the course, click here .

Surprisingly, it doesn't take that much hardware to get into CCS and experiment. A CCS cluster consists of a head node (that manages the cluster and schedules jobs) and compute nodes (which do the actual work). Since a head node can also server as a compute node, you can build a cluster with one computer --- any computer capable of running Windows Server 2003 64-bit. Of course, life is more interesting if you have 2 or more compute nodes, which forces you into the world of distributed programming and MPI.

In fact, take a peek at my portable cluster --- I'm using 2 Mac Minis as compute nodes and a Thinkpad as the head node (the other thinkpad is my normal working machine, and acts as a client to the cluster):

The mac minis make excellent portable Windows servers. I booted to the mac os, ran bootcamp, inserted the Windows 20003 CD, wiped the disk clean when it rebooted, and installed. Some of the drivers are missing (e.g. bluetooth and wireless), but the server hardware I needed --- monitor, hard disk and ethernet --- were found and installed fine. Once installed, I run them headless (you have to boot with a dvi to s-video adapter attached because the mini is looking for a monitor, but once booted you can remove the adapter and remote into the machine).

Right now I'm attending the annual Supercomputing conference, which is a great conference and getting better every year. 9,000 attendees, 87 miles of fiber optic cable, and more computing power than most countries. Tomorrow I'll blog about one of the things I learned that blew me away: the cost of executing an instruction is essentially 0, so optimization now is all about data locality and caching. The optimizing compiler world has been turned on its head.

Interested in LINQ?

joe-hummel — Sat, 04 Nov 2006 00:51:00 GMT

I'm sure you've heard about LINQ, but just in case you haven't, it stands for Language Integrated Query. I just finished a 60-page overview of LINQ, which is available as a PDF from O'Reilly:

http://www.oreilly.com/catalog/language1/?CMP=ILC-2RQ886833906&ATT=language1

The PDF represents the first in a 3-part series on LINQ; part 2 will focus specifically on LINQ for SQL, and part 3 will focus on LINQ for XML. Along with the PDFs I'll be presenting a series of MSDN webcasts on LINQ; I'll let you know when the live presentation dates become official.

LINQ offers SQL-like query support in C# and VB, allowing you to write queries --- against objects, XML documents, relational databases, and more --- with IntelliSense and strict type-checking. For example, given a set of Doctor objects, here's a query to select all the doctors living in Chicago:

var chicago = from d in doctors where d.City == “Chicago“ select d;

There's a CTP (May 2006) you can download in order to play with LINQ [1]. The technology is slated to appear sometime in 2007 with the 3.next release of .NET (i.e. the release to *follow* the upcoming 3.0 release). It's very interesting technology, I encourge you to learn about LINQ if you haven't already. Cheers!

[1] http://msdn.microsoft.com/data/ref/linq/

MSDNAA does provide access to Visual Studio Team System

joe-hummel — Thu, 17 Aug 2006 15:02:00 GMT

For the academics in the crowd...

I was under the impression that Microsoft's Academic Alliance was providing Visual Studio Team System as trial software that expires after 180 days. Turns out that if you contact MSDNAA, you can get a volume licensing key to distribute the full version of VSTS (“Visual Studio Team Suite“) to your students and install on your lab machines. VSTS includes software engineering tools and processes for the entire software lifecycle.

I'm not sure how to exactly go about getting the licensing key from MSDNAA, that's the next step :-)

Building documentation from XML comment files

joe-hummel — Sun, 06 Aug 2006 01:36:00 GMT

I've been a big fan of NDoc for generating professional looking documentation from XML comment files. Unfortunately, it seems development on NDoc has stalled of late, and support for generating documentation for .NET 2.0 code is still beta. The good news is that Microsoft has just released a CTP of the tool they use internally, called Sandcastle. This is an early (i.e. rough) release, but a good step in the right direction. Here's a blog entry from MS with more info:

http://blogs.msdn.com/sandcastle/default.aspx

See the bottom of that blog page for the download link. Note it's an early beta with a command-line interface. Cheers!

Rocky Lhotka and CSLA 2.0

joe-hummel — Sun, 02 Apr 2006 16:50:00 GMT

Webcast listeners know I'm a big fan of Rocky's work on distributed design. He's designed a framework (CSLA) for distributed N-tier systems, and recently updated his framework for .NET 2.0. If you ever get a chance to hear him speak at TechEd or PDC, do so. In fact, turns out Rocky was recently on .NET Rocks, and you can hear him talk about CSLA 2.0. Here's the link:

http://www.dotnetrocks.com/default.aspx?showID=172

Kudos to Andy M. for the heads up and link. Cheers!

Web Service URL should be a config setting --- it already is!

joe-hummel — Thu, 09 Feb 2006 03:52:00 GMT

During today's webcast on web services (Wed Feb 8), there was a question related to moving your web service from your dev box to the production box. When the web service moves, the URL will change, so what's the best way to protect your client code from this predictable change? We shouldn't need to re-reference and recompile the client app just because the web service moved. The answer of course is that the URL should be a .config setting. The detail I forgot was exactly what property to set at run-time after you read the URL from the .config file.

Duh, it's the .Url property! Let me finish this story, and then tell you an even better one :-) But first, the .Url property. The client starts by createing the web service object (which is really the proxy), and then sets the URL like this:

this.server = new EmployeeWebService.Employees();

this.server.Url = Properties.Settings.EmployeeWebServiceURL;

This assumes you have defined a .config setting named EmployeeWebServiceURL. It may seem backwards to create the web service object first and set the URL second, since don't you need the URL to create the web service object? Nope, because you're really just creating the proxy --- the web service isn't contacted until you make a method call, and that's when you need the URL.

So that's the first part of the story: create .config setting, and set the proxy's Url property before you call it. So off I got to update my demo code in VS 2005, I bring up the Properties page, click the Settings tab, and behold, the .config setting is already there! Turns out Visual Studio 2005 automatically defines an application-level setting for the project whenever you add a web reference. So in my demo code, in the BusinessTierClient project, there's a setting called “BusinessTierClient_EmployeeWebService_Employees“ that contains the URL for the web service. And the proxy is already coded to read this setting, so if you chance it, the proxy does the right thing. Very cool.

The only problem is that this .config setting is stored in the component's app.config file, which for a DLL, isn't around at run-time. So to make this work the way you want it to --- i.e. to expose the .config setting in the client-side .exe's config file --- you have to merge the DLL's app.config file with the client-side .exe's config file. We've done this already with other settings, e.g. the connection string needed by Data Access Tier has to be merged into web.config (for a web service) or remotingserver.exe.config (for a remoting server host). I'll update my demo and repost the demo + slides to the webcasts page, in this case app.config file associated with the EmployeeClientGUI has been updated, that's it.

Learning something new every day... Cheers,