A guide to hiring an MPI developer
MPI is the common way to write programs that run on many processes at once and pass data to each other as messages. It is the backbone of a large share of scientific and engineering software. Few teams need it, and those that do often struggle to find someone who has used it on real jobs. The sections below explain the work, the signs of a strong candidate and the habits that make remote work go well. They are written for research leads, engineering managers and founders whose code has outgrown one computer.
How a parallel job runs
An MPI program starts as many copies of itself. Each copy, called a rank, owns part of the data and works on it. When a rank needs a value held by another, it asks for it with a message. Between steps the ranks often meet to share results. The code that decides who owns what, and who talks to whom, is where most of the skill lies.
The jobs people hire for
Some teams need a new parallel program written from scratch. Others have a serial program that is now too slow and must be moved to many processes. A third group has an MPI program that already runs but hangs, scales badly or gives different answers from run to run. Each job calls for a different habit, so say which one you have when you talk to candidates.
Where an MPI developer fits among other roles
A C++ developer writes fast code for one machine and may know threads well but never have sent a message between nodes. A Python developer builds the data and analysis layer that often sits around a parallel core. A DevOps or cluster administrator keeps the machines running. The MPI developer is the person who makes the program itself work well across all of them.
When an MPI developer makes sense
- The job no longer fits in the memory of one machine.
- A run takes days, and your team needs it to take hours.
- A parallel program hangs at random, or the answers change between runs.
- Adding more nodes stops making the job faster.
- The scientist who wrote the code has left, and nobody else can maintain it.
If one bigger machine or a library that is already parallel solves the problem, you may not need custom MPI code. A developer who says so early saves you money.
Ask about the failures
A parallel demo is easy to produce. Ask what the candidate did the last time a job hung, or ran slower on more nodes. Strong answers describe how they found the cause, for example a rank that waited for a message nobody sent, and what they changed. Weak answers list tools without a story. Anyone who has done this work for real will have several stories.
Reading an MPI developer’s background
Request examples of programs that ran on many nodes, along with the language, the size of the runs and the problem that was solved. It helps when the person can describe a result such as a simulation that now finishes sooner, in their own words. Treat with care a background that only lists library names.
Questions to ask in interviews
- How do you decide how to split a large grid or data set between ranks?
- When do you choose a non-blocking call over a blocking one?
- How would you find the cause of a job that hangs on some runs only?
- What do you check to confirm a parallel result matches the serial one?
- How do you read a scaling plot, and what do you try when it flattens?
Good answers are specific and include real cases. Be wary of anyone who treats MPI as a list of function names.
Try a paid mini project
Give each finalist one small piece of your own problem. Good examples are a loop split across ranks with a check against the serial answer, a profile of a slow step with a short written report, or a fix for a known hang. Pay for the time spent. Then compare how the person tests, how readable the code is and how clearly they explain what they found.
Agree how results are checked
Parallel code can look fine and still be wrong. Agree early how results will be compared, what size of difference is acceptable and which test cases run before every change. Keep the code in a repository your company owns, together with the job scripts and a short note on how the data is divided. That note will save a great deal of time for the next person.
Working with your team
Walk your developer through the science or the business reason behind the program, because it tells them which shortcuts are safe. A brief weekly meeting and one shared list of tasks keep the whole team on the same page. Share access to the cluster early, since queue waits often set the real pace of the job.
Common mistakes
- Testing only on two or four processes.
- Splitting the work evenly in theory but not in the real data.
- Skipping a comparison with the serial result after each change.
- Rewriting everything before measuring where the time goes.
Looking at a wider role? Take a look at our C++ developers for fast native code, or our Python developers for data and research tools.