A guide to hiring a Hadoop developer
Apache Hadoop is open source software for storing and processing very large data across many machines. It grew up in the years when companies first needed to analyse more data than one server could hold, and many of those clusters still run today. A Hadoop developer keeps them healthy and makes them useful. If your reports, feeds or models depend on a cluster, or you are planning to leave one, this is the role to look at.
What the job covers
A Hadoop developer works on the data layer, not on the website or the app. They decide how files are laid out, how jobs are scheduled and how cluster capacity is shared. They read job logs, find why a run is late and fix the cause. They also write or review the Hive and Spark code that turns raw files into tables people can use. Much of the value is in steady care: jobs that finish on time, disks that never fill up and a team that knows who to call.
HDFS and file layout
HDFS is the file system of Hadoop. It splits big files into blocks and stores copies on several machines. Good layout matters. Many tiny files slow everything down, and a poor folder plan makes queries read more than they need. Column file formats such as Parquet and ORC cut the amount of data a query has to scan. A strong developer plans folders and partitions with the questions people ask in mind, and cleans up when old data piles up.
YARN and shared capacity
YARN decides which job gets memory and processor time. When many teams share one cluster, queues and limits keep one heavy job from starving the rest. A developer who understands YARN can tell whether a job is slow because it is waiting, because it asked for too little memory or because it was given too much. Ask candidates how they would investigate a job that keeps being killed.
Hive and SQL on big data
Hive lets analysts query files in HDFS with SQL-like statements. It works well when tables are partitioned in a way that matches how people filter data. It works badly when every query scans everything. The developer’s job is to keep table definitions clear, choose sensible partitions and tune queries that are too slow. For general query and warehouse work outside a cluster, see our database developers.
Spark on Hadoop
Spark is a processing engine that can run on YARN and read data from HDFS. Teams use it for transformations, joins and machine learning inputs, in Scala or Python. The usual problems are large shuffles, uneven data and jobs that ask for more memory than they use. A good developer reads the job plan, finds the costly step and rewrites it. If your Spark code is mostly Python, our Python developers may also fit.
Pipelines and scheduling
A pipeline is a chain of jobs that moves data from its source to a table someone trusts. It needs a scheduler, retries, alerts and a clear record of each run. Tools such as Airflow and Oozie handle this. The key habit is making each job safe to run twice, so a rerun after a failure does not double the numbers. Ask how a candidate would find out that a nightly job produced the wrong totals.
Security and access
Data on a cluster often includes customer or finance records. Kerberos is the usual way to confirm who is who, and tools such as Ranger set who can read what. A developer should know how to give people the access they need and no more, and how to show who touched a table. Share your rules early so they are built in and not added later.
Keeping a cluster running and moving off it
Some teams plan to stay on Hadoop for years, and others want to leave. Both are reasonable. Staying means patching, capacity planning and replacing old hardware. Leaving means choosing a target, such as Amazon EMR, Databricks, Snowflake or BigQuery, then moving one group of jobs at a time. The safest method is to run old and new side by side, compare the results and switch only when they match. A Hadoop developer can do both jobs, and the same person who knows the old code is well placed to move it.
How this role sits next to others
A data engineer usually works across SQL, pipelines and cloud warehouses, and may not have run a cluster. A database developer focuses on tables and queries inside a database. A Hadoop developer sits in between, with a deep view of distributed storage and jobs. Tell us what you run, and we will match the right person.
Typical reasons to hire
Some clients lost the one person who understood the cluster. Others have nightly jobs that keep running late, or a bill that no one can explain. Some have a deadline to leave old hardware and want a migration plan. The skills differ for each, so tell us which one is yours.
When a Hadoop developer makes sense
- Business reports or feeds depend on a cluster you already run.
- Jobs are slow, late or fail without a clear reason.
- The person who built the pipelines has left.
- You plan to move data and jobs to a cloud platform.
- You need to tighten access to sensitive data on a cluster.
If you are starting a new data platform from nothing, a cloud warehouse and a general data engineer may be simpler to run and staff. An engineer who tells you that at the start is working for you.
How to read a candidate’s history
Ask about one cluster they looked after, what ran on it and what went wrong. Have them describe how they found a slow job and what they changed. A list of tool names says little, since anyone can write Hive on a resume. A stronger sign is a person who can explain a night they were paged and the fix that stopped it coming back.
Questions that open up a good talk
- How would you find out why a daily job now runs far longer than before?
- What causes many small files in HDFS, and how do you deal with them?
- How do you make a pipeline safe to run twice?
- How would you check that a migrated table matches the old one?
- How do you give an analyst access to one table without opening the rest?
Look for reasons and real examples. Be careful with anyone who talks only about the size of the data and not about how they keep it reliable.
A short paid task
Set finalists the same small job taken from your own platform, such as reviewing one slow Hive query, reading a Spark job plan or drawing the order in which you would move three jobs to the cloud. Pay for the hours. Then compare the answers, the questions each person asked and how plainly they explain their choices.
Keeping the platform in good shape
Settle who approves a change, how jobs get tested before they go live and who watches the alerts. Ask for a short note listing each job, its owner, its schedule and what depends on it. Keep the cluster accounts, the code repository and the cloud accounts under your own name, with one login per person, so nothing is lost when people change.
The first weeks
Give your new developer read access to the cluster, the job code, the scheduler and the monitoring. A weekly call and one shared board make progress visible. Bring up the rough spots early, such as the job that is always late or the table nobody dares to change, since that is where a Hadoop specialist helps first.
Common slips
- Choosing on the name Hadoop alone, with no sign of running a live cluster.
- Moving everything to the cloud at once instead of job by job.
- Switching over before the new results are compared with the old ones.
- Leaving old data and small files to pile up until the disks are full.
Related help: our SQL developers suit reporting and query work, and our AWS developers suit a move to cloud services.