Spark and PySpark interview questions: answers checked against the docs
The distributed data processing questions data engineering roles get, answered in plain words and checked against the Apache Spark documentation.
Spark interviews cover the driver and executors, lazy evaluation with transformations and actions, RDDs versus DataFrames, shuffles, repartition versus coalesce, caching, broadcast joins and accumulators, shuffle partitions, adaptive query execution, data skew, Python UDF performance and Structured Streaming. Below are 13 questions; each answer lists the Apache Spark 4.3.0 documentation pages it was checked against.
Basic Spark and PySpark interview questions
Basic1. What are the driver and the executors in Spark?
The driver runs your main program: it creates the SparkSession, turns your code into a plan of tasks and schedules them. Executors are processes on the worker nodes that run those tasks and hold cached data. A cluster manager (Spark standalone, YARN or Kubernetes) allocates the resources. Pulling a large result to the driver with collect() is a classic way to run out of driver memory.
Checked against: Apache Spark docs: cluster mode overview
Basic2. What is lazy evaluation? What are transformations and actions?
Transformations (select, filter, join, groupBy) only describe a new dataset; Spark records them and computes nothing yet. An action (count, collect, show, a write) needs a result, so Spark then optimizes the whole chain and runs it as jobs. Laziness lets Spark combine steps and skip work it does not need.
PySpark snippet · illustrative, not run on this pagejobs = spark.read.parquet("/data/jobs") # nothing runs yet
remote = jobs.filter(jobs.remote == True).select("title") # still nothing
remote.count() # action: the job runs nowChecked against: Apache Spark docs: RDD programming guide
Basic3. What is the difference between an RDD, a DataFrame and a Dataset?
An RDD is the low-level distributed collection of objects, with no schema. A DataFrame is distributed data organized into named columns, like a table, and its operations go through Spark SQL's optimizer, so it is usually faster. A Dataset adds compile-time types on top of that in Scala and Java; in PySpark you work with DataFrames.
Checked against: Apache Spark docs: SQL, DataFrames and Datasets
Basic4. What is a shuffle, and why is it expensive?
A shuffle redistributes data across partitions so that related rows end up together, for example for groupBy, joins on non-co-partitioned data, distinct or repartition. It writes data to disk, serializes it and sends it over the network, so it is often the slowest part of a job. Reducing shuffles, or the data that goes into them, is a common optimization.
Checked against: Apache Spark docs: RDD programming guide
Intermediate Spark and PySpark interview questions
Intermediate5. What is the difference between repartition and coalesce?
repartition(n) performs a full shuffle and can increase or decrease the number of partitions, producing evenly sized partitions. coalesce(n) only decreases the number and avoids a full shuffle by merging existing partitions, which is cheaper but can leave them uneven. A common use of coalesce is reducing the number of output files after a filter.
PySpark snippet · illustrative, not run on this pagedf.repartition(200, "customer_id") # full shuffle, hash-partitioned by key
df.coalesce(10) # fewer partitions, no full shuffleChecked against: Apache Spark docs: RDD programming guide
Intermediate6. When should you cache or persist a DataFrame?
When the same DataFrame is used by several actions, for example in an iterative algorithm or several reports from one cleaned dataset. cache() keeps it after the first action computes it; persist() lets you choose the storage level. Cached data is recomputed from its lineage if a partition is lost. Unpersist it when finished, and do not cache data that is used only once.
PySpark snippet · illustrative, not run on this pageclean = raw.filter("status = 'active'").dropDuplicates(["id"]).cache()
clean.count() # first action fills the cache
clean.groupBy("region").count().show()
clean.unpersist()Checked against: Apache Spark docs: RDD programming guide
Intermediate7. What are broadcast variables, broadcast joins and accumulators?
A broadcast variable sends one read-only copy of a value, such as a small lookup table, to every executor. A broadcast join applies the same idea to joins: the small side is copied to every executor, so the large side does not need to be shuffled. Spark does this automatically for tables below spark.sql.autoBroadcastJoinThreshold (10 MB by default), and you can force it with a hint. Accumulators are variables that tasks can only add to, such as counters.
PySpark snippet · illustrative, not run on this pagefrom pyspark.sql.functions import broadcast
result = events.join(broadcast(countries), "country_code")Checked against: Apache Spark docs: SQL performance tuning, Apache Spark docs: RDD programming guide
Intermediate8. What does spark.sql.shuffle.partitions control?
The number of partitions Spark uses after a shuffle for joins and aggregations; the default is 200. Too few partitions makes tasks large and can spill to disk; too many creates overhead from tiny tasks. With adaptive query execution, Spark can coalesce small post-shuffle partitions at runtime, which makes the exact setting matter less.
PySpark snippet · illustrative, not run on this pagespark.conf.set("spark.sql.shuffle.partitions", "64")Checked against: Apache Spark docs: SQL performance tuning
Advanced Spark and PySpark interview questions
Advanced9. What is Adaptive Query Execution?
AQE re-optimizes a query while it runs, using statistics collected at shuffle boundaries. It can coalesce small shuffle partitions, switch a sort-merge join to a broadcast join when one side turns out to be small, and split skewed partitions. It is enabled by default since Spark 3.2.0.
Checked against: Apache Spark docs: SQL performance tuning
Advanced10. What is data skew, and how do you deal with it?
Skew means a few keys hold far more rows than the rest, so a few tasks run much longer than the others while the cluster sits idle. With AQE, Spark can split skewed partitions in sort-merge joins automatically. Other fixes: broadcast the smaller side, filter or handle very large keys separately, or add a salt to spread a hot key across several partitions and aggregate in two steps.
Checked against: Apache Spark docs: SQL performance tuning
Advanced11. Why are Python UDFs slow, and what are the alternatives?
A plain Python UDF processes one row at a time, and each value has to be serialized between the JVM and Python. Prefer built-in functions in pyspark.sql.functions, which run inside the engine. When you need Python, pandas (vectorized) UDFs exchange whole batches through Apache Arrow and are usually much faster than row-at-a-time UDFs.
PySpark snippet · illustrative, not run on this pageimport pandas as pd
from pyspark.sql.functions import pandas_udf
@pandas_udf("double")
def with_tax(amount: pd.Series) -> pd.Series:
return amount * 1.2
df.withColumn("gross", with_tax("net"))Checked against: PySpark docs: UDFs and UDTFs
Advanced12. How does Structured Streaming handle late data?
Structured Streaming treats a stream as a table that keeps growing and runs incremental queries on it. For event-time windows, a watermark tells Spark how late data may arrive: data within the watermark is always aggregated, while data later than the watermark may or may not be aggregated (Spark does not promise to drop it), and old window state can be cleaned up. Checkpointing stores progress so a restarted query continues where it stopped.
PySpark snippet · illustrative, not run on this page(events.withWatermark("event_time", "10 minutes")
.groupBy(window("event_time", "5 minutes"), "page")
.count())Checked against: Apache Spark docs: Structured Streaming APIs
Advanced13. Which settings and habits help Spark performance?
Read only the columns and rows you need (columnar formats such as Parquet let Spark skip the rest), avoid collect() on large data, prefer built-in functions to UDFs, watch the Spark UI for long tasks and spills, size partitions sensibly, and keep serialization cheap: the tuning guide singles out data serialization and memory as the main topics.
Checked against: Apache Spark docs: RDD programming guide, Apache Spark docs: tuning
How to practice with this page
- Explain each answer with what happens on the cluster: which stage, which shuffle.
- Read a query plan with explain() and find the exchanges.
- Practice a join between a large and a small table and explain the strategy Spark picks.
- Review SQL window functions; many Spark tasks are written in SQL.
Related interview prep and tools
Single-machine DataFrames, checked against the docs.
Open →QuestionsDatabricks interview questionsThe lakehouse platform built on Spark.
Open →QuestionsSnowflake interview questionsCloud data warehouse questions.
Open →QuestionsKafka interview questionsStreaming sources for Spark.
Open →Free toolResume summary generatorDraft a summary for a Data Engineer role; the title is filled in, you add your experience.
Open →Free toolLinkedIn headline generatorWrite a headline for a Data Engineer search.
Open →Disclosure: TryHired makes HIRED, a paid job-search program, so we have a conflict of interest when we mention it. This question bank is free, with no TryHired signup.
Questions people ask
What Spark topics come up often in data engineering interviews?
Is PySpark different from Spark?
Were these snippets run?
How this page is built
TryHired wrote these questions and answers. Statements about the technology were checked against the official documentation pages listed below, saved when the page was built (2026-11-16). The code snippets are short illustrations and were not run on this page. The question list is general practice: it does not predict what a particular employer will ask.
- Interviewers ask different questions for different roles and levels.
- Answers are short explanations, not complete references.
- No Spark code was run on this page; the snippets assume a SparkSession named spark. The answers follow the Spark 4.3.0 documentation; defaults can differ in older versions and managed platforms.
- Apache Spark docs: cluster mode overview (read 2026-11-15)
- Apache Spark docs: SQL performance tuning (read 2026-11-15)
- Apache Spark docs: RDD programming guide (read 2026-11-15)
- Apache Spark docs: SQL, DataFrames and Datasets (read 2026-11-15)
- Apache Spark docs: Structured Streaming APIs (read 2026-11-15)
- Apache Spark docs: tuning (read 2026-11-15)
- PySpark docs: UDFs and UDTFs (read 2026-11-15)