Skip to content
TryHired.
Interview prep · Technical · checked against the official docs

Spark and PySpark interview questions: answers checked against the docs

The distributed data processing questions data engineering roles get, answered in plain words and checked against the Apache Spark documentation.

Updated Nov 16, 2026 · Facts checked against the official documentation · questions: TryHired
Quick answer

Spark interviews cover the driver and executors, lazy evaluation with transformations and actions, RDDs versus DataFrames, shuffles, repartition versus coalesce, caching, broadcast joins and accumulators, shuffle partitions, adaptive query execution, data skew, Python UDF performance and Structured Streaming. Below are 13 questions; each answer lists the Apache Spark 4.3.0 documentation pages it was checked against.

Basic

Basic Spark and PySpark interview questions

Advice

Basic1. What are the driver and the executors in Spark?

The driver runs your main program: it creates the SparkSession, turns your code into a plan of tasks and schedules them. Executors are processes on the worker nodes that run those tasks and hold cached data. A cluster manager (Spark standalone, YARN or Kubernetes) allocates the resources. Pulling a large result to the driver with collect() is a classic way to run out of driver memory.

Checked against: Apache Spark docs: cluster mode overview

Basic2. What is lazy evaluation? What are transformations and actions?

Transformations (select, filter, join, groupBy) only describe a new dataset; Spark records them and computes nothing yet. An action (count, collect, show, a write) needs a result, so Spark then optimizes the whole chain and runs it as jobs. Laziness lets Spark combine steps and skip work it does not need.

PySpark snippet · illustrative, not run on this page
jobs = spark.read.parquet("/data/jobs")                     # nothing runs yet
remote = jobs.filter(jobs.remote == True).select("title")    # still nothing
remote.count()                                              # action: the job runs now

Checked against: Apache Spark docs: RDD programming guide

Basic3. What is the difference between an RDD, a DataFrame and a Dataset?

An RDD is the low-level distributed collection of objects, with no schema. A DataFrame is distributed data organized into named columns, like a table, and its operations go through Spark SQL's optimizer, so it is usually faster. A Dataset adds compile-time types on top of that in Scala and Java; in PySpark you work with DataFrames.

Checked against: Apache Spark docs: SQL, DataFrames and Datasets

Basic4. What is a shuffle, and why is it expensive?

A shuffle redistributes data across partitions so that related rows end up together, for example for groupBy, joins on non-co-partitioned data, distinct or repartition. It writes data to disk, serializes it and sends it over the network, so it is often the slowest part of a job. Reducing shuffles, or the data that goes into them, is a common optimization.

Checked against: Apache Spark docs: RDD programming guide

Intermediate

Intermediate Spark and PySpark interview questions

Advice

Intermediate5. What is the difference between repartition and coalesce?

repartition(n) performs a full shuffle and can increase or decrease the number of partitions, producing evenly sized partitions. coalesce(n) only decreases the number and avoids a full shuffle by merging existing partitions, which is cheaper but can leave them uneven. A common use of coalesce is reducing the number of output files after a filter.

PySpark snippet · illustrative, not run on this page
df.repartition(200, "customer_id")   # full shuffle, hash-partitioned by key
df.coalesce(10)                       # fewer partitions, no full shuffle

Checked against: Apache Spark docs: RDD programming guide

Intermediate6. When should you cache or persist a DataFrame?

When the same DataFrame is used by several actions, for example in an iterative algorithm or several reports from one cleaned dataset. cache() keeps it after the first action computes it; persist() lets you choose the storage level. Cached data is recomputed from its lineage if a partition is lost. Unpersist it when finished, and do not cache data that is used only once.

PySpark snippet · illustrative, not run on this page
clean = raw.filter("status = 'active'").dropDuplicates(["id"]).cache()
clean.count()                          # first action fills the cache
clean.groupBy("region").count().show()
clean.unpersist()

Checked against: Apache Spark docs: RDD programming guide

Intermediate7. What are broadcast variables, broadcast joins and accumulators?

A broadcast variable sends one read-only copy of a value, such as a small lookup table, to every executor. A broadcast join applies the same idea to joins: the small side is copied to every executor, so the large side does not need to be shuffled. Spark does this automatically for tables below spark.sql.autoBroadcastJoinThreshold (10 MB by default), and you can force it with a hint. Accumulators are variables that tasks can only add to, such as counters.

PySpark snippet · illustrative, not run on this page
from pyspark.sql.functions import broadcast
result = events.join(broadcast(countries), "country_code")

Checked against: Apache Spark docs: SQL performance tuning, Apache Spark docs: RDD programming guide

Intermediate8. What does spark.sql.shuffle.partitions control?

The number of partitions Spark uses after a shuffle for joins and aggregations; the default is 200. Too few partitions makes tasks large and can spill to disk; too many creates overhead from tiny tasks. With adaptive query execution, Spark can coalesce small post-shuffle partitions at runtime, which makes the exact setting matter less.

PySpark snippet · illustrative, not run on this page
spark.conf.set("spark.sql.shuffle.partitions", "64")

Checked against: Apache Spark docs: SQL performance tuning

Advanced

Advanced Spark and PySpark interview questions

Advice

Advanced9. What is Adaptive Query Execution?

AQE re-optimizes a query while it runs, using statistics collected at shuffle boundaries. It can coalesce small shuffle partitions, switch a sort-merge join to a broadcast join when one side turns out to be small, and split skewed partitions. It is enabled by default since Spark 3.2.0.

Checked against: Apache Spark docs: SQL performance tuning

Advanced10. What is data skew, and how do you deal with it?

Skew means a few keys hold far more rows than the rest, so a few tasks run much longer than the others while the cluster sits idle. With AQE, Spark can split skewed partitions in sort-merge joins automatically. Other fixes: broadcast the smaller side, filter or handle very large keys separately, or add a salt to spread a hot key across several partitions and aggregate in two steps.

Checked against: Apache Spark docs: SQL performance tuning

Advanced11. Why are Python UDFs slow, and what are the alternatives?

A plain Python UDF processes one row at a time, and each value has to be serialized between the JVM and Python. Prefer built-in functions in pyspark.sql.functions, which run inside the engine. When you need Python, pandas (vectorized) UDFs exchange whole batches through Apache Arrow and are usually much faster than row-at-a-time UDFs.

PySpark snippet · illustrative, not run on this page
import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def with_tax(amount: pd.Series) -> pd.Series:
    return amount * 1.2

df.withColumn("gross", with_tax("net"))

Checked against: PySpark docs: UDFs and UDTFs

Advanced12. How does Structured Streaming handle late data?

Structured Streaming treats a stream as a table that keeps growing and runs incremental queries on it. For event-time windows, a watermark tells Spark how late data may arrive: data within the watermark is always aggregated, while data later than the watermark may or may not be aggregated (Spark does not promise to drop it), and old window state can be cleaned up. Checkpointing stores progress so a restarted query continues where it stopped.

PySpark snippet · illustrative, not run on this page
(events.withWatermark("event_time", "10 minutes")
       .groupBy(window("event_time", "5 minutes"), "page")
       .count())

Checked against: Apache Spark docs: Structured Streaming APIs

Advanced13. Which settings and habits help Spark performance?

Read only the columns and rows you need (columnar formats such as Parquet let Spark skip the rest), avoid collect() on large data, prefer built-in functions to UDFs, watch the Spark UI for long tasks and spills, size partitions sensibly, and keep serialization cheap: the tuning guide singles out data serialization and memory as the main topics.

Checked against: Apache Spark docs: RDD programming guide, Apache Spark docs: tuning

Practice

How to practice with this page

Advice
  • Explain each answer with what happens on the cluster: which stage, which shuffle.
  • Read a query plan with explain() and find the exchanges.
  • Practice a join between a large and a small table and explain the strategy Spark picks.
  • Review SQL window functions; many Spark tasks are written in SQL.
Next

Related interview prep and tools

Disclosure: TryHired makes HIRED, a paid job-search program, so we have a conflict of interest when we mention it. This question bank is free, with no TryHired signup.

FAQ

Questions people ask

What Spark topics come up often in data engineering interviews?
Driver and executors, lazy evaluation, transformations versus actions, narrow versus wide transformations and shuffles, partitioning, caching, joins and broadcast joins, skew, AQE, UDFs, and Structured Streaming basics.
Is PySpark different from Spark?
PySpark is Spark's Python API. The engine and the concepts are the same; PySpark adds Python-specific topics such as UDF performance and pandas interoperability.
Were these snippets run?
No. There is no Spark runtime on the build machine, so the PySpark snippets are illustrations and no output is shown. They assume an existing SparkSession called spark and DataFrames with the columns used.
Method

How this page is built

TryHired wrote these questions and answers. Statements about the technology were checked against the official documentation pages listed below, saved when the page was built (2026-11-16). The code snippets are short illustrations and were not run on this page. The question list is general practice: it does not predict what a particular employer will ask.

What this page can't tell you
  • Interviewers ask different questions for different roles and levels.
  • Answers are short explanations, not complete references.
  • No Spark code was run on this page; the snippets assume a SparkSession named spark. The answers follow the Spark 4.3.0 documentation; defaults can differ in older versions and managed platforms.
Sources
  1. Apache Spark docs: cluster mode overview (read 2026-11-15)
  2. Apache Spark docs: SQL performance tuning (read 2026-11-15)
  3. Apache Spark docs: RDD programming guide (read 2026-11-15)
  4. Apache Spark docs: SQL, DataFrames and Datasets (read 2026-11-15)
  5. Apache Spark docs: Structured Streaming APIs (read 2026-11-15)
  6. Apache Spark docs: tuning (read 2026-11-15)
  7. PySpark docs: UDFs and UDTFs (read 2026-11-15)