Retrieve Azure Databricks cell output

Question

How can I retrieve the output of a specific cell in an Azure Databricks notebook?

Answer 1

To fetch the output of a single cell in an Azure Databricks notebook, you can use the display() function or the %python magic command to capture the output of that cell. Here's how you can do it:

1. Using display() function (Python or Scala):

In a Databricks notebook cell, if you have a command or expression that produces some output (e.g., a DataFrame or plot), you can use the display() function to capture and show the output.

# Python example
df = spark.createDataFrame([(1, "John"), (2, "Alice"), (3, "Bob")], ["id", "name"]) display(df)


// Scala example
val df = Seq((1, "John"), (2, "Alice"), (3, "Bob")).toDF("id", "name") display(df)

The display() function will render the output of the DataFrame or any other visualizations directly below the cell where it is called.

2. Using %python magic command (Python only):

If you are using a Python cell and want to store the output of a command or expression in a variable, you can use the %python magic command.

pythonCopy code

%python df = spark.createDataFrame([(1, "John"), (2, "Alice"), (3, "Bob")], ["id", "name"])

The %python magic command will execute the Python code in the cell and store the output in the variable df, which you can use in subsequent cells.

Note that the %python magic command is specific to Python cells, and you can use %scala for Scala cells if needed.

Keep in mind that if you want to use the output from one cell in another cell, you must ensure that the cells are executed in order. The output of a cell becomes available for use in later cells after it has been executed. If you want to control the order of cell execution, you can use the "Run All" or "Run All Above" options from the Databricks notebook toolbar.