In PySpark, globals() is a Python built-in function that returns a dictionary representing the current global symbol table. It contains all global variables (functions, classes, objects, and variables) defined in the current script or interactive session.
While globals() is not specific to PySpark, you can use it in PySpark scripts to inspect or manipulate global variables, including PySpark DataFrame objects.
Example: -
from pyspark.sql import SparkSession, DataFrame
spark = SparkSession.builder.appName("Globals Example").getOrCreate()
# Define some DataFrames
df1 = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ["id", "name"])
df2 = spark.createDataFrame([(3, 'Charlie')], ["id", "name"])
# Check for all DataFrame objects
for name, obj in globals().items():
if isinstance(obj, DataFrame):
print(f"{name} is a DataFrame")
df1 is a DataFrame
df2 is a DataFrame