pyspark 'DataFrame' object has no attribute '_get_object_id'

Posted by Arica Deslauriers on Friday, May 17, 2024

I am trying to run some code, but getting error:

'DataFrame' object has no attribute '_get_object_id'

The code:

items = [(1,12),(1,float('Nan')),(1,14),(1,10),(2,22),(2,20),(2,float('Nan')),(3,300), (3,float('Nan'))] sc = spark.sparkContext rdd = sc.parallelize(items) df = rdd.toDF(["id", "col1"]) import pyspark.sql.functions as func means = df.groupby("id").agg(func.mean("col1")) # The error is thrown at this line df = df.withColumn("col1", func.when((df["col1"].isNull()), means.where(func.col("id")==df["id"])).otherwise(func.col("col1"))) 
1

2 Answers

You can't reference a second spark DataFrame inside a function, unless you're using a join. IIUC, you can do the following to achieve your desired result.

Suppose that means is the following:

#means.show() #+---+---------+ #| id|avg(col1)| #+---+---------+ #| 1| 12.0| #| 3| 300.0| #| 2| 21.0| #+---+---------+ 

Join df and means on the id column, then apply your when condition

from pyspark.sql.functions import when df.join(means, on="id")\ .withColumn( "col1", when( (df["col1"].isNull()), means["avg(col1)"] ).otherwise(df["col1"]) )\ .select(*df.columns)\ .show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 12.0| #| 1| 14.0| #| 1| 10.0| #| 3|300.0| #| 3|300.0| #| 2| 21.0| #| 2| 22.0| #| 2| 20.0| #+---+-----+ 

But in this case, I'd actually recommend using a Window with pyspark.sql.functions.mean:

from pyspark.sql import Window from pyspark.sql.functions import col, mean df.withColumn( "col1", when( col("col1").isNull(), mean("col1").over(Window.partitionBy("id")) ).otherwise(col("col1")) ).show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 10.0| #| 1| 12.0| #| 1| 14.0| #| 3|300.0| #| 3|300.0| #| 2| 22.0| #| 2| 20.0| #| 2| 21.0| #+---+-----+ 

I think you are using Scala API, in which you use (). In PySpark, use [] instead.

0

ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJobHBrZmiDcoSOqbCsqJGnuG6wwK2Yn6qRorJuu8GjnJysXZ2utHnNqGSarKSntqPB055koJ2kYryjtsScq2ahlA%3D%3D