Pandas UDF Failure #435

Activity

1. khageshbatra

   added

   bugSomething isn't workingSomething isn't working

   on May 4, 2023
2. sreedhar-guda commented on Aug 22, 2023

   

   sreedhar-guda

   on Aug 22, 2023

   Last edited by sreedhar-guda

   More actions

   Similar issue noticed with openai.Embedding.create as well. No such issues when running it like a regular function.

   Possible workaround:
   Set OpenAI related environment variables before openai import statement as shown below:

   import os
   os.environ["OPENAI_API_KEY"]=<api-key-value> 
   os.environ["OPENAI_API_TYPE"]="azure"
   os.environ["OPENAI_API_BASE"]="https://<your-resource-name>.openai.azure.com/"
   os.environ["OPENAI_API_VERSION"]="2023-05-15"
   
   import openai
   
   rest of the code....
   
3. vandustry commented on Oct 17, 2023

   

   vandustry

   on Oct 17, 2023

   More actions

   This may be loosely related (but if you don't necessarily need to use pandas, it may solve the problem):

   I had similar issues with pyspark.sql.functions.udf (running it as a regular function works, inside a udf not). Unfortunately, the work-around suggested by @sreedhar-guda did not work for me in this case. However, I think I figured out the issue/origin:

   udf uses workers with their own environment variables. That means, you have to broadcast "global" variables to their environment. You can do this using spark.sparkContext.broadcast. Here is what it would look like:

   # Define broadcast-variables.
   broadcast_api_key = spark.sparkContext.braodcast(os.environ.get("OPENAI_API_KEY"))
   broadcast_api_type = spark.sparkContext.braodcast("azure")
   broadcast_api_base = spark.sparkContext.braodcast("")
   broadcast_api_version = spark.sparkContext.braodcast("2023-03-15-preview")
   
   # Define the UDF function  
   def generate_text(input):  
       openai.api_key = broadcast_api_key.value
       openai.api_type = broadcast_api_type.value
       openai.api_base = broadcast_api_base.value
       openai.api_version = broadcast_api_version.value
   
       prompt = ""
       message = prompt + n  
             
       responseGPT = openai.ChatCompletion.create(  
           engine="GPT35Model",  
           messages=[{  
               "role": "user",  
               "content": message  
           }],  
           temperature=0.1,  
           max_tokens=800,  
           top_p=0.95,  
           frequency_penalty=0,  
           presence_penalty=0,  
           stop=None  
       )  
             
       return responseGPT["choices"][0]["message"]["content"]
         
   df = df.withColumn("results", udf(generate_text, StringType())(col(column_name)))
   
4. rattrayalex commented on Nov 10, 2023

   

   rattrayalex

   on Nov 10, 2023

   Contributor

   More actions

   We recently rewrote the library entirely; if the problem persists in v1, please open a new issue (ideally with a replit that can reproduce the problem).
5. rattrayalex

   closed this as completedon Nov 10, 2023
6. stainless-app

   added a commit that references this issue on Mar 27, 2025

   docs(api): improve docstrings (#435)

   02619e4
7. safa0

   added a commit that references this issue on Apr 27, 2025

   Start and finish streaming trace in impl metod (openai#540)

   ...

   This issue will close once commit 616d8e7 is merged into the 'main' branch.

   616d8e7