Add support for psycopg2 driver.

[jackwotherspoon]

Support PostgreSQL psycopg2 driver with connector.

[gunar]

Hey @jackwotherspoon, I've hit this exact problem this week. Do you have an estimate? Any rough ideas would be greatly appreciated. Thanks in advance.

[jackwotherspoon]

@gunar This has been prioritized and will be worked on in early 2022.

In the meantime, you can connect to Cloud SQL using Python and psycopg2 with the Cloud SQL Auth Proxy through Unix sockets or TCP. For an explanation and example see here: https://cloud.google.com/sql/docs/postgres/connect-admin-proxy#expandable-2

For the Unix sockets sample just swap out part of the pg8000 code for this code snippet:

pool = sqlalchemy.create_engine(
    sqlalchemy.engine.url.URL.create(
        drivername="postgresql+psycopg2",
        username=db_user,  # e.g. "my-database-user"
        password=db_pass,  # e.g. "my-database-password"
        database=db_name,  # e.g. "my-database-name"
        query={
            "host": "{}/{}".format(
                db_socket_dir,  # e.g. "/cloudsql"
                instance_connection_name)  # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
        }
    ),
    **db_config
)

Hope this helps for the time being! Thanks for the feedback!

[gunar]

@jackwotherspoon Thanks man. I'm not inclined to go with the Cloud SQL Auth Proxy as that's one more thing for the user to remember to execute.

I guess I could make the python script itself spawn the process for the proxy—has anyone attempted that?

Also, do I need sqlalchemy in order to connect to the proxy or can I connect directly with psycopg2?

[kemics]

Hi @jackwotherspoon , thank you for all your work on cloud sql connectors

Do you have any estimates for psycopg2 support? It would be really nice to have

By the way the issue has p2 priority but it has the biggest number of thumb-ups😉

[jackwotherspoon]

Hi @kemics, thanks for the kind words and showing your enthusiasm towards the Cloud SQL Python connector.

I currently can't provide a specific timeline for psycopg2 support because there are a lot of factors at play here but I can provide some details for you.

In order to support psycopg2 connections with the connector we require the ability to pass in a pre-existing SSL connection and skip a couple of postgres database level protocols (this is the hard part and is required because Cloud SQL connections connect to a proxy server and not directly to the database itself).

I have been in talks with the psycopg2 team about helping make some changes in psycopg2 to help support these type of connections, the work also requires additional upstream changes into libpq.

Overall, there are a lot of moving parts but I will continue to keep the ball rolling on this and make sure that I update progress here on the issue for more transparency.

Thanks again for the gentle nudge here and for showing interest in the Python connector, much appreciated.

Have a great day :)

[kemics]

Thanks @jackwotherspoon , great explanation 👍

I'm a fan of what you are doing here and have successfully tried this repo with sqlalchemy and also https://github.com/GoogleCloudPlatform/cloud-sql-go-connector . Works like a charm and a huge improvement in terms of security

[gunar]

@jackwotherspoon

With regards to:

For the Unix sockets sample just swap out part of the pg8000 code for this code snippet:

Is there any way to use my already logged in gcloud IAM authentication and not have to pass in username and password as part of the code? (I'm trying to get away from .env files).

[jackwotherspoon]

@gunar

Is there any way to use my already logged in gcloud IAM authentication and not have to pass in username and password as part of the code? (I'm trying to get away from .env files).

This can be done in a fairly straight-forward way for a service account authentication using Manual IAM Database Authentication. You just need to add your service account as an IAM database user to your Cloud SQL instance and grant it the Cloud SQL Instance User IAM role alongside the usual Cloud SQL Client role.

Once this is done, the below snippet should work in any environment that uses the service account (i.e. Cloud Run, Cloud Functions, App Engine etc.) to connect to Cloud SQL via Public IP (unix sockets is recommended for Public IP, see below comment for Private IP TCP connection snippet). See comments below for running locally.

import google.auth
from google.auth.transport.requests import Request
import sqlalchemy
import psycopg2

# uncomment below two lines to set application credentials to service account locally
# import os
# os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "path/to/key.json"

# get application default credentials
creds, _ = google.auth.default()

# refresh credentials if not valid
if not creds.valid:
  request = Request()
  creds.refresh(request)

# IAM service account database user, postgres removes suffix
sa_iam_user = (creds.service_account_email).removesuffix(
  ".gserviceaccount.com"
)

# create SQLAlchemy connection pool
pool = sqlalchemy.create_engine(
  sqlalchemy.engine.url.URL.create(
    drivername="postgresql+psycopg2",
    username=sa_iam_user, # login to database as service account user
    password=str(creds.token), # manual IAM database authn
    database=db_name,  # e.g. "my-database-name"
    query={
      "host": "{}/{}".format(
          "/cloudsql",  # e.g. "/cloudsql"
          instance_connection_name)  # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
    }
  ),
)

# ... use connection pool