python: How adbc_driver_manager.AdbcConnection gets outgoing call headers

I use Python adbc_driver_flightsql to connect to Flight SQL Server implemented in Java, my Arrow Flight SQL Server implements Auth2 CallHeaderAuthenticator, and the AuthResult returned by authenticate contains the Bearer Token.

db = adbc_driver_flightsql.connect(uri="grpc://xxx:xx", db_kwargs={
            adbc_driver_manager.DatabaseOptions.USERNAME.value: "root",
            adbc_driver_manager.DatabaseOptions.PASSWORD.value: "",
        })
conn = adbc_driver_manager.AdbcConnection(db)

It looks like the Bearer Token is in the outgoing call headers and I want to get the Bearer Token from Python adbc_driver_manager.AdbcConnection and use it afterwards, like this:

db = adbc_driver_flightsql.connect(uri="grpc://xxx:xxx", db_kwargs={
            adbc_driver_flightsql.DatabaseOptions.AUTHORIZATION_HEADER.value: "Bearer <token>",
        })
conn = adbc_driver_manager.AdbcConnection(db)

I need help, thanks~

Hmm, what are you trying to accomplish? We already send the bearer token back to the server in future requests - you don't need to explicitly set it.

All I need is send the bearer token back to the server in future requests, is this supported on ADBC 0.6?
When I tested on python ADBC 0.6 the bearer token did not return to the server

AuthResult is returned like this:

private AuthResult createAuthResultWithBearerToken(String token) {
        return new AuthResult() {
            @Override
            public void appendToOutgoingHeaders(CallHeaders outgoingHeaders) {
                outgoingHeaders.insert(Auth2Constants.AUTHORIZATION_HEADER,
                        Auth2Constants.BEARER_PREFIX + token);
            }

            @Override
            public String getPeerIdentity() {
                return token;
            }
        };
    }

It is supported (we integration test against Dremio which does this)

If you try a nightly build (https://gemfury.com/arrow-adbc-nightlies) you can set ADBC_DRIVER_FLIGHTSQL_LOG_LEVEL=debug to confirm what headers are being sent, do you see it there?

In fact I'm referring to Dremio's BearerTokenAuthenticator.

thanks! , I will try a nightly build

cool! , indeed send the bearer token back to the server.

~~I observed:~~
~~When creating adbc_driver_manager.AdbcConnection, the bearer token returned by authentication will only be used once (used in the first RPC getFlightInfoSqlInfo).~~

~~After that, the first statement request will generate a new bearer token again, and subsequent statement requests will send the bearer token back to the server.~~

~~Is this expected? This has misled me before.~~

Sorry, can you elaborate? Does any request get sent without a bearer token?

Sorry, the above description is not accurate, correction:

adbc_driver_manager.AdbcConnection will trigger authenticate 4 times, two of the handshakes will use BASIC_PREFIX, Why do we need two handshakes?

1. 
methodName: `arrow.flight.protocol.FlightService/Handshake`
incomingHeaders: `authorization=Basic cm9vdDo`
2.
methodName: `arrow.flight.protocol.FlightService/GetFlightInfo`
incomingHeaders: `authorization=Bearer test_token)`
3. 
methodName: `arrow.flight.protocol.FlightService/Handshake`
incomingHeaders: `authorization=Basic cm9vdDo`
4.
methodName: `arrow.flight.protocol.FlightService/DoGet`
incomingHeaders: `authorization=Bearer test_token)`

stmt.execute_query() will trigger the fifth authenticate, correct use of bearer tokens:

5.
methodName: `arrow.flight.protocol.FlightService/GetFlightInfo`
incomingHeaders: `authorization=Bearer test_token)`

import pyarrow

import adbc_driver_flightsql.dbapi as flight_sql
import adbc_driver_flightsql
import adbc_driver_manager

db = adbc_driver_flightsql.connect(uri="grpc://xxx:10478", db_kwargs={
            adbc_driver_manager.DatabaseOptions.USERNAME.value: "root",
            adbc_driver_manager.DatabaseOptions.PASSWORD.value: "",
        })

conn = adbc_driver_manager.AdbcConnection(db)

stmt = adbc_driver_manager.AdbcStatement(conn)

stmt.set_sql_query("select * from tpch.hdf5 limit 1000000;")
stream, rows = stmt.execute_query()

@zeroshade

CC @zeroshade

Hmm, that's definitely odd that it's triggering that many authentications, generally that would mean it's creating multiple connections (we do the auth whenever we start a connection), which might make sense if it was sending back multiple FlightEndpoints (since we attempt to download them in parallel) but I'm pretty sure that dremio doesn't do that (unless they've changed it). So i'll take a look and see if i can figure out why it's initializing so many handshakes and connections and get back to you

I've kicked off the release already, but it would be good to log connection events as well (I'll file a ticket) to make this clearer

#1087

Thanks, I will add more information later.

Solved, impl auth2 in Arrow 13.0.0
Close this issue.

Navigation Menu

Repository navigation

python: How `adbc_driver_manager.AdbcConnection` gets outgoing call headers #1079

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

python: How adbc_driver_manager.AdbcConnection gets outgoing call headers #1079

Description

Activity

lidavidm commented on Sep 19, 2023

xinyiZzz commented on Sep 19, 2023

xinyiZzz commented on Sep 19, 2023

lidavidm commented on Sep 19, 2023

xinyiZzz commented on Sep 19, 2023

xinyiZzz commented on Sep 19, 2023

lidavidm commented on Sep 19, 2023

xinyiZzz commented on Sep 19, 2023

lidavidm commented on Sep 19, 2023

zeroshade commented on Sep 20, 2023

lidavidm commented on Sep 20, 2023

lidavidm commented on Sep 20, 2023

xinyiZzz commented on Sep 21, 2023

xinyiZzz commented on Oct 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

python: How `adbc_driver_manager.AdbcConnection` gets outgoing call headers #1079