bigquery_agent_analytics_plugin - Agent Engine (Not Working)

[dvdeveloper]

Hello, I am trying to implement bigquery_agent_analytics_plugin using agent engine. The SA has the permissions, but I cannot get the logs to be recorded with this new implementation. Has anyone been able to resolve this? Cloud Run works, but not agent engine. I am leaving some sample code in case anyone can help me.

agent.py

from google.adk.agents.llm_agent import Agent
from google.adk.models.google_llm import Gemini
from log.registro  import before_agent_callback, after_agent_callback, before_model_callback, after_model_callback , after_tool_callback
from log.langfuse_config import prompt
from google.genai import types
from google.adk.tools.retrieval import VertexAiRagRetrieval
from vertexai import rag
from utils.template_utils import escape_template_variables
from google.adk.tools.bigquery import BigQueryToolset, BigQueryCredentialsConfig


# --- BigQuery Agent Analytics (logs) ---

import os
import logging
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin,
    BigQueryLoggerConfig,
)

BQ_PROJECT_ID = "tXXXXXX"
BQ_DATASET_ID = "agent_engine"
BQ_TABLE_ID = "XXXXX"


bq_logging_plugin = BigQueryAgentAnalyticsPlugin(
    project_id=BQ_PROJECT_ID,
    dataset_id=BQ_DATASET_ID,
    table_id=BQ_TABLE_ID,
    location="us-central1",  # Especificar location explícitamente
)

"""
credentials, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
bigquery_toolset = BigQueryToolset(
    credentials_config=BigQueryCredentialsConfig(credentials=credentials)
)
 """

PROMPT = escape_template_variables(prompt.prompt)
LANGFUSE_AGENT_NAME = prompt.config['AGENT_NAME']
MODEL_NAME = prompt.config['MODEL']

PROJECT_ID = prompt.config['PROJECT_ID']
LOCATION = prompt.config['LOCATION']
CORPUS_ID = prompt.config['CORPUS_ID']
RAG_CORPUS_ID = f"projects/{PROJECT_ID}/locations/{LOCATION}/ragCorpora/{CORPUS_ID}"

rag_tool = VertexAiRagRetrieval(
    name="BaseConocimiento",
    description="Base de datos de conocimiento asociado Dexter Agente",
    rag_resources=[rag.RagResource(rag_corpus=RAG_CORPUS_ID)],
)

root_agent = Agent(
    name=LANGFUSE_AGENT_NAME,
    model=MODEL_NAME,
    instruction=PROMPT,
    tools=[rag_tool],
    before_agent_callback=before_agent_callback,
    after_agent_callback=after_agent_callback,
    before_model_callback=before_model_callback,
    after_model_callback=after_model_callback,
    after_tool_callback=after_tool_callback,
)

update.py

import vertexai
from vertexai import agent_engines
from vertexai.preview import reasoning_engines
from agente.agent import root_agent, bq_logging_plugin
import uuid

vertexai.init(
    project="XXXXXXXXX",
    location="us-central1",
    staging_bucket="gs://XXXXXXXXX"
)

# AdkApp acepta agent y plugins por separado
app_adk = reasoning_engines.AdkApp(
    agent=root_agent,
    plugins=[bq_logging_plugin],
    enable_tracing=True,
)

gcs_dir_name = str(uuid.uuid4())
remote_app = agent_engines.update(
    agent_engine=app_adk,
    resource_name="projects/XXXXX/locations/us-central1/reasoningEngines/X06XXX68445XXX70XXXX",
    service_account="sa-agents@XXXXXX",
    requirements=[
        "google-adk>=1.21.0",
        "google-cloud-aiplatform[adk,agent_engines]>=1.21.0",
        "llama-index",
        "langfuse",
        "google-cloud-secret-manager",
        "google-cloud-firestore",
        "google-cloud-bigquery",
        "google-cloud-bigquery-storage",
        "google-cloud-storage",
    ],
    extra_packages=["./agente","./log","./utils"],
    gcs_dir_name=f"tXXXXX{gcs_dir_name}",
    env_vars={
        "GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY": "true",
        "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT": "true",
    }
)

[haiyuan-eng-google]

Hi @dvdeveloper thanks for reporting this bug, can you do below steps to help figure out the root cause?

Check Cloud Logging: Go to the Google Cloud Console, navigate to Logs Explorer, and filter by resource.type="reasoning_engine". Look for any "Permission Denied" or "Field not found" errors related to BigQuery or the ADK internal processes.

Version Match: Ensure that the google-adk and google-cloud-aiplatform versions are consistent between your local development (where Cloud Run likely works) and the requirements you've listed in update.py. Also make sure google-adk version running on Agent Engine is >= 1.21.0

[Dustan-FullStory]

One other thing I want to mention for your investigation, and I'm not sure if this is the same for the original reporter or not, but I have to send logs to BigQuery in a separate project from where the agents are running.

I have all of the right permissions for the reasoning engine service account in that other project (other queries to BQ work fine from running agents) and I don't see any sort of permissions errors in the cloud logging console, but I wanted to mention that in case it was some weird edge case that might not have been caught in testing this.

[haiyuan-eng-google]

Hi @Dustan-FullStory

Thanks for the extra context! "Works locally but fails silently in Agent Engine" usually points to swallowed logs or missing optional dependencies in the remote container.

Since you're using cross-project logging (Project A -> Project B), here is the likely checklist to fix it:

1. Enable Remote Logging (Critical) In Agent Engine, the plugin's internal error logs (handled by logging.getLogger("google_adk")) often don't show up in Cloud Logging default views unless explicitly configured. Please add this to the top of your
   agent.py
   (before deploying) to force errors to visible stdout:

import logging
# Force ADK logs to info/debug level and ensure they print
logging.basicConfig(level=logging.INFO)
logging.getLogger("google_adk").setLevel(logging.DEBUG)

This will likely reveal the exact permission or import error.

2. Cross-Project Permissions (Specific to Write API) Even if standard queries work, the Storage Write API requires specific quotas.

Quota Project: The request might be failing because it's charging "Project B" (destination) for streaming ingestion, but the credentials verify "Project A" (source).
Fix: Ensure the Service Account running the agent has bigquery.tables.updateData on the destination table AND that Project A has the BigQuery API enabled.

3. Debug Configuration To rule out buffering issues (where the container freezes before flush): Update your plugin config to flush immediately:

bq_logger_plugin = BigQueryAgentAnalyticsPlugin(
    ...,
    config=BigQueryLoggerConfig(
        batch_size=1,
        batch_flush_interval=0.1
    )
)

Let us know if enabling the logging reveals a specific 403 or ImportError!

[Dustan-FullStory]

I've redeployed the agent with the suggestions and the behavior is pretty much the same (though I am seeing the agent interactions in the Cloud Logging console of course, so the debug logging is working).

I tried to wrap the plugin callbacks with some logging like this:

  # plugin callbacks with logging to track execution
  original_before_run = bq_plugin.before_run_callback
  async def logged_before_run(*args, **kwargs):
      logger.info("BigQuery Plugin: before_run_callback FIRED")
      result = await original_before_run(*args, **kwargs)
      logger.info("BigQuery Plugin: before_run_callback COMPLETED")
      return result
  bq_plugin.before_run_callback = logged_before_run

The plugin initializes successfully it seems, but never fires for some reason:

  2026-01-16T21:06:06.977030Z [INFO] Plugin config: BigQueryLoggerConfig(enabled=True, event_allowlist=None, event_denylist=None, max_content_length=512000, table_id='agent_events_v2', clustering_fields=['event_type', 'agent', 'user_id'], log_multi_modal_content=True, retry_config=RetryConfig(max_retries=3, initial_delay=1.0, multiplier=2.0, max_delay=10.0), batch_size=1, batch_flush_interval=0.1, shutdown_timeout=10.0, queue_max_size=10000, content_formatter=None, gcs_bucket_name='[STAGING BUCKET NAME]', connection_id=None)

  2026-01-16T21:06:06.976192Z [INFO] Creating BigQueryAgentAnalyticsPlugin instance...
  2026-01-16T21:06:06.976463Z [INFO] ✓ BigQuery Analytics plugin created successfully
  2026-01-16T21:06:06.976688Z [INFO] ✓ Target: [CORRECT TABLE NAME]
  2026-01-16T21:06:06.976933Z [INFO] Plugin type: <class 'google.adk.plugins.bigquery_agent_analytics_plugin.BigQueryAgentAnalyticsPlugin'>
  2026-01-16T21:06:06.977343Z [INFO] Plugin.table_id attribute: [CORRECT TABLE NAME]
  2026-01-16T21:06:06.977816Z [INFO] Plugin.project_id attribute: [CORRECT TARGET PROJECT]
  2026-01-16T21:06:06.977833Z [INFO] Plugin.dataset_id attribute: [CORRECT DATASET NAME]
  2026-01-16T21:06:06.978155Z [INFO] Plugin._started: False
  2026-01-16T21:06:06.978446Z [INFO] Wrapped BigQuery plugin callbacks with logging
  2026-01-16T21:06:06.978736Z [INFO] Plugins list length: 1
  2026-01-16T21:06:06.979440Z [INFO] Plugins list: [<google.adk.plugins.bigquery_agent_analytics_plugin.BigQueryAgentAnalyticsPlugin object at 0x7fd53c6c5fd0>]
  2026-01-16T21:06:06.979775Z [INFO] BigQuery Analytics plugin WILL BE ADDED to App
  2026-01-16T21:06:06.980395Z [INFO] App created successfully. App.plugins: [<google.adk.plugins.bigquery_agent_analytics_plugin.BigQueryAgentAnalyticsPlugin object at 0x7fd53c6c5fd0>]
  2026-01-16T21:06:06.980639Z [INFO] Number of plugins in App: 1

The BigQuery Plugin: before_run and BigQuery Plugin: after_run stuff never appears in the logs, even as agent interactions are taking place.

[haiyuan-eng-google]

It looks wired, @Dustan-FullStory is that possible you share me the actual agent code you deployed to agent engine? That would be very helpful for debugging purpose.

[Dustan-FullStory]

As it is the property of my employer I don't think I can easily do that without getting some approvals. I will however create a mock-agent with nothing sensitive in it and test it. If it still doesn't work I will share that.