[BigQuery Data Agents](/about/bigquery-data-agents) November 3, 2026 # BigQuery Data Engineering Agent: pipelines in plain English Use the BigQuery Data Engineering Agent to turn your GA4 agent layer into a Dataform pipeline, set team rules in GEMINI.md, and fix failed runs. ![](/images/65d75b0ae8d69ef95785a5d6_david-pombar.webp) **[David Pombar](/authors-guides/david-pombar)** Swiss army knife at Trackingplan 7 min read · 1586 words In [guide 4](/guides/ga4-views-for-bigquery-agents) you built the tables your data agent reads, by hand, with a handful of `CREATE TABLE` statements. That works for a sample. On real data those tables need to be rebuilt every day, in the right order, with tests that fail when a key goes missing. That is a pipeline, and writing one is exactly the job of the Data Engineering Agent. This guide uses it to turn your agent layer into a Dataform pipeline, gives it your team’s rules, and shows how it helps when a run fails. The agent writes the code; you stay the engineer who reviews it. ![Black and white photo of the interior of a sugar beet factory around 1900, with rows of valves, pipes, and workers on the floor](/images/bigquery-data-engineering-agent-factory.jpg) A pipeline is plumbing: every valve in the right order, every day. Image: unknown author, Wikimedia Commons, public domain **Goal of this guide:** a Dataform pipeline, written with the agent, that rebuilds your `agent_layer` tables with tests. ** You need:** the tables and SQL from guide 4, and the project with billing from [guide 5](/guides/create-a-bigquery-data-agent). ## What the agent does, and what it doesn’t The [Data Engineering Agent](https://docs.cloud.google.com/bigquery/docs/data-engineering-agent-pipelines) builds pipelines from prompts and changes existing ones. It writes each step as Dataform SQLX, shows you a draft, and applies it when you accept. It can also transform raw tables into structured ones and troubleshoot failed runs. What it doesn’t do matters as much: **it cannot execute pipelines**. You review the code, run it, and schedule it. It also can’t follow URLs in a prompt, and it doesn’t take natural-language commands for notebooks or data preparation files. Sampling and profiling the data to write good SQL runs BigQuery jobs, which are billed like any other. ## Set it up Enable the BigQuery API, the Gemini Data Analytics API, and the Gemini for Google Cloud API, and make sure Gemini in BigQuery is turned on. Your user needs three roles: - Dataform Code Editor (`roles/dataform.codeEditor`) - BigQuery Job User (`roles/bigquery.jobUser`) - Gemini Data Analytics Data Agent Stateless User (`roles/geminidataanalytics.dataAgentStatelessUser`) Then, in BigQuery, click **Create new > Pipeline**, and choose ** Ask agent**. ## Step 1: Ask for the pipeline ![Diagram of the Data Engineering Agent workflow: you write a prompt, the agent proposes a draft pipeline of SQLX nodes, you review each node, apply, preview the data, then run or schedule the pipeline yourself.](/images/bigquery-data-engineering-agent-flow.webp) The agent drafts; you review, apply, run, and schedule. It never runs anything on its own. Describe the pipeline as you would to a colleague: the source, the outputs, and the rules. A prompt for the agent layer: ```text Create a pipeline that reads the GA4 export in bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_* and builds four tables in the dataset agent_layer: 1. sessions: one row per session. session_key is user_pseudo_id plus the ga_session_id event parameter. Include session_date, device_category, country, landing_page (first page_location), source and medium (first values of the source and medium event parameters), page_views, and converted (true if the session had a purchase). 2. purchases: one row per purchase event, with transaction_id, session_key, purchase_date, revenue (ecommerce.purchase_revenue) and items. 3. purchase_items: one row per item in a purchase, with transaction_id, purchase_date, item_name, item_category, quantity and item_revenue. 4. page_views: one row per page_view event, with session_key, viewed_at, page_location and page_title. Partition every table by its date column. Add an assertion that session_key is unique in sessions and that transaction_id is not null in purchases. ``` The more of your guide 4 decisions you put in the prompt, the less the agent decides for you. It is the same lesson as [guide 6](/guides/check-bigquery-agent-answers), for code instead of answers. ## Step 2: Review the draft The agent proposes a graph of nodes, one per table or step. Click each node and read its SQLX in the **Configuration** tab, with the same eye as an agent’s query: - Is the session key built from both IDs? - Does `source` come from the event parameters, not `traffic_source`? - Is revenue in `purchases` only, never summed after unnesting items? - Are the assertions there, and do they test what you asked? If something is wrong, say so in the next prompt (“in sessions, take medium from the event parameters, not from traffic\_source”) rather than editing by hand. The agent keeps the context of the conversation, and the correction stays in the history for whoever reads it later. When the draft is right, click **Apply**. Then use the data preview or ** Run task** on a node to check its output before running the whole pipeline. ## Step 3: Give it your team’s rules Every team has conventions: table prefixes, required columns, which datasets a pipeline may write to, which tests are mandatory. Write them once in an instructions file, `GEMINI.md`, and the agent applies them to every prompt. In the agent, open **Pipeline instructions > Create instructions file**: ```markdown # Pipeline rules - Write only to datasets that start with agent_ or staging_. - Every table is partitioned by its date column and has a description. - Every column has a description written for a data agent: what it means and when to use it rather than a similar column. - Every table with a key has a uniqueness assertion on it. - Never read traffic_source for session-level attribution. - Dates are in the GA4 property time zone; say so in the column description. ``` To share the same rules across pipelines, keep the instructions in a central Dataform repository and link it from each working repository, with **Use instructions from external repository**. Google’s [overview](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview) describes the setup. The column description rule is the one that ties this guide to the rest of the series: a pipeline that writes descriptions for every column keeps your data agent’s context in step with the tables, without anyone remembering to update it. ## Step 4: Run, schedule, and work in Dataform Run the pipeline once by hand, check the four tables against the ones you built in guide 4, and then schedule it, for example daily after the GA4 export lands. The pipeline lives in a Dataform repository, so it has version history, and you can open it in a development workspace. The agent works there too: in a Dataform development workspace, **Ask Agent** opens the same assistant, and it applies its changes directly to the workspace files. That is the place for changes to a pipeline that is already in production, reviewed like any other code change. ## Step 5: When a run fails A pipeline that runs every day will fail some day: a column renamed upstream, an assertion that catches duplicates, a quota. Open the failed run in the **Executions** tab and click ** Investigate**. The agent analyzes the root cause and recommends what to change, and you can then ask it to fix the code. Read its diagnosis as you read any agent’s claim. An assertion that fails on duplicate `transaction_id` values may be a bug in the SQL, or it may be the tracking sending the purchase event twice. The agent sees the pipeline and the tables; it can’t see the website. If the duplicates are in the raw export, the fix is upstream, in the tracking, not in the pipeline. ## Example prompts to try - “Add a table of daily revenue by medium, built from purchases and sessions, and an assertion that revenue is never negative.” - “Add a new\_vs\_returning column to sessions, using the ga\_session\_number event parameter.” - “Change every table to rebuild only the last three days on each run instead of the whole history.” - “Explain what this pipeline does, node by node, for someone new to the team.” Google’s [pipelines page](https://docs.cloud.google.com/bigquery/docs/data-engineering-agent-pipelines) has more, including prompts that use BigQuery AI functions in the generated SQL. ## Exercise Build the four-table pipeline with the prompt above, review every node against your guide 4 SQL, and write down each difference. Then add a `GEMINI.md` with three of your own rules and ask for one more table. Check that the new table follows the rules without you repeating them. ## How to know you got it right - Your pipeline rebuilds the four tables, partitioned, with assertions. - You reviewed every node’s SQLX before applying it. - Your `GEMINI.md` rules show up in tables you didn’t mention them for. - You know what Investigate does on a failed run, and when the fix belongs upstream. ## Resources - [Use the Data Engineering Agent to build and modify data pipelines](https://docs.cloud.google.com/bigquery/docs/data-engineering-agent-pipelines) and the [Data Engineering Agent overview](https://docs.cloud.google.com/gemini/data-agents/data-engineering-agent/agent-overview), Google Cloud docs. - [Dataform overview](https://docs.cloud.google.com/dataform/docs/overview), Google Cloud docs: repositories, workspaces, assertions, and scheduling. - [Mastering GA4 With Google BigQuery](https://www.teamsimmer.com/all-courses/mastering-ga4-with-google-bigquery/), Simmer: its automation and integrations course covers Dataform pipelines on GA4 data. Paid. ## FAQ **What is the BigQuery Data Engineering Agent?** An agent in BigQuery that builds, changes, and troubleshoots data pipelines from prompts in plain language. It writes Dataform SQLX for each step, shows you a draft to review, and applies it when you accept. It doesn’t run or schedule the pipeline itself; you do. **How do I open the Data Engineering Agent?** In BigQuery, click Create new, then Pipeline, then Ask agent, and type your prompt. You can also use it from a development workspace in a Dataform repository with Ask Agent, where it edits the workspace files directly. **What is GEMINI.md in the Data Engineering Agent?** An instructions file the agent reads on every prompt, for rules such as naming conventions, required tests, or which datasets to write to. You create it from Pipeline instructions, and you can keep one in a central Dataform repository so every pipeline follows the same rules. **Can the Data Engineering Agent fix a failed pipeline?** Yes. Open the failed run in the Executions tab and click Investigate. The agent analyzes the root cause and recommends a fix, and you can then ask it to change the code. You review and rerun the pipeline yourself. **Next:** [The Data Science Agent: notebooks that write themselves](/guides/bigquery-data-science-agent). Part of[BigQuery data agents: a guide series from zero to expert](/guides/bigquery-data-agents) On this page 1. [*01*What the agent does, and what it doesn’t](#what-the-agent-does-and-what-it-doesn-t) 2. [*02*Set it up](#set-it-up) 3. [*03*Step 1: Ask for the pipeline](#step-1-ask-for-the-pipeline) 4. [*04*Step 2: Review the draft](#step-2-review-the-draft) 5. [*05*Step 3: Give it your team’s rules](#step-3-give-it-your-team-s-rules) 6. [*06*Step 4: Run, schedule, and work in Dataform](#step-4-run-schedule-and-work-in-dataform) 7. [*07*Step 5: When a run fails](#step-5-when-a-run-fails) 8. [*08*Example prompts to try](#example-prompts-to-try) 9. [*09*Exercise](#exercise) 10. [*10*How to know you got it right](#how-to-know-you-got-it-right) 11. [*11*Resources](#resources) 12. [*12*FAQ](#faq) **Stop babysitting tags** Trackingplan audits your real user data around the clock and flags issues before they cost you. [Start free trial →](https://app.trackingplan.com/signup) ![](/images/65d75b0ae8d69ef95785a5d6_david-pombar.webp) **David Pombar** Swiss army knife at Trackingplan Read more from David, a Senior Product Strategist with 18+ years in digital product development and an atypical error detection knack. Share Copy link [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.trackingplan.com%2Fguides%2Fbigquery-data-engineering-agent) [X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.trackingplan.com%2Fguides%2Fbigquery-data-engineering-agent&text=BigQuery%20Data%20Engineering%20Agent%3A%20pipelines%20in%20plain%20English) Read next [All Guides →](/guides) - [ 6 minBigQuery Data AgentsNovember 3, 2026 ### BigQuery Data Science Agent: analysis in a notebook Use the Data Science Agent in BigQuery Studio notebooks to explore GA4 data, forecast revenue, and segment sessions, with a plan you approve step by step. by **David Pombar**](/guides/bigquery-data-science-agent) - [ 6 minBigQuery Data AgentsNovember 3, 2026 ### Test a BigQuery data agent before you share it Build a set of questions with known answers, rerun it after every change to your data agent, and use conversation logs to find the questions you missed. by **David Pombar**](/guides/evaluate-a-bigquery-data-agent) - [ 8 minBigQuery Data AgentsNovember 3, 2026 ### GA4 views for BigQuery agents: build the readable layer Turn the raw GA4 export into narrow tables a data agent can read: sessions, purchases, items, and pages, with descriptions written for the agent. by **David Pombar**](/guides/ga4-views-for-bigquery-agents) Trackingplan ## See everything. Miss nothing. Your implementations audited around the clock with real-time, real-user data. Real-time alerts about errors or changes in your data, campaigns, pixels, privacy and consent. Let AI flag issues before they cost you. [Start free trial →](https://app.trackingplan.com/signup) [Book a demo](/demo)