Tutorial: Databricks Free Edition with Palmer Penguins
Palmer field guide · Data engineering and AI · 16 November 2026
The field team recorded 344 penguin observations over three seasons. How many have a weight? Follow the records from the original CSV to a dashboard and a chat answer you can check. When you outgrow the free tier, the production-scale version of this discipline is my book MLOps with Databricks on Azure End-to-End.

Contents
Start with the field brief for hands-on work or the platform overview for a reading-first route. Choose Manual or Genie for each lab.
| Lab | Field exercise | Manual | Genie | Reading |
|---|---|---|---|---|
| 01 | Create your work area | Manual | Genie | Field notes |
| 02 | Split the survey into two deliveries | Manual | Genie | Field notes |
| 03 | Count the records and weights | Manual | Genie | Field notes |
| 04 | Load the first file into Bronze | Manual | Genie | Field notes |
| 05 | Keep each observation, then count by group | Manual | Genie | Field notes |
| 06 | Let the late file start the job | Manual | Genie | Field notes |
| 07 | Find out why the job failed | Manual | Genie | Field notes |
| 08 | Rebuild the summary on canvas | Manual | Genie | Field notes |
| 09 | Match the two names for each species | Manual | Genie | Field notes |
| 10 | Inspect access and lineage | Manual | Genie | Field notes |
| 11 | Define “weighed” once | Manual | Genie | Field notes |
| 12 | Publish the station brief | Manual | Genie | Field notes |
| 13 | Teach the analyst’s agent | Manual | Genie | Field notes |
| 14 | Test a weight prediction | Manual | Genie | Field notes |
| 15 | Find an observation by similarity | Manual | Genie | Field notes |
| 16 | Use a tested lookup in chat | Manual | Genie | Field notes |
| 17 | Catch a fluent wrong answer | Manual | Genie | Field notes |
| 18 | Record the human decision | Manual | Genie | Field notes |
| 19 | Serve the station brief | Manual | Genie | Field notes |
| 20 | Answer the station lead | Manual | Genie | Field notes |
Briefing and reference
- Free Edition overview
- Summit updates
- Field brief
- Acceptance checks
- Sources and credits
- Offline archive step 1
- Offline archive step 2
- Genie offline archive
- Diagnosing a Genie answer
- Choosing an agent
- Platform overview
Answer key
Answers for labs 01 to 05 · Answers for labs 06 to 10 · Answers for labs 11 to 15 · Answers for labs 16 to 20
Free Edition overview
What Free Edition includes
You can run this tutorial in a browser with a free Databricks account. Choose Free Edition when you sign up. It replaced Community Edition in 2025 and is intended for personal, non-commercial work.
Account and compute
Choose Free Edition when signing up. No credit card or cloud contract is required. Your account has one workspace and one metastore, serverless compute and managed storage. Python and SQL are supported. The single SQL warehouse is capped at 2X-Small.
Quotas and recovery
The published limits include five concurrent job tasks per account and one active pipeline per pipeline type. If you exhaust fair use, compute can pause for the remainder of the day or, in extreme cases, the month. Your data and settings persist through the pause.
AI, apps and databases
You can experiment with selected foundation models, model serving and tool calling. AI Search allows one endpoint with one search unit. Direct Vector Access, Knowledge Assistant, GPU serving and provisioned throughput are excluded. You can run up to three Databricks Apps. An app automatically stops 24 hours after its latest start, update or redeployment. Lakebase allows one scale-to-zero project.
Administration
Free Edition omits R and Scala, custom compute configurations, the account console, SSO/SCIM, private networking and compliance controls. Outbound networking is restricted. The service has no SLA or official support. Commercial use is prohibited.
Genie terminology
Use Genie Code when you want help building a table or fixing a notebook. Ask a Genie Agent when you want an answer from data it can read. Genie One is a chat interface that can find an agent for your question.
The more you know
Eligible users can verify their identity for limited AI Runtime access. This guide uses ordinary serverless compute and its workspace volume.
Sources · Free Edition overview · Edition limits · Free Edition vs. trial · Genie Code · Genie execution boundaries
Summit updates
What changed at the summit
The summit covered everything from connecting a sales system to streaming events as they happen. With the penguin data, you can try the same basic sequence on a smaller scale: bring in a file, turn it into tables and start a job when the next file arrives.
01 Lakeflow Connect
Databricks announced 100+ built-in connectors across its ecosystem. Managed SaaS and database connectors have source-specific credentials and prerequisites. Lab 04 uses the standard file-connector route. Zerobus Ingest also supports direct event writes to Delta, with writes under five seconds reported by Databricks. The lab measures file arrivals separately.
02 Spark Declarative Pipelines
Real-Time Mode targets latency as low as five milliseconds in suitable continuous flows. In labs 04–06 the pipeline reads each new file once. When the second file arrives, you can watch the island totals change.
03 Lakeflow Jobs
Databricks reported 50+ integrations. In lab 06, file arrival starts a job with a pipeline task and a dependent notebook. The job starts when the file arrives. You can see which file and run to check if the totals look wrong.
04 Designer, Genie Code and ZeroOps
Designer reached general availability in June, and Databricks explicitly confirms Free Edition access. Genie Code can edit its canvas. ZeroOps can investigate failures and propose fixes but requires an enabled workspace preview. Lab 07 uses task output, Genie Code and a repair run.
05 Unity Catalog and Genie Ontology
Unity Catalog shows who can read a table and where its data came from. Genie Ontology reached general availability in November 2026. Genie combines definitions you publish with context it learns from assets and usage. Lab 09 supplies species names. Lab 11 supplies the rule for “weighed”. Lab 13 tests whether the agent actually uses that rule.
Sources · Summit announcement · Lakeflow file connectors · ZeroOps requirements · Real-Time Mode · Designer availability · November release notes · Genie Ontology
Field brief
The Palmer field brief
The field team has finished three seasons of penguin surveys. The station lead asks: “How many observations did we make on each island, and how many have a recorded weight?” You’ll follow one CSV all the way to a dashboard and a chat answer the lead can check.
The archive
Palmer Station Antarctica LTER and Kristen Gorman and collaborators collected the field data. The palmerpenguins project distributes the simplified CC0 file. It contains 344 records from three species, three islands and three survey years. The second source is a three-name extract from the ITIS taxonomy record in lab 09.
Why make three tables?
Bronze keeps the entries as the field team sent them. Silver gives you one row per observation with measurements you can use in calculations. Gold answers a question people ask often, such as “How many records have a weight on each island?” You can always go back to the individual entries to check a total.
Where your files go
The examples use your writable catalog, workspace. The project’s tables live in workspace.palmer_lab and its files go in /Volumes/workspace/palmer_lab/landing.
What runs where
Each notebook holds code in cells. For Python, choose Serverless in the notebook’s compute menu. A Serverless SQL warehouse runs SQL cells only. If you open a Python cell and see a warehouse selected, switch the compute to Serverless. For a saved notebook run by a job, choose Serverless again in the job task. Files saved in a volume stay there after the notebook session ends.
How to work through each lab
For a reading-first pass, start with platform overview. Each lab links to its field notes. For hands-on work choose one continuous route. Track A starts in Manual lab 01 with the commands you type. Track B starts in Genie lab 01 and walks through all 20 labs using Genie. Each B lab includes its own setup, prompts, actions and results. Genie’s code can look different. Run it in the named editor and check the result.
Verification
The counts, missing weights, lookups and prediction errors were checked against the source data. Your messages confirm the volume, the Bronze repair and a successful file-triggered delivery with Gold at 344 / 342 / 2. For the rest, compare your own run with the checkpoint in each section.
The more you know
Already have bronze_obs? Check for 224 rows from first.csv in lab 04. Then continue with Silver and Gold in Manual lab 05 or Genie lab 05.
Sources · Publisher’s CSV · Dataset and original studies · CC0 licence · ITIS taxonomic record · Genie Ontology · Choose notebook compute
Platform overview
How the pieces fit together
Follow one penguin observation through the platform. The field worker records a weight, the station counts it, and a colleague asks about it in chat. Along the way, the storage, the calculation and the meaning of the result each have a different owner.
What is a lakehouse?
A lakehouse lets you keep files in cloud storage and work with them as reliable, queryable tables. The publisher’s CSV in landing is an ordinary file. A Delta table adds a transaction log to column-oriented data files. The log records which files belong to a committed version. Someone reading during a write sees a consistent table snapshot. This is one reason the team can query results while new observations are arriving.
Which service does which job?
| When the team wants to… | Where it happens in this survey |
|---|---|
| Keep the original or name a table | A Unity Catalog volume holds the source CSV. The catalog and schema give the tables names, ownership and permissions. |
| Turn deliveries into a report | Spark processes data in the pipeline. Bronze captures inputs, Silver cleans measurements and Gold groups them. Lakeflow Jobs decides when the pipeline and its check run. |
| Ask a question or publish an answer | A SQL warehouse runs analytical queries. A metric view defines “weighed”. A dashboard, a Genie Agent and an app can present the resulting numbers. |
| Record a human decision | Lakebase Postgres holds the current review state. Checking the case changes the review record while the original observation still lacks a weight. |
A fast way to read this guide
Each of the next 20 sections follows one lab. Read the opening question, then try “Check your understanding” from memory. The answer key explains the reasoning. The end of each section links to both practical routes. In either route, “Field notes” brings you back to the corresponding field notes.
Sources · Lakehouse fundamentals · Delta tables and transactions · Work with volume files · Unity Catalog privileges · Pipeline refresh behavior
📚 Where this goes next. These labs run on Free Edition quotas with one small CSV. The moment the same pipeline has to survive real volumes, schema drift, model registries and deployment gates, you are in MLOps territory. MLOps with Databricks on Azure End-to-End covers that ground with the same hands-on discipline, and its free code companion lives on this site.
Sources and credits
Sources and data credits
Each lab links to the Databricks instructions it uses. This section also tells you where the field records and the scientific names came from, so you can check either one against its source. The screens in this tutorial follow Databricks on AWS.
Free Edition and the summit
For account rules, use the Free Edition signup and limits pages. The summit announcement covers connectors and speed claims across the platform. The October Designer update confirms access on Free Edition. The ZeroOps guide explains which preview you would need.
Where Genie gets its answers
Genie can use definitions you publish and clues it learns from workspace activity. In lab 11 you supply a definition yourself: “weighed” counts records with a body mass. Unity Catalog controls which data a user can read.
Data sources
Palmer Station Antarctica LTER and Kristen B. Gorman collected the observations. Allison Horst, Alison Hill and Kristen Gorman prepared palmerpenguins (2020). The project repository links the original three Environmental Data Initiative study records. The simplified CSV is distributed under CC0. The genus record from the Integrated Taxonomic Information System supplies the scientific names in lab 09.
Reproducibility
Both ways of getting the CSV give you the same 344 rows and eight columns. The notebook checks the contents before it saves the file. If your workspace can’t reach the publisher, the appendix contains the file itself. All the work stays in Databricks.
Edition note
Rebuilt 15 November 2026 for catalog workspace. Source checks for the platform guide were performed on 14 November. Your transcript confirms the project volume. Your screenshot also shows Bronze rows after Genie repaired its Python file. For counts and later steps, use the checks in your own workspace.
Following up on an agent answer
Diagnosing a Genie answer explains the agent settings that change what a colleague sees, how a question is answered and how you test it.
Sources · Free Edition overview · Publisher’s CSV · ITIS taxonomic record · Editorial design reference
About the author
Mohammed Brückner is an enterprise architect in Munich, building production machine learning platforms on Azure and Databricks. More at mohammed-brueckner.com — and for talks or workshops on data platforms and enterprise AI, see Speaking & Workshops.
Books: Machine Learning Operations (MLOps) with Databricks on Azure End-to-End — the production-scale continuation of this tutorial — and Platform Economies (2026), on what happens when the API becomes the product.