Skip to main content

Apache Iceberg

Keep your telemetry in your own object storage and let Huntbase query it where it lies. In most cases all we need is a bucket and a scoped access key — we create and operate the Apache Iceberg catalog for you, in your bucket. If you already run an Iceberg REST catalog or use AWS Glue, point us at that instead.

info

Apache Iceberg has its own guide: Data lake. This page summarises the connection settings.

CategoryData warehouse
DirectionYour lake — Huntbase reads it in place and never writes events to it
Sign-inStatic Access Keys
Query languagesKQL (Huntbase Lake), Search (Huntbase Lake)
Websitehttps://huntbase.io

Before you start​

Huntbase signs in to Apache Iceberg with Static Access Keys. Create the credential in Apache Iceberg first, then keep it to hand for the Connect step.

  1. Pick a bucket and prefix for your telemetry, such as s3://your-bucket/huntbase/.
  2. On AWS, attach an IAM policy to a dedicated IAM user that allows s3:ListBucket on the bucket and read and write object actions such as s3:GetObject and s3:PutObject on arn:aws:s3:::your-bucket/huntbase/* only.
  3. In the IAM console, choose Security credentials for that user, then choose Create access key in Access keys.
  4. On Retrieve access keys, choose Show or Download .csv file. This is your only chance to save the secret access key.
  5. Note the object-store endpoint and Region.
Watch out for
  • Read access is used for queries, and write access only for the Iceberg table metadata that the catalog maintains. Huntbase never writes your event data and never reads outside the prefix you scope the key to.
  • Creating the connection doesn't provision the store. It's provisioned after the connection is created, and only then does it join federated queries.
  • Your lake is read-only to Huntbase. Ingest keys can't send to it — every key sends to the Huntbase data lake.
  • The rest catalog type connects only to a REST catalog that needs no authentication. Apache Polaris, Databricks Unity Catalog, AWS S3 Tables and Snowflake Open Catalog can't be connected yet.

For the vendor's own instructions, see AWS example policy for S3 bucket object access.

tip

Use a dedicated, read-only credential for Huntbase where the product allows it. Huntbase only needs to read.

Connect Apache Iceberg​

  1. Go to Connections and click New connection, or click New connection on the Apache Iceberg product page.
  2. On Product, pick Apache Iceberg and choose the Owner.
  3. On Details, give the connection a Name and, optionally, a Description.
  4. On Connect, fill in the settings and credentials described below.
  5. On Verify, review the summary and click Create & check.

For everything else on the setup page — saving a draft, I'll do this later, and what each check result means — see Connections.

Settings​

Where your Apache Iceberg lives:

FieldRequiredNotes
How your tables are cataloguedYesmanaged needs nothing you have to install. glue uses the Data Catalog you already have on AWS. rest points at a catalog service you operate. One of managed, glue, rest. Defaults to managed.
Object Store EndpointYesHTTP(S) endpoint of your object store. Pick the AWS regional endpoint, or type your own for MinIO, Ceph, Cloudflare R2 or any other S3-compatible store. One of https://s3.us-east-1.amazonaws.com, https://s3.us-east-2.amazonaws.com, https://s3.us-west-2.amazonaws.com, https://s3.eu-west-1.amazonaws.com, https://s3.eu-west-2.amazonaws.com, https://s3.eu-central-1.amazonaws.com, https://s3.ap-southeast-2.amazonaws.com.
RegionYesFor MinIO and similar stores, use the region string your store expects. This is usually us-east-1.
Bucket / warehouse locationYesThe bucket and prefix, such as s3://your-bucket/huntbase/. For the managed catalog, this is the only location you need to provide. Shown only when How your tables are catalogued is managed or glue.
Iceberg REST Catalog URIYesHTTP(S) endpoint of the Iceberg REST catalog you run, such as https://catalog.example.eu:8181. The catalog must not require authentication. Catalogs that do — Apache Polaris, Databricks Unity Catalog, AWS S3 Tables and Snowflake Open Catalog — can't be connected yet. Shown only when How your tables are catalogued is rest.

Advanced settings​

These settings are under Advanced. You can usually leave them alone.

FieldRequiredNotes
Glue Catalog IDNoOnly when the Glue catalog lives in a different AWS account from the access key. Shown only when How your tables are catalogued is glue.

Credentials​

The only Method is Static Access Keys. Enter a Credential label (for example Production), then fill in:

FieldRequiredNotes
Access Key IDYesS3-compatible access key id. Secret — not shown again after you save it.
Secret Access KeyYesS3-compatible secret access key. Secret — not shown again after you save it.

Query it​

Apache Iceberg isn't queried through a connection picker. Once the store is provisioned and In federation, Huntbase reads it in place, alongside the Huntbase data lake, with the connectionless lake languages:

LanguageUse it for
KQL (Huntbase Lake)Kusto over the Huntbase data lake and every lake of yours in federation. A KQL (Huntbase Lake) query you run from Query reads your lakes too. One that Auto runs when you press Enter reads the Huntbase data lake only.
Search (Huntbase Lake)Searching events by value, then filtering by clicking. Enter searches the Huntbase data lake only. To include your lakes, press ⌘/Ctrl+Enter, or pick them in the stores pill, which then reads Lake + N.

Your raw events stay in your bucket. Every result row carries a store column naming the store it came from, and a store a query skipped is reported above the results. Reads of your lake are estimated before they run and count against your organization's daily scan budget. See Data lake.

Next steps​