HyperDB in Airtable

HyperDB is a powerful feature that allows Enterprise Scale customers to work with large datasets (up to 100M rows) in a new storage layer outside of a base. HyperDB tables can be populated from CSV files or connected directly to other cloud sources, making it easy to bring your important data into Airtable. Builders can access the data stored in HyperDB by using published data sets in the Data Library. New features like on-demand sync allow your organization to operationalize massive datasets across your complex workflows.

HyperDB prerequisites

Before using HyperDB, ensure you have:

• An Enterprise Scale plan

• Enterprise Admin or Integration Admin permissions

• Data in CSV format or access to Snowflake, Salesforce, or other supported integrations (if using external integrations)

• A primary key field that is unique per row in your dataset

• Data with no more than 100M rows

• HyperDB is not yet available for our European data residency customers

Creating a new HyperDB table

To create a HyperDB table:

1. Open your admin panel.

2. Click HyperDB.

3. Click Add table.

4. Enterprise Hub only - Select the org unit you want to use.

5. You’ll then choose whether to import data one time (via a CSV file upload) or set up a recurring import.

   1. CSV file imports are covered in this section.

   2. Instructions for setting up recurring integrated imports vary by import type.

      1. Databricks

      2. Salesforce

      3. Snowflake

Configuring CSV files for one time imports

1. After clicking the Local CSV file option, click Next.

2. Drop your CSV file or browse your computer and select your preferred file. Click Upload 1 file.

   1. Note that you are only able to upload a single file.

3. You will then be brought to a preview of the table that will be created from the CSV. You’ll need to map out the schema for your table.

   1. The system will attempt to detect appropriate field types, but you can modify these as needed by clicking the dropdown icon on each field that needs adjusting.

   2. You can also hop ahead to the “Review field mapping” step and review field mappings and our automatic suggestions.

   3. * Important: Once you choose field types, you cannot change them in HyperDB, so ensure the field types you’ve selected are the best fit for your organization’s needs.

4. Select a primary field (unique ID field).

5. Optionally, select a secondary field to help with searching.

   1. This should be human-readable and will enable users to perform friendly name searches when linking records via on-demand sync.

6. Review field mapping once more and then click Continue.

7. Next, enter a name and description for your HyperDB table. When your import contains one or more date fields, you’ll need to set a timezone for those date fields as well.

8. Click Save. Wait for your dataset to process. You can return to the specific HyperDB table’s page to check the status of your import. This may take some time depending on the size of your table.

Publishing data sets to your data library

To enable usage of HyperDB table data in bases and apps across your organization, you will need to publish a subset of the data. Filtering is an important step because many HyperDB tables will exceed the storage limits of individual bases. To create data sets from a HyperDB table:

1. If you aren’t already in admin panel, open your admin panel and click HyperDB.

2. Under the “Table” column, click the name of the table that you want to publish as a data set.

3. Scroll down to the “Published data sets” section and click Published data sets.

4. Click Create data set.

5. Choose which fields you want included in the data set by clicking the checkbox next to each field’s name. You can also click Select all and uncheck the box next to any fields you don’t want to include if that is an easier option. When you are finished, click Next.

6. (Optional, but encouraged) Next, you will define filters to scope your data set to a specific slice of the larger HyperDB table dataset.

   1. Click + Add condition to get started with filtering. More information about filtering in Airtable is available here.

   2. You can also review how many records are in the data set in the lower-left corner of the window. You can publish a data set of any size, but we will tell you if the data set is larger than what can be directly synced into a base.

   3. Once you’ve configured your preferred filters, click Next.

7. Next, you’ll set sync behavior. After selecting an option, click Next. There are two options to choose from:

   1. (Recommended) The default setting will is to Block syncs to other bases. If you choose not to allow downstream syncs, end users will not be able to “daisy-chain” syncs to other bases.

   2. You can also choose to Allow syncs to other bases.

8. On the next screen, there are several settings to configure:

   1. Enter a name and description for the data set.

   2. Choose the “Data set owner.” By default, this will be you, the person setting it up, but you can select another user if necessary.

   3. Define the audience. This can be specific user groups or org units (Enterprise Hub only), or you can make it available to everyone at your organization.

9. Once you are done configuring the data set, click Publish.

Viewing and managing published data sets

To view or manage published data sets:

1. Open your admin panel.

2. Click HyperDB.

   1. All published data sets from a HyperDB table are displayed here.

3. Click the name of a HyperDB table that you want to view or manage.

4. More details to see which bases are syncing your published data.

5. (Optional): Click the … icon at the end the data data you modify.

Using HyperDB in a base

After creating a HyperDB table and a published data set, collaborators can access it through the data library in their bases. To use a HyperDB table in a base:

1. Open your Airtable home screen.

2. Open the base where you want to use HyperDB data.  

3. Click + Add or import.

   1. Builders who are in the admin's defined audience can find published data sets in the data library.

4. Locate and select your HyperDB data located under the “Add from data library” section.

   1. You will likely need to click the xx more date sets > option. This will open a window where you can browse or search for a specific data set by name.

   2. After selecting your data, a preview window opens.

5. Click Add this data.

6. There are two primary ways to use a HyperDB dataset in a base, each with its own settings to adjust:

   1. Pull records in on-demand - All records from the published HyperDB dataset will be available to choose from a linked record field. The table you are syncing will only show records that are actively linked. For this option, you’ll need to:

      1. Select an existing table where a new linked record field will be created.

      2. Select a linked field that will connect records in the HyperDB dataset to another table. You can either allow Airtable to create a new linked record field by clicking the + Create new field option or select an existing field from the table you selected in the previous step. Note that this will convert the existing field to be a linked record field type if it isn’t already a linked record field type.

      3. When records are unlinked, you’ll toggle whether or not that will “Automatically delete records from synced table.” That is to say, if the records aren’t automatically deleted, then they will remian in the synced table.

   2. Filter and add as a smaller set - Alternatively, you can set up one or more record filters that will create a synced table that only contains the records matching your filter conditions. This a good option when you know that a certain subset of records are the only ones that will ever be relevant to the base you are working in.

      1. Select the Filter and add as a smaller set option.

      2. Add one or more conditions or condition groups. Learn more about filtering Airtable records in this support article. For larger datasets, be sure to pay attention to the number of records that will be added into the table based on the filters you have added. We will warn you when the dataset is over the record limits for a table/base. In those cases, you’ll need to add additional filters to pare down the number of records or consider using the “On-demand” option discussed above.

7. No matter the way you select to use the HyperDB dataset, you’ll click Create when you have finished configuring the settings to your preferences.

Adjusting existing HyperDB tables from admin panel

For all of the options listed below:

1. Open your admin panel.

2. Click HyperDB.

3. Search or browse for a table and then click the table’s name under the “Table” column.

Adding a field to an existing HyperDB table

1. Click the Edit fields dropdown button.

2. Click Add field.

3. Give the field a name and select a field type.

4. Click Save.

Removing a field from an existing HyperDB table

1. Click the Edit fields dropdown button.

2. Click Delete fields.

3. Click the checkbox next to the name of one or more fields.

4. Click Delete.

   1. When deleting fields you will see a warning of which bases are syncing this data into their bases.

   2. You might want to consider contacting the owner of those bases before making changes to the data set.

5. Click Delete again.

Updating the HyperDB table with a new CSV

1. Click the … button next to the current name of the dataset.

2. Click Update table from CSV.

3. Choose whether to Update existing table or Overwrite table.

4. Click Next.

5. Drag and drop the new CSV file and click Upload 1 file.

   1. When overwriting a table, the message “This step is irreversible” will appear, to remind you that the previous dataset will not be accessible any longer.

6. After the file has finished uploading, click Confirm.

   1. Any new fields present in the updated CSV will be added to the table.

   2. Any field re-ordering present in the updated CSV will be respected by the updated table.

Renaming a HyperDB table

1. Click the … button next to the current name of the dataset.

2. Click Rename table.

3. Adjust the name and click Save.

Deleting a HyperDB table

1. Click the … button next to the current name of the dataset.

2. Click Delete table.

3. A confirmation modal will appear with the message “This HyperDB table cannot be restored. All data sets associated will be deleted and downstream syncs will break.”

4. If you are sure you want to delete the HyperDB dataset, then type the name of the table in the appropriate box and then click Confirm.

HyperDB limitations and dependencies

• HyperDB tables require the Enterprise Scale plan

• Data must be under 100M rows

• Cannot change field types after table creation

• Single select fields are limited to 1,000 options

• Cannot remove single select options after creating a HyperDB table

• Cannot add or remove fields in tables that use native integrations like Salesforce or Snowflake

FAQs