Shortcuts in Microsoft OneLake allow you to unify your data across domains, clouds, and accounts by creating a single virtualized data lake for your entire enterprise. All Fabric experiences and analytics engines can connect directly to your existing data sources, such as Azure, AWS, and OneLake, through a common namespace. OneLake manages all permissions and credentials, so there's no need to configure each separate Fabric experience to connect to each data source. Additionally, you can use shortcuts to remove edge copies of data and reduce processing latency associated with data copies and staging.
Microsoft Fabric is currently in PREVIEW. This information relates to a preliminary product that may change significantly prior to release. Microsoft makes no warranties, express or implied, with respect to the information provided in this document.
What are shortcuts?
Shortcuts are objects in OneLake that point to other storage locations. The location can be internal or external to OneLake. The location that a shortcut points to is known as the "Target" path of the shortcut. The location where the shortcut appears is known as the "Shortcut Path". Shortcuts appear as folders in OneLake and can be transparently used by any experience or service accessing OneLake. Shortcuts behave similarly to symbolic links. They are an object independent of the target. If a shortcut is removed, the destination is not affected. If the destination path is moved, renamed or deleted, the shortcut may be broken.
Where can I create shortcuts?
Shortcuts can be created in Lakehouses and KQL databases. Additionally, shortcuts created in these items can point to other OneLake locations, ADLS Gen2, or Amazon S3 storage accounts.
When creating shortcuts in Lakehouse, it's important to understand the theme's folder structure. Lakehouses is made up of two top-level folders: the "Tables" folder and the "Files" folder. The "Tables" folder represents the managed part of the Lakehouse, while the "Files" folder is the unmanaged part of the Lakehouse. In the "Tables" folder, you can only create shortcuts at the top level. Shortcuts are not supported in other subfolders of the "Tables" folder. If the shortcut destination contains data in delta\parket format, Lakehouse will automatically sync the metadata and recognize the folder as a table. In the "Files" folder, there are no restrictions on where you can create shortcuts. They can be created at any level of the folder hierarchy. Table recognition does not occur in the "Files" folder.
When you create a shortcut in a KQL database, it appears in the "Shortcuts" folder in the database. The KQL database treats shortcuts as external tables. To query the shortcut, use the "external_table" function in the Kusto query language.
Where can I access the shortcuts?
Any fabric or non-fabric service that can access data in OneLake can use shortcuts. The shortcuts are transparent to any service that accesses the data through the OneLake API. The shortcuts just show up as another folder in the lake. This allows Spark, SQL, Real-Time Analytics, and Analysis Services to use shortcuts when querying data.
spark - spark
Spark notebooks and Spark jobs can use shortcuts created in OneLake. Relative file paths can be used to directly read data from shortcuts. Also, if a shortcut is created in the "Tables" section of Lakehouse and it is in delta format, it can also be read as a managed table using Spark SQL syntax.
df = chispa.read.format("delta").load("Tables/MyShortcut")display(df)
df = chispa.sql("SELECCIONE * DE MyLakehouse.MyShortcut LIMIT 1000")display(df)
The delta format does not support tables with spaces in the name. Any shortcut containing a space in the name will not be detected as a delta table in the lake house.
Shortcuts in the Lakehouse table section can also be read through the Lakehouse SQL endpoint. This can be accessed through the mode selector in Lakehouse or through SQL Server Management Studio (SSMS).
SELECT ABOVE (100) *OF [MyLakehouse].[dbo].[MyShortcut]
Shortcuts in KQL databases are recognized as external tables. To query the shortcut, use the "external_table" function in the Kusto query language.
external_table('My shortcut')| take 100
KQL databases currently do not support data in delta format. Tables in a KQL database are only exported to OneLake as parquet files. Shortcuts in KQL databases that contain delta-formatted data in the target are not recognized as tables.
Power BI datasets can be created for Lakehouses that contain shortcuts in the Lakehouse table section. When the dataset is running in direct maritime mode, Analysis Services can read the data directly from the direct access.
Applications and services outside of the fabric can also access shortcuts through the OneLake API. OneLake supports a subset of the ADLS Gen2 and blob storage APIs. For more information on the OneLake API, seeAccess to OneLake with API.
Types of shortcuts
OneLake shortcuts support multiple file system data sources. These include internal OneLake locations, Azure Data Lake Storage Gen2, and Amazon S3.
OneLake Internal Shortcuts
OneLake's internal shortcuts allow you to reference data in existing drug articles. These elements include lakehouses, KQL databases, and data warehouses. The shortcut can point to a folder location within the same theme, between themes in the same workspace, or even between themes in different workspaces. When creating a shortcut between elements, the element types do not have to match. For example, you can create a shortcut in Lakehouse that points to data in a data store.
When accessing data through a shortcut to another OneLake location, the caller's identity will be used to authorize access to the data on the shortcut's destination path*. This user must have permissions on the destination location to read the data.
When you access shortcuts through Power BI or T-SQL datasets,the identity of the calling user is not passed to the shortcut target.Instead, the identity of the owner of the calling element is passed, delegating access to the calling user.
You can also create shortcuts to ADLS Gen2 storage accounts. When creating ADLS shortcuts, the destination path can point to any folder in the hierarchical namespace. The destination path must contain at least one container name.
ADLS shortcuts must point to the DFS endpoint of the storage account.
ADLS shortcuts do not support private endpoints.
ADLS shortcuts use a delegated authorization model. In this model, the creator of the shortcut provides a credential for the ADLS shortcut, and all access to that shortcut will be authenticated with that credential. The supported delegate types are Account Key, SAS Token, OAuth, and Service Principal.
SAS sheet- must contain at least the following permissions: Read, List, Execute
OAuth identity- You must have the Storage Blob Data Reader, Storage Blob Data Contributor, or Storage Blob Data Owner role on the storage account.
Supervisor- You must have the Storage Blob Data Reader, Storage Blob Data Contributor, or Storage Blob Data Owner role on the storage account.
You can also create shortcuts to Amazon S3 accounts. When creating shortcuts to Amazon S3, the destination path must contain at least one bucket name. S3 doesn't natively support hierarchical namespaces, but you can use prefixes to mimic a directory structure. You can include prefixes in the shortcut path to further reduce the scope of data available through the shortcut. When accessing data via an S3 shortcut, the prefixes will be rendered as folders.
S3 shortcuts must point to the https endpoint of the S3 bucket. Example:
S3 Shortcuts do not support private endpoints.
S3 shortcuts use a delegated authorization model. In this model, the creator of the shortcut provides a credential for the S3 shortcut, and all access to that shortcut will be authenticated with that credential. The supported delegated credentials are a key and a secret for an IAM user.
The IAM must have at least read-only (Get, List) permissions on the bucket that the shortcut points to.
S3 shortcuts are read-only. They do not support write operations, regardless of the IAM user's permissions.
How Shortcuts Use Cloud Connections
ADLS and S3 passthrough authorization is delegated through the use of cloud connections. When a new ADLS or S3 shortcut is created, a user creates a new connection or selects an existing connection to the data source. When a connection is established to a shortcut, this is considered a "bind" operation. Only users with permission to connect can perform the bind operation. If a user does not have permissions for the connection, they cannot create new shortcuts using that connection.
Permissions for shortcuts are controlled by a combination of the permissions on the shortcut path and the destination path. When a user accesses a shortcut, the more restrictive permission of the two locations is applied. Therefore, a user who has read/write permissions to Lakehouse but only read permissions to the shortcut target will not be able to write to the shortcut target path. Similarly, a user who only has read permissions on Lakehouse, but reads/writes to the shortcut target, will also not be able to write to the shortcut target path.
roles in the workplace
The following table lists the permissions related to shortcuts for each role in the workspace. For more information, seeroles in the workplace.
|Create a shortcut||Y1||Y1||Y1||-|
|Read the contents of the shortcut file/folder||Y2||Y2||Y2||-|
|Write to the shortcut destination location||Y3||Y3||Y3||-|
|Read data from shortcuts in table section i Lakehouse via TDS endpoint||Y||Y||Y||Y|
1The user must have a role that grants write permission to the shortcut location and, at a minimum, read permission to the target location.
2The user must have a role that grants read permission to both the shortcut location and the target location.
3The user must have a role that grants write permission to both the shortcut location and the target location.
How do shortcuts handle deletions?
Shortcuts do not perform overlapping deletes. When you perform a delete operation on a shortcut, you only delete the shortcut object. The data in the shortcut destination remains unchanged. However, if you perform a delete operation on a file or folder in a shortcut and you have permissions on the shortcut destination to perform the delete operation, the files and/or folders on the destination will be deleted. The following example illustrates this point.
User A has a lake house with the following route:
my shortcutis a shortcut that points to an ADLS Gen2 account that containsFoo\Barmapper
Delete a shortcut object
User A performs a delete on the following path:
In this case,my shortcutis removed from Lakehouse. Shortcuts do not cascade deletes, therefore files and folders in the ADLS Gen2 accountFoo\Barremain unaffected.
Deleting content referenced by a shortcut
User A performs a delete on the following path:
In this case, if user A has write permissions on the ADLS Gen2 accountBardirectory is removed from the ADLS Gen2 account.
Lowering of the working area
When you create shortcuts between multiple quilt elements in a workspace, you can visualize the shortcut relationships through the line view of the workspace. choosefamily viewbutton () in the upper right corner of the Workspace Explorer.
Line display is limited to a single work area. Shortcuts to locations outside of the selected workspace are not displayed.
Known issues and limitations
- The maximum number of shortcuts per cloth is 10,000.
- The maximum number of shortcuts on a single OneLake route is 10.
- The maximum number of shortcuts for shortcut links is 5.
- ADLS and S3 shortcut destination paths cannot contain reserved characters from RCF 3986, section 2.2.
- OneLake destination shortcut paths must not contain "%" characters.
- Shortcuts do not support non-latin characters.
- Copy Blob api does not support ADLS or S3 shortcuts.
- The copy function does not work on shortcuts that point directly to ADLS containers. It is recommended that you create ADLS shortcuts to a directory that is at least one level below a container.
- OneLake shortcuts pointing to ADLS or S3 shortcuts are not supported.
- Additional shortcuts cannot be created in ADLS or S3 shortcuts.
- creating shortcuts