OneLake Shortcuts - Microsoft Fabric (2023)

Shortcuts in Microsoft OneLake allow you to unify your data across domains, clouds, and accounts by creating a single virtualized data lake for your entire enterprise. All Fabric experiences and analytics engines can connect directly to your existing data sources, such as Azure, AWS, and OneLake, through a common namespace. OneLake manages all permissions and credentials, so there's no need to configure each separate Fabric experience to connect to each data source. Additionally, you can use shortcuts to remove edge copies of data and reduce processing latency associated with data copies and staging.

What are shortcuts?

Shortcuts are objects in OneLake that point to other storage locations. The location can be internal or external to OneLake. The location that a shortcut points to is known as the "Target" path of the shortcut. The location where the shortcut appears is known as the "Shortcut Path". Shortcuts appear as folders in OneLake and can be transparently used by any experience or service accessing OneLake. Shortcuts behave similarly to symbolic links. They are an object independent of the target. If a shortcut is removed, the destination is not affected. If the destination path is moved, renamed or deleted, the shortcut may be broken.

Where can I create shortcuts?

Shortcuts can be created in Lakehouses and KQL databases. Additionally, shortcuts created in these items can point to other OneLake locations, ADLS Gen2, or Amazon S3 storage accounts.

Lake House

When creating shortcuts in Lakehouse, it's important to understand the theme's folder structure. Lakehouses is made up of two top-level folders: the "Tables" folder and the "Files" folder. The "Tables" folder represents the managed part of the Lakehouse, while the "Files" folder is the unmanaged part of the Lakehouse. In the "Tables" folder, you can only create shortcuts at the top level. Shortcuts are not supported in other subfolders of the "Tables" folder. If the shortcut destination contains data in delta\parket format, Lakehouse will automatically sync the metadata and recognize the folder as a table. In the "Files" folder, there are no restrictions on where you can create shortcuts. They can be created at any level of the folder hierarchy. Table recognition does not occur in the "Files" folder.

KQL database

When you create a shortcut in a KQL database, it appears in the "Shortcuts" folder in the database. The KQL database treats shortcuts as external tables. To query the shortcut, use the "external_table" function in the Kusto query language.

Where can I access the shortcuts?

Any fabric or non-fabric service that can access data in OneLake can use shortcuts. The shortcuts are transparent to any service that accesses the data through the OneLake API. The shortcuts just show up as another folder in the lake. This allows Spark, SQL, Real-Time Analytics, and Analysis Services to use shortcuts when querying data.

spark - spark

Spark notebooks and Spark jobs can use shortcuts created in OneLake. Relative file paths can be used to directly read data from shortcuts. Also, if a shortcut is created in the "Tables" section of Lakehouse and it is in delta format, it can also be read as a managed table using Spark SQL syntax.

df = chispa.read.format("delta").load("Tables/MyShortcut")display(df)

df = chispa.sql("SELECCIONE * DE MyLakehouse.MyShortcut LIMIT 1000")display(df)

sql

Shortcuts in the Lakehouse table section can also be read through the Lakehouse SQL endpoint. This can be accessed through the mode selector in Lakehouse or through SQL Server Management Studio (SSMS).

SELECT ABOVE (100) *OF [MyLakehouse].[dbo].[MyShortcut]

real-time analysis

Shortcuts in KQL databases are recognized as external tables. To query the shortcut, use the "external_table" function in the Kusto query language.

external_table('My shortcut')| take 100

analysis services

Power BI datasets can be created for Lakehouses that contain shortcuts in the Lakehouse table section. When the dataset is running in direct maritime mode, Analysis Services can read the data directly from the direct access.

no substance

Applications and services outside of the fabric can also access shortcuts through the OneLake API. OneLake supports a subset of the ADLS Gen2 and blob storage APIs. For more information on the OneLake API, seeAccess to OneLake with API.

https://onelake.dfs.fabric.microsoft.com/MyWorkspace/MyLakhouse/Tables/MyShortcut/MyFile.csv

Types of shortcuts

OneLake shortcuts support multiple file system data sources. These include internal OneLake locations, Azure Data Lake Storage Gen2, and Amazon S3.

OneLake Internal Shortcuts

OneLake's internal shortcuts allow you to reference data in existing drug articles. These elements include lakehouses, KQL databases, and data warehouses. The shortcut can point to a folder location within the same theme, between themes in the same workspace, or even between themes in different workspaces. When creating a shortcut between elements, the element types do not have to match. For example, you can create a shortcut in Lakehouse that points to data in a data store.

When accessing data through a shortcut to another OneLake location, the caller's identity will be used to authorize access to the data on the shortcut's destination path*. This user must have permissions on the destination location to read the data.

ADLS Shortcuts

You can also create shortcuts to ADLS Gen2 storage accounts. When creating ADLS shortcuts, the destination path can point to any folder in the hierarchical namespace. The destination path must contain at least one container name.

Access:

ADLS shortcuts must point to the DFS endpoint of the storage account.
Example:https://nombredecuenta.dfs.core.windows.net/

Authorization:

ADLS shortcuts use a delegated authorization model. In this model, the creator of the shortcut provides a credential for the ADLS shortcut, and all access to that shortcut will be authenticated with that credential. The supported delegate types are Account Key, SAS Token, OAuth, and Service Principal.

• SAS sheet- must contain at least the following permissions: Read, List, Execute

• OAuth identity- You must have the Storage Blob Data Reader, Storage Blob Data Contributor, or Storage Blob Data Owner role on the storage account.

• Supervisor- You must have the Storage Blob Data Reader, Storage Blob Data Contributor, or Storage Blob Data Owner role on the storage account.

S3 Shortcuts

You can also create shortcuts to Amazon S3 accounts. When creating shortcuts to Amazon S3, the destination path must contain at least one bucket name. S3 doesn't natively support hierarchical namespaces, but you can use prefixes to mimic a directory structure. You can include prefixes in the shortcut path to further reduce the scope of data available through the shortcut. When accessing data via an S3 shortcut, the prefixes will be rendered as folders.

Access:

S3 shortcuts must point to the https endpoint of the S3 bucket. Example:https://bucketname.s3.region.amazonaws.com/

Authorization:

S3 shortcuts use a delegated authorization model. In this model, the creator of the shortcut provides a credential for the S3 shortcut, and all access to that shortcut will be authenticated with that credential. The supported delegated credentials are a key and a secret for an IAM user.

The IAM must have at least read-only (Get, List) permissions on the bucket that the shortcut points to.

How Shortcuts Use Cloud Connections

ADLS and S3 passthrough authorization is delegated through the use of cloud connections. When a new ADLS or S3 shortcut is created, a user creates a new connection or selects an existing connection to the data source. When a connection is established to a shortcut, this is considered a "bind" operation. Only users with permission to connect can perform the bind operation. If a user does not have permissions for the connection, they cannot create new shortcuts using that connection.

permissions

Permissions for shortcuts are controlled by a combination of the permissions on the shortcut path and the destination path. When a user accesses a shortcut, the more restrictive permission of the two locations is applied. Therefore, a user who has read/write permissions to Lakehouse but only read permissions to the shortcut target will not be able to write to the shortcut target path. Similarly, a user who only has read permissions on Lakehouse, but reads/writes to the shortcut target, will also not be able to write to the shortcut target path.

roles in the workplace

The following table lists the permissions related to shortcuts for each role in the workspace. For more information, seeroles in the workplace.

¹The user must have a role that grants write permission to the shortcut location and, at a minimum, read permission to the target location.

²The user must have a role that grants read permission to both the shortcut location and the target location.

³The user must have a role that grants write permission to both the shortcut location and the target location.

How do shortcuts handle deletions?

Shortcuts do not perform overlapping deletes. When you perform a delete operation on a shortcut, you only delete the shortcut object. The data in the shortcut destination remains unchanged. However, if you perform a delete operation on a file or folder in a shortcut and you have permissions on the shortcut destination to perform the delete operation, the files and/or folders on the destination will be deleted. The following example illustrates this point.

delete example

User A has a lake house with the following route:

MyLakehouse\Files\my shortcut\Foo\Bar

my shortcutis a shortcut that points to an ADLS Gen2 account that containsFoo\Barmapper

Delete a shortcut object

User A performs a delete on the following path:

MyLakehouse\Files\my shortcut

In this case,my shortcutis removed from Lakehouse. Shortcuts do not cascade deletes, therefore files and folders in the ADLS Gen2 accountFoo\Barremain unaffected.

Deleting content referenced by a shortcut

User A performs a delete on the following path:

MyLakehouse\Files\my shortcut\Foo\Bar

In this case, if user A has write permissions on the ADLS Gen2 accountBardirectory is removed from the ADLS Gen2 account.

Lowering of the working area

When you create shortcuts between multiple quilt elements in a workspace, you can visualize the shortcut relationships through the line view of the workspace. choosefamily viewbutton () in the upper right corner of the Workspace Explorer.

Known issues and limitations

• The maximum number of shortcuts per cloth is 10,000.
• The maximum number of shortcuts on a single OneLake route is 10.
• The maximum number of shortcuts for shortcut links is 5.
• ADLS and S3 shortcut destination paths cannot contain reserved characters from RCF 3986, section 2.2.
• OneLake destination shortcut paths must not contain "%" characters.
• Shortcuts do not support non-latin characters.
• Copy Blob api does not support ADLS or S3 shortcuts.
• The copy function does not work on shortcuts that point directly to ADLS containers. It is recommended that you create ADLS shortcuts to a directory that is at least one level below a container.
• OneLake shortcuts pointing to ADLS or S3 shortcuts are not supported.
• Additional shortcuts cannot be created in ADLS or S3 shortcuts.

Next step

• creating shortcuts