Microsoft OneLake i Fabric, OneDrive up to data | Microsoft Web Blog | microsoft storage (2023)

See Arun Ulagaratchagan's blog post to read it in fullMicrosoft Fabric Preview Announcement.

Organizations are investing heavily in data lake strategies with the vision of having a central place to store all their data, breaking down silos, and simplifying data blending, analysis, security, governance, and discovery.

In reality, the vision is largely an illusion. Enterprise data lakes are primarily implemented as custom projects using raw storage covered with massive sticky code designed to enable scalability, collaboration, compliance, security, and governance. Data masking patterns with lakes controlled by separate business domains adds additional overhead and fragmentation with multiple teams managing their own lake resources. To break down these silos, these organizations are creating more complicated solutions with complex data movement to facilitate sharing and reuse. And for all of this to be useful for the business side, IT organizations must also create data warehouses, data marts, and cubes that create additional copies of marine data. The resulting data lake implementation is often a complex and difficult to manage system, full of silos and redundant data.

Introducing Microsoft OneLake: "OneDrive for data." OneLake is a complete, rich, ready-to-use enterprise data lake delivered as a SaaS service. Just as organizations use OneDrive for their documents, they now have OneLake for their data. OneLake is at the heart of Fabric's lake-centric approach. This gives clients:

• Yo datasøfor the entire organization on a large scale
• A copy of datafor use in multiple scanning engines
• A security modellive naturally with data on the lake (coming soon)
• A centralized OneLake data center for data discovery and management

OneLake enhances collaboration on a single data lake for the entire organization. Each Fabric tenant will have exactly one OneLake where all data for all projects and for all users will be stored. OneLake is automatically available with all Fabric tenants with no additional resources to configure or manage.

Regulated by default with distributed ownership for collaboration

The tenant concept is a unique advantage of a SaaS service. Set clear governance and compliance boundaries that are controlled by the tenant administrator, and all data in OneLake is governed by tenant policies. This tightly controlled system allows OneLake to be open for any user to add their own contributions to OneLake from anywhere in the organization without any friction.

Just as any Office user can create a new Teams channel or SharePoint page without coordinating with the admin, OneLake enables similar distributed ownership across workspaces. Workspaces allow different parts of the organization to work independently while building the same data lake. Each workspace has its own administrator and access control. Each workspace is powered by a capacity that resides in an area selected by the user. This means OneLake fully scales for customers doing business in multiple countries and natively supports local data residency requirements. OneLake spans the globe with different workspaces living in different countries while still being part of the same logical lake.

Data mesh and domains

With Microsoft OneLake, we offer a unified data lake that eliminates all data silos. However, the possibilities extend further. OneLake also allows you to organize and manage data in onelogical pathallowing different business groups to effectively manage and control their own data. This pattern is known as a "data mesh".

With Onelake's native support for data mesh, organizations can easily definecommercial domainssuch as marketing, sales, human resources and more. Once the domains are defined and contain the respective OneLake data, various consumption and management functions light up the domain. This allows for more optimized consumption for business users and more granular control per domain for administrators.

For example, data owners and enterprises can discover and consume Onelake data filtered based on their areas of interest, and administrators can delegate settings at the domain level, allowing for different definitions and controls on a per-user basis. Business Unit.

With OneLake's built-in domains, OneLake is the first data lake to provide built-in support for datamesh-as-a-service.

Open to all levels

OneLake is open to all levels. OneLake is based on Azure Data Lake Storage Gen2 and can support any file type, structured or unstructured. All Fabric data elements, such as data warehouses and lakes, automatically store their data in OneLake in delta parquet format. This allows data engineers to load a lake using Spark, SQL developers to load data into fully transactional data warehouses using T-SQL, and all contributors to build the same data lake.

OneLake supports the same ADLS Gen2 APIs and SDKs to be compatible with existing ADLS Gen2 applications, including Azure Databricks. The data in OneLake can be treated as one big ADLS storage account for the entire organization. Each Fabric workspace appears as a container within this storage account, while different data items appear as folders under those containers.

OneLake File Explorer for Windows

OneLake acts like OneDrive for data. Like OneDrive, OneLake data is easily accessible from within Windows using OneLake File Explorer for Windows. In Windows, you can easily navigate all your workspaces, data items, upload, download, or change files, just like you can in OneDrive. OneLake File Explorer simplifies data lakes and makes them accessible even to non-technical business users.

Install OneLake File Explorer for Windows!

OneLake aims to bring you the most value from a single data copy without data movement or duplication. You no longer need to copy data just to use it with another engine, or break down silos so data can be analyzed with other data.

Shortcuts allow you to connect data between business domains without data movement

A large organization will typically have many data domains with different data owners. Shortcuts provide connections between different data elements across domains so that data can be virtualized into a single data product without data duplication, data movement, or change of data ownership.

A shortcut is a symbolic link. It acts as metadata pointing from one data location to another. They look like Windows shortcuts. When you create a shortcut from one location to another location, the files will appear in the shortcut location as if they were physically present. Tables from one store can be made available to another store without copying data from the store to the store. Since all the data is already in OneLake, you can simply create a shortcut from the repository to the lakehouse and the data will appear in the lakehouse as if you copied it. Because the data is not copied, there is no secondary copy to maintain. When data is changed in the warehouse, these changes are automatically reflected in the lake house.

Shortcuts are used to consolidate data across workspaces and domains without changing ownership of the data. The same data can be used multiple times indifferent locations, while the original owner is still responsible for uploading and managing it.

Direct access to Azure Data Lake Store gen2

Organizations already have their data in lakes outside of OneLake. We have also extended the shortcuts to support these data stores. You can create shortcuts to existing ADLS gen2 accounts so that all your data can be virtualized in OneLake and the data appears as if it physically exists. Owners of these accounts can continue to manage them independently of OneLake.

Shortcuts to S3 make OneLake the first multicloud data lake

OneLake goes beyond Microsoft and Azure to become the first multi-cloud data lake with direct access to Amazon S3 buckets. Through shortcuts, S3 buckets can be virtualized on OneLake. Your data is mapped to the same unified namespace and can be accessed through the same APIs, including the ADLS gen 2 APIs. Power BI notebooks, SQL queries, and reports can span multiple clouds without end-users They should know that they are doing it. Transparent intelligent caching (coming soon) will bring data closer to computation and reduce egress costs.

Dataverse Shortcuts (coming soon)

Dataverse generates shortcuts to Microsoft Fabric, generates Synapse Lakehouse, and an SQL endpoint for Dynamics 365 data and PowerApps that enable next-generation Power BI capabilities. This direct integration between Dataverse and Microsoft Fabric eliminates the need to create and maintain custom ETL pipelines or use third-party data integration tools. Dataverse Shortcuts ensure that data always remains in Dataverse, and as data is updated in Dynamics 365, changes are automatically reflected in Power BI reports.

Data analysts will be able to launch Microsoft Fabric directly from the PowerApps experience. Data engineers can launch Microsoft Fabric with Synapse Link and work with data with Python or Spark laptops. Direct integration between Dataverse and Microsoft Fabric saves a lot of time and effort.

To participate in the preliminary visithttps://aka.ms/DataverseExtendsToFabricand take a lookDataverse and Microsoft Fabric IntegrationBlog.

One copy of data with multiple analysis engines

Compute powers all analytics experiences in Fabric. With OneLake in Fabric, compute is completely separated from storage. While OneLake represents the single data warehouse for the entire organization, multiple other Fabric analytics computers can access the same copy of data without having to import it into another copy. It is no longer necessary to copy data just to use it with another engine. You can always choose the best motor for the job you are trying to do.

For example, imagine you have a team of SQL engineers building a fully transactional data warehouse. They can use the T-SQL engine and all the power of T-SQL to create tables, transform and load data into tables. If a data scientist wants to make use of this data, he no longer needs to go through a special Spark/SQL driver. All data is stored in OneLake in delta parquet format. Data scientists can use the full power of the Spark engine and its open source libraries directly on data.

Business users can build Power BI reports directly on top of OneLake using the new Direct Lake mode in the Analysis Services engine. The Analysis Services engine is what powers Power BI datasets and has always offered two ways to access data, import and query directly. Direct Lake mode gives users full import speedsinyou need to copy the data combining the best of import and direct query. Learn more about Direct Lake athttps://aka.ms/DirectLake.

If you have a data engineering team that prefers to use Spark to build a lake house, they can use notebooks to put their data into OneLake in delta/parquet format. This data can be automatically consumed by all engines. The same is true for data obtained by other engines using the ADLS DFS APIs or practically aggregated through shortcuts. When you define your organization's data strategy, you no longer need to optimize for different teams with different skills and preferences. Teams that want to work with SQL can do so. Teams that want to work with Spark can work with Spark. Computers that use other engines to get their data can continue to do so. They are all building the same data lake. There are no silos.

Managing data security (table, column, and row levels) across different data engines is an ongoing nightmare for customers. OneLake will bring with it a universal security model that allows you to define security definitions once. Unlike other solutions that require you to define security definitions in another layer, these security definitions will live in OneLake along with the data. Security definitions are applied uniformly across all engines inside and outside of the Fabric. This model will arrive soon.

Finally, OneLake has provided a central solution for all data, but how can this data be accessed, discovered, managed, and reused? These aspects are critical as organizations increasingly require easy access and discovery of high-quality data for reuse, data-driven decision-making and insights. We are excited to introduce the OneLake Data Center (an evolution of the Power BI Data Center). ThatOneLake data centerIt acts as a centralized interface to all data contained in OneLake, including data warehouses, lake houses and their SQL terminals, KQL databases, data marts, and data sets. The OneLake data center is the central location for easy data discovery, management, and reuse.

With OneLake's data center, users can view data across their business domains and filter to view a specific domain of interest to them, view all approved authorized data in one place, and view all user-owned data to make the data management as easy as possible in one central location.

OneLake's data center is particularly powerful for users who access data across multiple workspaces. OneLake's Data Center Explorer offers an intuitive and efficient way to navigate through workspaces to find specific data items. With Explorer, users can access large amounts of data faster and easier.

Once data is discovered, users can perform a wide range of actions: explore its properties and tables, identify if it is marked as sensitive and should be treated with caution, trace the data pipeline, and perform impact analysis on data spaces. work, reuse that data and build on it. at the top, find valuable insights and make informed business decisions or take further action.

The OneLake data center is integrated into multiple experiences within the Fabric service and Power BI Desktop. This integration ensures that users can quickly and easily find the data they need in any context and in a consistent manner. For example, in Power BI Desktop, users can access the OneLake data center experience to explore and connect to available elements, avoiding the need to create new data sources. This approach fosters a culture of data reuse and helps organizations achieve their goals more effectively.

• • Read the OneLake documentation:https://aka.ms/onelakedocs

Microsoft Fabric is currently in preview. Try all Fabric has to offer by signing up for the free trial, no credit card information required. Everyone who signs up gets a fixed Fabric test capability that can be used for any feature or capability, from data integration to building machine learning models. Existing Power BI Premium customers can simply activate Fabric through the Power BI admin portal. After July 1, 2023, Fabric will be enabled for all Power BI tenants.

Sign up for the free trial. For more information readSubstance test documents.

To learn more about Microsoft Fabric, consider:

• • Sign up for Microsoft Fabric free trial
  • Visit the Microsoft Fabric website
  • To read the more detailed Fabric experience announcement blogs:
    • Data Factory Error on Fabric Blog
    • Synapse Data Engineering erfaring i Fabric blog
    • Synapse Data Science-erfaring i Fabric-blog
    • Synapse Data Warehousing erfaring i Fabric blog
    • Synapse Real-Time Analytics Experience on Fabric Blog
    • Power BI announcement blog
    • Data Activator erfaring i Fabric blog
    • Administration and management at Stofblog
    • Microsoft 365 Data Integration i Fabric Blog
    • Dataverse and Microsoft Fabric IntegrationBlog
  • Explore the technical documentation of the substance
  • Reading the free Getting Started with Fabric eBook
  • Scan the fabric throughRoute
  • see things for freeWeb seminarserie
  • Unionsfabric communityto ask your questions, share your comments and learn from others
  • visitingIdeas de Microsoft Fabricto submit suggestions for improvements and vote on your colleagues' ideas