We're excited to share the latest features and improvements in Azure Synapse Data Explorer, also known as Kusto, that promise to make your data analysis experience smoother and more productive than ever.
We are pleased to announce the following new features and capabilities:
1. Synapse Real Time Analytics and Microsoft Fabric (Preview)
Earlier today, we launched Microsoft Fabric, an all-in-one analytics solution for enterprises that covers everything from data movement to data science to real-time analytics to business intelligence. Microsoft Fabric brings together new and existing components of Power BI, Azure Synapse, and Azure Data Explorer into a single, integrated environment.
The platform is built on a Software as a Service (SaaS) foundation, which takes simplicity and integration to a whole new level. Kusto is available as part of the Realtime Analytics experience in Fabric, providing all of the great query, performance, and scale capabilities customers are used to with Azure Synapse Data Explorer in a SaaSified experience.Sign up for the free trial.
For more information readReal-time analytics on the drug advertising blogyRTA documents of substances.
2. Free clusters (GA)
Ever since we launched Kusto externally as "Azure Data Explorer", we've been looking for ways to make Kusto and KQL available to everyone, regardless of whether they have an Azure subscription. Last year, we announced the start of Kusto's free offerings in Preview, allowing anyone interested in big data analytics to claim their own free personal cluster. Kusto Free Cluster Deals Are Now Officially GA From Preview!Learn more.
3. Azure Event Hubs - Send data to Kusto directly from the Azure Event Hubs page (coming soon)
Azure Event Hubs is a highly scalable data streaming platform that can handle millions of events per hour. second. Event hubs represent the "front door" to an event pipeline, often referred to as an event recorder in solution architectures.
Coming soon, with just a few clicks, you'll be able to bring your data from the Event Hubs resource page in the Azure portal directly into Kusto with no additional configuration required.
The Azure Data Explorer interface makes it easy to preview, fast load, transform, and query your event hub data. Once the data is in Azure Data Explorer, you can also visualize your insights with charts and graphs, making it easy to share your findings with others.
4. Azure Functions input and output binding (preview)
You can now integrate your Azure functions with Kusto viarestraints- a declarative way to connect external resources to Azure functions. With Kusto bindings, you can seamlessly read and write data from your Azure Functions with minimal code in a declarative manner by leveraging the Azure Functions framework.There are two types of links:
- Input link: read Kusto data.
- Output link: write data to Kusto.
This new feature makes it easy to create data processing pipelines that involve Kusto and allows you to easily integrate Kusto with other Azure services using Azure Functions.Learn more.
5. Azure Log Alerts support for Azure Data Explorer (preview)
Azure Monitor alerts allow you to monitor your Azure and application telemetry to quickly identify issues impacting your service. Azure Monitor Alerts now has support for running queries against Azure Data Explorer (ADX) tables and even join data between ADX and data in Log Analytics and Application Insights.
As part of this newly added support, log alert rules now support managed identities for Azure resources, so you can see and control the exact permissions of your log alert rule.
To write queries for Log Search Alerts (LSAs), use ADX('
6. Cosmos DB Synapse Link hasta ADX (GA)
Enabling near real-time analytics on Cosmos DB data in a managed environment, ie. Azure Data Explorer data connection will be generally available (GA) soon.
When using stream ingestion, the latency between Cosmos DB and ADX can be as low as subseconds. Now you can have direct Power BI queries on your Cosmos DB data, generating analytics queries in a second. You can control how data from JSON documents is mapped to tables and transform it to suit your use case.
This brings the best of both worlds: low/fast latency transactional workloads with Azure Cosmos DB and ad hoc/fast analytics with Azure Data Explorer.Learn more.
7. Serilog (GA) rod
Serilog is a popular logging framework for .NET applications. Serilog allows developers to control which log statements are emitted with arbitrary granularity based on logger name, level, and message pattern.Serilog the sink, also known as an aggregator, to Azure Data Explorer streams your log data to Azure Data Explorer, where you can analyze and visualize your logs in real time.
8. Log Stick (GA)
NLog is a popular logging framework for .NET applications. It provides a flexible and extensible way to log messages to various destinations such as files, databases, and event loggers.
NLog allows developers to configure log messages to filter, format, and redirect based on severity and context.Kusto NLog sinkis a library that allows sending log messages from an NLog logger to a Kusto cluster. It is based on the Kusto .NET client library, which provides a set of APIs for interacting with Kusto.
9. Support reading Delta external table (Preview)
You can now create external tables on Delta Lake. External ADX tables work with data that is stored and managed outside of the Azure Data Explorer cluster.
Delta Lake is a popular open source storage layer. This new feature allows you to query delta tables with ADX for scenarios where you want to query directly from delta tables.
The table creation syntax automatically supports schema derivation from your delta tables, but you can also specify it explicitly.Learn more.
10. External table read support for Cosmos db, MySQL and Postgres (coming soon)
Until now it was possible to query cosmos db, mySQL and postgres using the following plugins: (cosmosdb_sql_request, mysql_request y postgresql_request).
The only SQL database that supported external tables was SQL Server.
At the end of June, we will also introduce the ability to create external tables of this type of database, which will allow you to write queries in a more convenient way.
11. Improved performance for SQL requests (GEORGIA)
SQL queries will be more optimized, both when using external tables and when usingsql_request_plugin. Whenever possible, Kusto will insert predicates and functions into the SQL query.
For example, the following query was used to extract all the data from the Student external table, and then Kusto Engine applied the filter to FirstName and counted the results:
`external_table("Students") | where Name == “Bob” | count'
With push-down predicates, instead of pulling all the data from SQL, Kusto Engine will send a query to SQL similar to this:
`SELECT COUNT(*) FROM Estudiantes WHERE FirstName = 'Bob'`
This will reduce the load on SQL Server and also on Kusto Engine, resulting in much more efficient queries that use fewer resources on both sides.
12. New cosine similarity function for vector similarity (GA) search
If you are dealing with Large Language Models (LLM), OpenAI, and Embeds, you can use ADX for vector similarity search. We have released a new custom function cosine_series_similarity_fl to perform vector similarity searches on vectors stored in ADX. Learn more.
13. Kusto emulator on Linux (GA)
You can now run Kusto Emulator as a Linux container.
Unlike Windows Container, which was released almost a year ago, Linux Container is smaller, loads faster, and runs on both Windows and Linux. You can still do local development and automated testing, but now in Linux environments and with a smaller footprint!
14. Python Plugin (GA) Improvements:
HePython add-onallows you to execute a User Defined Function (UDF) using a Python script.
we got youi upgraded the python image to 3.10.8 and the latest packages and iimproved sandboxing technology to use Hyper-V containers, which are more secure and perform better on the latest SKUs
15. Followers – Role Level (GA) Sharing
The trace database feature allows you to attach a database located in another cluster to your Azure Data Explorer cluster.
Until now, you could only share specific tables and external tables with other clusters, but you had no control over specific features. But now you can share specific features, giving you full control over which entities to share.
You can usefeatures to include, yfunctions to excludeparameters to control this behavior. Learn more.
16. Confidential Computing Support for Azure Data Explorer (Preview)
The Azure Data Explorer Confidential solution is based on AMD EPYC SEV-SNP technologies. This AMD technology adds an advanced level of security to virtualized environments by encrypting all data residing in memory and offering a hierarchical paging model that allows a VM administrator to access only the VM's memory. This creates an isolated memory region on each VM that is inaccessible to other VM managers and the hypervisor manager.
The ADX Confidential computing solution is based on the AMD ECasv5 family of SKUs. To create a confidential Azure Data Explorer cluster, select an ECASV5 SKU during cluster creation. You can use ARM from the Azure portal to deploy or migrate to a trusted compute cluster.
17. New Azure Policies (GA)
Five new built-in Azure policies are now available to help you improve your security posture con ADX.
- Access to the public network in Azure Data Explorer must be disabled
- Configure Azure Data Explorer to disable public network access
- Azure Data Explorer cluster must use a private link
- Azure Data Explorer must use a SKU that supports private links
- Configure Azure Data Explorer clusters with private endpoints
Learn more.
18. Azure for Operators (AFO)
The Azure AIOps for Carriers program helps carriers modernize some of the world's most complex networks using the same technologies that power Microsoft Azure.
Two new services were announced at Mobile World Congress,Azure Operator InsightsyAzure Carrier Services Administrator. Azure Operator Insights enables the collection and analysis of massive amounts of network data collected from complex multi-party or multi-vendor network functions. Provides insights into carrier-specific workloads to help carriers understand the health of their networks and the quality of their subscribers' experiences. Azure Operator Insights leverages Azure Data Explorerunique ability to analyze complex data sets to provide near real-time information to operators. Operators looking to reduce operational costs combine Azure Operator Insights with Azure Operator Service Manager to automate changes to their network based on the same insights.
19. Well Structured Review for ADX (GA)
The Well-Architected Framework (WAF) allows customers to evaluate e.g.their workload according to the following pillars:
- reliability
- cost management
- operational excellence
- Security
- Performance efficiency.
Azure Data Explorer is now included as one of the available WAF analytics services. After evaluating your workload, you'll receive pragmatic recommendations based on your specific needs. Over time, you can improve your score by following your personal recommendations.Learn more.
20. Kusto Detective Agency Season 2
And finally, we are happy to present Season 2 of your beloved Kusto Detective Agency (KDA), a playful way to learn Kusto Query Language (KQL).
Thousands of data enthusiasts have participated in Season 1 so far, proudly displaying their badges on social media platforms. With Season 2, we're taking the KDA experience to the next level with more cases, more fun, and even more exciting prizes to win.
Recruitment now at:https://detective.kusto.io/
We'd love to hear your feedback and overall experience with these new features.