Category: Engineering

Introduction In the initial article, LLM Powered Data Classification, we addressed how we integrated Large Language Models (LLM) to automate governance-related metadata generation. The LLM integration enabled us to resolve challenges in Gemini, such as restrictions on the customisation of machine learning classifiers and limitations of resources to train a customised model. Gemini is a […]

As the season of giving approaches, we at Databricks have been making our list and checking it twice–but instead of toys and treats, we’ve been wrapping up powerful performance improvements for our users. Through analyzing billions of production queries and listening closely to our community’s wishes, we’re excited to deliver a package of enhancements that […]

We’re excited to announce that materialized views (MVs) and streaming tables (STs) are now Generally Available in Databricks SQL on AWS and Azure. Streaming tables offer simple, incremental ingestion from sources like cloud storage and message buses with just a few lines of SQL. Materialized views precompute and incrementally update the results of queries so […]

When serving machine learning models, the latency between requesting a prediction and receiving a response is one of the most critical metrics for the end user. Latency includes the time a request takes to reach the endpoint, be processed by the model, and then return to the user. Serving models to users that are based […]

Introduction GrabX is Grab’s central platform for product configuration management. It has the capacity to control any component within Grab’s backend systems through configurations that are hosted directly on GrabX. GrabX clients read these configurations through an SDK, which reads the configurations in a way that’s asynchronous and eventually consistent. As a result, it takes […]

We’re excited to announce the General Availability of Publish to Microsoft Power BI Service from Unity Catalog, an integration that makes it easy to create Power BI web reports from your Unity Catalog data in just a few clicks. This feature enables seamless catalog integration and data model sync, allowing you to publish datasets directly […]

We are updating this blog to show developers how to leverage the latest features of Databricks and the advancements in Spark. Most data warehouse developers are very familiar with the ever-present star schema. Introduced by Ralph Kimball in the 1990s, a star schema is used to denormalize business data into dimensions (like time and product) […]

Founded in 2017, Logically is a leader in using AI to augment clients’ intelligence capability. By processing and analyzing vast amounts of data from websites, social platforms, and other digital sources, Logically identifies potential risks, emerging threats, and critical narratives, organizing them into actionable insights that cybersecurity teams, product managers, and engagement leaders can act […]

Introduction As the complexity of data retrieval requirements continue to grow, traditional search methods often struggle to provide relevant and accurate results, especially for nuanced or conceptual queries. Vector similarity search has emerged as a powerful technique for finding semantically similar information. It refers to finding vectors in a large dataset that are most similar […]

Introduction Retrieval-Augmented Generation (RAG) is a powerful process that is designed to integrate direct function calling to answer queries more efficiently by retrieving relevant information from a broad database. In the rapidly evolving business landscape, Data Analysts (DAs) are struggling with the growing number of data queries from stakeholders. The conventional method of manually writing […]