Category: Engineering

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines. It includes new native, highly scalable connectors for databases including MySQL, Postgres, SQL Server and Oracle and enterprise applications like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow and Google Analytics. Users can transform […]

Over the last year, we have seen a surge of commercial and open-source foundation models showing strong reasoning abilities on general knowledge tasks. While general models are an important building block, production AI applications often employ Compound AI Systems, which leverage multiple components such as tuned models, retrieval, tool use, and reasoning agents. AI systems […]

Delta Lake has over 20M+ monthly downloads. BigQuery, now with first-party support for Delta Lake, builds on Delta’s rich connector ecosystem and seamlessly integrates with Databricks. In this blog, we will cover:   Delta Lake on Google Cloud Building an open data lakehouse with Databricks and BigQuery How to read Delta Lake in BigQuery Delta […]

Profile-guided optimisation (PGO) is a technique where CPU profile data for an application is collected and fed back into the next compiler build of Go application. The compiler then uses this CPU profile data to optimise the performance of that build by around 2-14% currently (future releases could likely improve this figure further). High level view of how […]

Looking to give your listings an instant glow up? eBay is making it easier than ever with an AI-powered background enhancement tool that helps sellers seamlessly transform their product photos from everyday snapshots into professional grade imagery.  Background enhancement lets you remove a less-than-glamorous background from an existing photo using eBay’s background removal tool, leaving […]

We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared with storing these data as JSON strings, while maintaining the flexibility for supporting highly nested and evolving schema. Working with semi-structured data has long been a foundational capability of the Lakehouse. Endpoint […]

Delta Lake has proven to be the most popular and fastest lakehouse format over the years. Delta Lake Universal Format (UniForm), now available in GA, builds on Delta Lake’s rich connector ecosystem to combine Delta Lake’s superior price-performance with access to every tool in your stack. With Delta Lake UniForm, you can write a single […]

In a previous post, we introduced our systems for running marketing campaigns. Although we sent millions of messages daily, we had little insight into their effectiveness. Did they engage our users with our promotions? Did they encourage more transactions and bookings? As Grab’s business expanded and the number of marketing campaigns increased, understanding the impact […]

Following the announcement we made around a suite of tools for Retrieval Augmented Generation, today we are thrilled to announce the general availability of Mosaic AI Vector Search in Databricks. What is Mosaic AI Vector Search? Vector Search enables developers to improve the accuracy of their Retrieval Augmented Generation (RAG) and generative AI applications through similarity […]

Following the announcement we made around a suite of tools for Retrieval Augmented Generation, today we are thrilled to announce the general availability of Databricks Vector Search.  What is Databricks Vector Search? Vector Search enables developers to improve the accuracy of their Retrieval Augmented Generation (RAG) and generative AI applications through similarity search over unstructured […]