Category: Engineering

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this series is covered in Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming – we recommend reading the first part before reading this post. In the Project Lightspeed update blog, we […]

Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the Spark SQL engine. Most incremental and streaming workloads on the Databricks Lakehouse Platform are powered by Structured Streaming, including Delta Live Tables and Auto Loader. We have seen exponential growth in Structured […]

Introduction In the domain of data processing, data analysts run their ad hoc queries on the data lake. The lake serves as an interface between our analytics and production environment, preventing downstream queries from impacting upstream data ingestion pipelines. To ensure efficient data processing in the data lake, choosing appropriate storage formats is crucial. The […]

The Grab superapp offers a comprehensive array of services from ride-hailing and food delivery to financial services. This creates multifaceted user journeys, traversing homepages, product pages, checkouts, and interactions with diverse content, including advertisements and promo codes. Background: Why ads and attribution matter in our superapp Ads are crucial for Grab in driving user engagement […]

Five university students are headed to eBay for summer internships this year after claiming top prize at the 2023 eBay University Machine Learning Challenge. The annual competition asks students to dream up innovative solutions to real-world ecommerce problems, rewarding top teams with valuable work experience while recruiting promising new talent to the company. This year’s […]

For the past two years, Databricks has collaborated with leading consulting partners to build innovative solutions for industry, migration, and data and AI use cases. Based on a foundation of proven customer deployments, Databricks Brickbuilder Solutions and Accelerators package together the experience and knowledge of our partners to help businesses unlock the full potential of […]

Today, we’re pleased to share that Lauren Wilcox, Sr. Director of Responsible AI at eBay, was named a 2023 ACM Distinguished Member by the Association for Computing Machinery (ACM). This prestigious recognition is awarded to those who have made significant contributions to the field of computing. Wilcox was nominated based on her research contributions in […]

In the fast-paced world of on-demand delivery, maintaining safe marketplaces is a complex undertaking. Grab, a leading superapp in Southeast Asia, operates GrabFood and GrabMart, two popular marketplaces that connect consumers with a wide range of food and daily necessities. With more than 100k listings for different items updated daily by our merchants across eight different countries, Grab […]

Introduction In this digital age, companies collect multitudes of data that enable the tracking of business metrics and performance. Over the years, data analytics tools for data storage and processing have evolved from the days of Excel sheets and macros to more advanced Map Reduce model tools like Spark, Hadoop, and Hive. This evolution has […]

Ray is an open-source unified compute framework that simplifies scaling AI and Python workloads in a distributed environment. Since we introduced support for running Ray on Databricks, we’ve witnessed numerous customers successfully deploying their machine learning use cases, which range from forecasting and deep reinforcement learning to fine-tuning LLMs. With the release of Ray version […]