Category: Engineering

Today, we’re excited to announce the launch of Brickbuilder Unity Catalog Accelerators. This is an expansion to the Brickbuilder Accelerator program, which pairs the expertise of system integrators and consulting partners with their proven frameworks and pre-built code to help organizations quickly implement a specific methodology or Databricks Data Intelligence Platform capability. The Brickbuilder Program, […]

The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set of capabilities described in this blog post will be available starting with the upcoming Apache Spark 4.0 and Databricks Runtime 14.3. Write more confident DataFrame transformations with DataFrame equality test functions Working […]

Grab is Southeast Asia’s leading superapp, providing a suite of services that brings essential needs to users throughout the region. Its offerings include ride-hailing, food delivery, parcel delivery, mobile payments, and more. With safety, efficiency, and user-centered design at heart, Grab remains dedicated to solving everyday issues and improving the lives of millions. As the […]

The pace of innovation is faster than ever before, and through our AI work at eBay, we believe we’ve unlocked three major tracks to developer productivity: utilizing a commercial offering, fine-tuning an existing Large Language Model (LLM), and leveraging an internal network. Each of these tracks requires additional resources to integrate, but it’s not a […]

eBay’s Chief AI Officer, Nitzan Mekel-Bobrov, recently joined the Bloomberg Intelligence podcast “Tech Disruptors” to discuss his role, how he sees the rapidly changing landscape of AI, eBay’s place in the larger ecommerce industry, and more. Nitzan noted that eBay is in a unique position to take advantage of new tools, thanks to the combination of […]

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this series is covered in Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming – we recommend reading the first part before reading this post. In the Project Lightspeed update blog, we […]

Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the Spark SQL engine. Most incremental and streaming workloads on the Databricks Lakehouse Platform are powered by Structured Streaming, including Delta Live Tables and Auto Loader. We have seen exponential growth in Structured […]

Introduction In the domain of data processing, data analysts run their ad hoc queries on the data lake. The lake serves as an interface between our analytics and production environment, preventing downstream queries from impacting upstream data ingestion pipelines. To ensure efficient data processing in the data lake, choosing appropriate storage formats is crucial. The […]

The Grab superapp offers a comprehensive array of services from ride-hailing and food delivery to financial services. This creates multifaceted user journeys, traversing homepages, product pages, checkouts, and interactions with diverse content, including advertisements and promo codes. Background: Why ads and attribution matter in our superapp Ads are crucial for Grab in driving user engagement […]

Five university students are headed to eBay for summer internships this year after claiming top prize at the 2023 eBay University Machine Learning Challenge. The annual competition asks students to dream up innovative solutions to real-world ecommerce problems, rewarding top teams with valuable work experience while recruiting promising new talent to the company. This year’s […]