Category: Engineering

With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use. This blog post walks you through the key improvements. Here’s a rundown of some of the most important features added in Apache Spark 3.4 and 3.5 in 2023: Spark Connect introduces a […]

Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming‘s internal state data: the State Reader API. The State Reader API sets itself apart from well-known Spark data formats such as JSON, CSV, Avro, and Protobuf. Its primary purpose is facilitating the development, debugging, and troubleshooting of stateful Structured […]

Introduction Let’s explore a common scenario – your team is eager to leverage open source LLMs to build chatbots for customer support interactions. As the model handles customer inquiries in production, it might go unnoticed that some inputs or outputs are potentially inappropriate or unsafe. And only in the midst of an internal audit—if you […]

Today, we are excited to announce the general availability of Feature Serving. Features play a pivotal role in AI Applications, typically requiring considerable effort to be computed accurately and made accessible with low latency. This complexity makes it harder to introduce new features to improve the quality of applications in production. With Feature Serving, you […]

eBay is committed to the responsible use of AI. We see unique opportunities to develop AI-powered customer tools and services, which must be implemented safely while meeting our community’s needs. We have adopted the following key principles: 1. Inclusivity, Equity, and Fairness: eBay strives to enable equitable and fair AI experiences  Building with an inclusivity […]

Today, we’re excited to announce the launch of Brickbuilder Unity Catalog Accelerators. This is an expansion to the Brickbuilder Accelerator program, which pairs the expertise of system integrators and consulting partners with their proven frameworks and pre-built code to help organizations quickly implement a specific methodology or Databricks Data Intelligence Platform capability. The Brickbuilder Program, […]

The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set of capabilities described in this blog post will be available starting with the upcoming Apache Spark 4.0 and Databricks Runtime 14.3. Write more confident DataFrame transformations with DataFrame equality test functions Working […]

Grab is Southeast Asia’s leading superapp, providing a suite of services that brings essential needs to users throughout the region. Its offerings include ride-hailing, food delivery, parcel delivery, mobile payments, and more. With safety, efficiency, and user-centered design at heart, Grab remains dedicated to solving everyday issues and improving the lives of millions. As the […]

The pace of innovation is faster than ever before, and through our AI work at eBay, we believe we’ve unlocked three major tracks to developer productivity: utilizing a commercial offering, fine-tuning an existing Large Language Model (LLM), and leveraging an internal network. Each of these tracks requires additional resources to integrate, but it’s not a […]

eBay’s Chief AI Officer, Nitzan Mekel-Bobrov, recently joined the Bloomberg Intelligence podcast “Tech Disruptors” to discuss his role, how he sees the rapidly changing landscape of AI, eBay’s place in the larger ecommerce industry, and more. Nitzan noted that eBay is in a unique position to take advantage of new tools, thanks to the combination of […]