Category: Engineering

Introduction In this blog post, we’ll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling you to count unique values, merge sketches, and estimate distinct counts with precision and efficiency. These implementations use the Apache Datasketches library for consistency with the open source community and easy integration […]

We are thrilled to introduce the support of a new SQL feature in Apache Spark and Databricks: Lateral Column Alias (LCA). This feature simplifies complex SQL queries by allowing users to reuse an expression specified earlier in the same SELECT list, eliminating the need to use nested subqueries and Common Table Expressions (CTEs) in many […]

Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our sincere appreciation to the Apache Spark community for their invaluable contributions to the Spark 3.5 release. Aligned with our mission to make Spark more accessible, versatile, and efficient than ever before, this […]

Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval augmented generation (RAG) architecture is quickly becoming the industry standard for developing chatbots because it combines the benefits of a knowledge base (via a vector store) and generative models (e.g. GPT-3.5 and […]

As part of MLflow 2’s support for LLMOps, we are excited to introduce the latest updates to support prompt engineering in MLflow 2.7.  Assess LLM project viability with an interactive prompt interfacePrompt engineering is a great way to quickly assess if a use case can be solved with a large language model (LLM). With the […]

Artificial Intelligence (AI) has evolved and improved dramatically over the past 18 months, and the technology has myriad applications and amazing potential for the future of ecommerce. But with such lightning-quick advancements, it’s vital that we in the technology industry, especially companies like eBay with the size, scope, and personnel to guide the future of […]

We’ve been hard at work on the next version of a new, magical listing experience, which uses AI to analyze, research, and extrapolate information from a small amount of data provided by the seller — greatly simplifying the process of listing an item on eBay. In the first version, a seller could add in a […]

We are thrilled to announce that you can run even more workloads on Databricks’ highly efficient multi-user clusters thanks to new security and governance features in Unity Catalog Data teams can now develop and run SQL, Python and Scala workloads securely on shared compute resources. With that, Databricks is the only platform in the industry […]

We are thrilled to announce great enhancements to onboard more workloads to Unity Catalog clusters in shared access mode, Databricks’ highly efficient, secure multi-user clusters. Data teams can now develop and run SQL, Python and Scala workloads securely on shared compute resources. With that, Databricks is the only platform in the industry offering fine-grained access […]

With the rapid advancement of neural network-based techniques and Large Language Model (LLM) research, businesses are increasingly interested in AI applications for value generation. They employ various machine learning approaches, both generative and non-generative, to address text-related challenges such as classification, summarization, sequence-to-sequence tasks, and controlled text generation. Organizations can opt for third-party APIs, but […]