TL;DR: Apache Spark 4 lets you build first-class data sources in pure Python. If your reader yields Arrow RecordBatch objects, Spark ingests them with reduced Python↔JVM serialization overhead. I used this to ship a ROOT data format reader for PySpark.

I recently chased an Oracle

Apache Spark is renowned for its speed and efficiency in handling large-scale data processing. However, optimizing Spark to achieve maximum performance requires a precise understanding of its inner workings. This blog post will guide you through establishing a Spark Performance Lab with essential tools and techniques aimed at enhancing Spark performance through detailed metrics analysis.

In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features.

TL;DR Explore a step-by-step example of troubleshooting Apache Spark job performance using flame graph visualization and profiling. Discover the seamless integration of Grafana Pyroscope with Spark for streamlined data collection and visualization.

Dive into a comprehensive load-testing exploration using Apache Spark with CPU-intensive workloads.

This blog post is about building a getting-started example for semantic search using vector databases and large language models (LLMs), an example of retrieval augmented generation (RAG) architecture. You can find the accompanying notebook at this link. See also the SWAN gallery.

Are you looking at some resources to get you up to speed with popular Deep Learning and Data processing frameworks? This blog entry provides a curated collection of notebooks that will help you kickstart your journey.

You can find the notebooks at this link. See also the SWAN gallery.

This document describes some basic CPU load testing

Topic: 

We are in a golden age for distributed data processing, with an abundance of tools and solutions emerging from industry and open source. High Energy Physics (HEP) experiments at the LHC stand to profit from all this progress, as they are data-intensive operations with several hundreds of Petabytes of data to collect and process.

In today’s post I will describe the process of integrating OIDC implicit flow with ORDS running on Tomcat against Keycloak service. May sound complicated, but we’ll break it down into individual components so we know what we’re talking about.

TLDR; Apache Spark 3.0 comes with many improvements, including new features for memory monitoring.

Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination. Distributed applications can use it to maintain configuration information, implement naming, provide synchronization and group services.

In today’s post, we’ll be talking about the possible ways to manage the static images/CSS/JS that come shipped with APEX, when running on ORDS. They are separate resources (not contained in the DB like some other APEX images) necessary for your APEX applications look and behave the way they’re intended to. If you and your users browse your internet using lynx (see image below) feel free to skip this one. Otherwise - dig in!