Apache Spark is renowned for its speed and efficiency in handling large-scale data processing. However, optimizing Spark to achieve maximum performance requires a precise understanding of its inner workings. This blog post will guide you through establishing a Spark Performance Lab with essential tools and techniques aimed at enhancing Spark performance through detailed metrics analysis.

In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features.

TL;DR Explore a step-by-step example of troubleshooting Apache Spark job performance using flame graph visualization and profiling. Discover the seamless integration of Grafana Pyroscope with Spark for streamlined data collection and visualization.

Dive into a comprehensive load-testing exploration using Apache Spark with CPU-intensive workloads.

This blog post is about building a getting-started example for semantic search using vector databases and large language models (LLMs), an example of retrieval augmented generation (RAG) architecture. You can find the accompanying notebook at this link. See also the SWAN gallery.

Are you looking at some resources to get you up to speed with popular Deep Learning and Data processing frameworks? This blog entry provides a curated collection of notebooks that will help you kickstart your journey.

You can find the notebooks at this link. See also the SWAN gallery.

This document describes some basic CPU load testing

Topic: 

We are in a golden age for distributed data processing, with an abundance of tools and solutions emerging from industry and open source. High Energy Physics (HEP) experiments at the LHC stand to profit from all this progress, as they are data-intensive operations with several hundreds of Petabytes of data to collect and process.

In today’s post I will describe the process of integrating OIDC implicit flow with ORDS running on Tomcat against Keycloak service. May sound complicated, but we’ll break it down into individual components so we know what we’re talking about.

TLDR; Apache Spark 3.0 comes with many improvements, including new features for memory monitoring.

Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination. Distributed applications can use it to maintain configuration information, implement naming, provide synchronization and group services.

In today’s post, we’ll be talking about the possible ways to manage the static images/CSS/JS that come shipped with APEX, when running on ORDS. They are separate resources (not contained in the DB like some other APEX images) necessary for your APEX applications look and behave the way they’re intended to. If you and your users browse your internet using lynx (see image below) feel free to skip this one. Otherwise - dig in! 

Creating PDFs in APEX after ORDS 19.1

Until 19.1 ORDS provided a built-in printing engine based on Apache FOP which allowed you to download a PDF version of your reports and XLS-FO templates in a very easy manner. However in ORDS 18.4.0 release notes we could find information that this feature is deprecated and will be removed in future release. This is exactly what happened with the release of ORDS 19.2.

So what actually happened? 

This is from Oracle’s release notes of ORDS 19.2:

Summary: This post details a solution for distributed deep learning training for a High Energy Physics use case, deployed using cloud resources and Kubernetes. You will find the results for training using CPU and GPU nodes. This post also describes an experimental tool that we developed, TF-Spawner, and how we used it to run distributed TensorFlow on a Kubernetes cluster.