Tag: block
-
Unlocking Apache Spark Performance: Three Open-Source Tools We Use at CERN
Unlocking Apache Spark Performance: Three Open-Source Tools We Use at CERN Apache Spark is incredibly powerful, but anyone who has worked with it long enough knows the feeling: Why is this job suddenly slower today? Why are executors running out of memory? Why is one stage taking 90% of the runtime? What exactly is Spark
-
Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap
Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap I recently chased an Oracle performance issue where most reads were sub-millisecond (cache), but a thin band around ~10 ms (spindles) dominated total wait time. Classic bimodal latency: the fast band looked fine in averages, yet the rare slow band owned the delay. To investigate, and
-
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features. Their latest updates have brought forward significant enhancements, including column indexes, bloom filters. This blog post delves into these new features, exploring their
-
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope TL;DR Explore a step-by-step example of troubleshooting Apache Spark job performance using flame graph visualization and profiling. Discover the seamless integration of Grafana Pyroscope with Spark for streamlined data collection and visualization. The Puzzle of the Slow Query Set within the framework
-
CPU Load Testing Exercises: Tools and Analysis for Oracle Database Servers
CPU Load Testing Exercises: Tools and Analysis for Oracle Database Servers This document describes some basic CPU load testing exercises on three different types of database servers used by the Oracle Service at CERN. It reports on the tests performed, tools used for data gathering, data analysis, findings, and lessons learned. Motivations CPU usage is
-
Apache Spark 3.0 Memory Monitoring Improvements
Apache Spark 3.0 Memory Monitoring Improvements TLDR; Apache Spark 3.0 comes with many improvements, including new features for memory monitoring. This can help you troubleshooting memory usage and optimizing the memory configuration of your Spark jobs for better performance and stability, see SPARK-23429 and SPARK-27189. The problem with memory Memory is key for the performance
-
Oracle REST Data Services running on Tomcat – Basic Authentication using JNDI Realm
Oracle REST Data Services running on Tomcat – Basic Authentication using JNDI Realm What do we want to achieve? We want to protect our REST endpoints using Basic Authentication and authenticate the requests against our users directory (LDAP). We also want to manage the privileges centrally, through the ORDS Roles and Privileges (https://oracle-base.com/articles/misc/oracle-rest-data-services-ords-authentication#ords-roles-and-privileges), so no
-
Evaluation of Erasure Coding in Hadoop 3
Evaluation of Erasure Coding in Hadoop 3 Authored By: Nazerke Seidan, Emil Kleszcz, Zbigniew Baranowski Published By: CERN IT-DB-SAS In this post, we will dive into the evaluation of the Erasure Coding feature of Hadoop 3 that I worked on this summer as a CERN Openlab intern. The evaluation has been performed on one of
-
Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo
Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo Topic: This post describes a data pipeline for a machine learning task of interest in high energy physics: building a particle classifier to improve event selection at the particle detectors. The pipeline is built using tools from the “Big Data
-
Tool to visualise block distribution on Hadoop (HDFS) cluster
Tool to visualise block distribution on Hadoop (HDFS) cluster Distributed systems always bring new challenges for administrators and users. This is the case with HDFS, the default distributed file system that Hadoop uses for storing data. In order to face these challenges, tools to facilitate administration and usage of these systems are developed. At CERN,