Tag: CERN
-
Unlocking Apache Spark Performance: Three Open-Source Tools We Use at CERN
Unlocking Apache Spark Performance: Three Open-Source Tools We Use at CERN Apache Spark is incredibly powerful, but anyone who has worked with it long enough knows the feeling: Why is this job suddenly slower today? Why are executors running out of memory? Why is one stage taking 90% of the runtime? What exactly is Spark
-
CERN PGDay 2026 is here!
CERN PGDay 2026 is here! After a successful first edition in 2025, CERN PGDay returns in 2026 as a regular gathering for PostgreSQL users and enthusiasts in Suisse Romande (western Switzerland). Co-organized by CERN and SwissPUG, the event offers a chance to connect, share ideas, and exchange experiences in the vibrant Geneva region — home
-
Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches)
Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches) TL;DR: Apache Spark 4 lets you build first-class data sources in pure Python. If your reader yields Arrow RecordBatch objects, Spark ingests them with reduced Python↔JVM serialization overhead. I used this to ship a ROOT data format reader for PySpark. A PySpark reader
-
Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap
Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap I recently chased an Oracle performance issue where most reads were sub-millisecond (cache), but a thin band around ~10 ms (spindles) dominated total wait time. Classic bimodal latency: the fast band looked fine in averages, yet the rare slow band owned the delay. To investigate, and
-
ATLAS DCS Analysis with Apache Spark and Jupyter Notebooks
ATLAS DCS Analysis with Apache Spark and Jupyter Notebooks The ATLAS Detector Control System (DCS) at CERN is essential for ensuring optimal detector performance. Each year, the system generates tens of billions of time-stamped sensor readings, presenting considerable challenges for large-scale data analysis. Although these data are stored in Oracle databases that excel in real-time
-
Kepler’s Mars Orbit Analysis with Python Notebooks & AI-Assisted Coding
Kepler’s Mars Orbit Analysis with Python Notebooks & AI-Assisted Coding Johannes Kepler ’s analysis of Mars’ orbit stands as one of the greatest achievements in scientific history, revealing the elliptical nature of planetary paths and establishing the foundational laws of planetary motion. In this post, you will explore how you can recreate Kepler’s revolutionary findings
-
CERN PGDay 2025 is here!
CERN PGDay 2025 is here! CERN PGDay 2025 builds on the experience of past PostgreSQL events at CERN and a newly established collaboration with SwissPUG. It will create an opportunity for PostgreSQL users and enthusiasts to meet in the French-speaking part of Switzerland in order to network and exchange their experiences. We plan to turn
-
Building an Apache Spark Performance Lab: Tools and Techniques for Spark Optimization
Building an Apache Spark Performance Lab: Tools and Techniques for Spark Optimization Apache Spark is renowned for its speed and efficiency in handling large-scale data processing. However, optimizing Spark to achieve maximum performance requires a precise understanding of its inner workings. This blog post will guide you through establishing a Spark Performance Lab with essential
-
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features. Their latest updates have brought forward significant enhancements, including column indexes, bloom filters. This blog post delves into these new features, exploring their
-
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope TL;DR Explore a step-by-step example of troubleshooting Apache Spark job performance using flame graph visualization and profiling. Discover the seamless integration of Grafana Pyroscope with Spark for streamlined data collection and visualization. The Puzzle of the Slow Query Set within the framework