Tag: Performance
-
SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads
SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads SparkMeasure SparkMeasure simplifies the collection and analysis of Apache Spark task metrics data. It is also intended as a working example of how to use Spark listeners for collecting and processing Spark performance metrics. The work on sparkMeasure has been previously presented in this blog
-
Performance Analysis of a CPU-Intensive Workload in Apache Spark
Performance Analysis of a CPU-Intensive Workload in Apache Spark Topic: This post is about techniques and tools for measuring and understanding CPU-bound and memory-bound workloads in Apache Spark. You will find examples applied to studying a simple workload consisting of reading Apache Parquet files into a Spark DataFrame. Why are the topics discussed here relevant
-
Diving into Spark and Parquet Workloads, by Example
Diving into Spark and Parquet Workloads, by Example Topic: In this post you can find a few simple examples illustrating important features of Spark when reading partitioned tables stored in Parquet, in particular with a focus on performance investigations. The main topics covered are: Motivations: The combination of Spark and Parquet currently is a very
-
Apache Spark 2.0 Performance Improvements Investigated With Flame Graphs
Apache Spark 2.0 Performance Improvements Investigated With Flame Graphs Topic: This post is about performance optimizations introduced in Apache Spark 2.0, in particular whole-stage code generation. A test case is introduced and investigated with diagnostic tools. Introduction: performance troubleshooting of a slow query using parallel query execution in a Hadoop cluster The idea for this
-
Hadoop performance troubleshooting with stack tracing, an introduction.
Hadoop performance troubleshooting with stack tracing, an introduction. Topic: This post is about profiling and performance tuning of distributed workloads and in particular Hadoop applications. You will learn of a profiler application we have developed and how it has successfully been applied to tuning Sqoop to improve the throughput of data transfer from Oracle to
-
Linux Perf Probes for Oracle Tracing
Linux Perf Probes for Oracle Tracing Topic: this post is about Linux perf and uprobes for tracing and profiling Oracle workloads for advanced troubleshooting. Context The recent progress and maturity of some of the Linux dynamic tracing tools has raised interest in applying these techniques to Oracle troubleshooting and performance investigations. See Brendan Gregg‘s web
-
Extended Stack Profiling – Ideas, Tools and Comments
Extended Stack Profiling – Ideas, Tools and Comments Topic: This post provides a short summary and pointers to previous work on Extended Stack Profiling for troubleshooting and performance investigations. Understanding the workload is an important part of troubleshooting activities. We seek answers to questions like: what is the system doing, where is the time spent,
-
Oracle Wait Events Investigated With Extended Stack Profiling and Flame Graphs
Oracle Wait Events Investigated With Extended Stack Profiling and Flame Graphs Topic: this post is about investigating Oracle wait events using stack profiles and flame graphs extended with OS-process state and Oracle wait event details. Context: The case of the DB Time > CPU Time + Wait Time Oracle instrumentation provides wait event and CPU
-
Linux Kernel Stack Profiling and Flame Graphs Applied to Oracle Investigations
Linux Kernel Stack Profiling and Flame Graphs Applied to Oracle Investigations Topic: This blog post is about kernel stack profiling and visualization with flame graphs. Context: Stack profiling and flame graphs are very useful tools and techniques for troubleshooting and investigating workloads at the OS-level and understand which code path take most of the execution
-
Heat Map Visualization of Latency Histograms for NetApp C-Mode
Heat Map Visualization of Latency Histograms for NetApp C-Mode Topic: This post is about collecting and visualizing I/O latency histograms for NetApp filers in C-mode. Motivations: The drill down of I/O latency is an important technique for troubleshooting and benchmarking storage. Average latency values can hide details of what is happening on the storage. Think