Tag: Java
-
Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches)
Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches) TL;DR: Apache Spark 4 lets you build first-class data sources in pure Python. If your reader yields Arrow RecordBatch objects, Spark ingests them with reduced Python↔JVM serialization overhead. I used this to ship a ROOT data format reader for PySpark. A PySpark reader
-
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters
Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features. Their latest updates have brought forward significant enhancements, including column indexes, bloom filters. This blog post delves into these new features, exploring their
-
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope
Enhancing Apache Spark Performance with Flame Graphs: A Practical Example Using Grafana Pyroscope TL;DR Explore a step-by-step example of troubleshooting Apache Spark job performance using flame graph visualization and profiling. Discover the seamless integration of Grafana Pyroscope with Spark for streamlined data collection and visualization. The Puzzle of the Slow Query Set within the framework
-
Performance Comparison of 5 JDKs on Apache Spark
Performance Comparison of 5 JDKs on Apache Spark Dive into a comprehensive load-testing exploration using Apache Spark with CPU-intensive workloads. This blog provides a comparative analysis of five distinct JDKs’ performance under heavy-duty tasks generated through Spark. Discover a meticulous breakdown of our testing methodology, tools, and insightful results. Keep in mind, our observations primarily
-
Distributed application cache for Kubernetes running Java Hibernate applications with Oracle Coherence Community Edition
Distributed application cache for Kubernetes running Java Hibernate applications with Oracle Coherence Community Edition Introduction While working on a data set it is important that it stays easily and quickly accessible. Hibernate second-level caching with Coherence offers applications a resource optimized solution that keeps frequently used data in memory, by distributing it among different application
-
Apache Spark 3.0 Memory Monitoring Improvements
Apache Spark 3.0 Memory Monitoring Improvements TLDR; Apache Spark 3.0 comes with many improvements, including new features for memory monitoring. This can help you troubleshooting memory usage and optimizing the memory configuration of your Spark jobs for better performance and stability, see SPARK-23429 and SPARK-27189. The problem with memory Memory is key for the performance
-
Building and documenting REST APIs with ORDS
Building and documenting REST APIs with ORDS Introduction In the first part of the article we will provide an overview of how you can use Oracle REST Data Services for providing APIs directly from your PL/SQL code . The second part covers how to document our Web services using Swagger. Lets begin with a couple
-
Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo
Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo Topic: This post describes a data pipeline for a machine learning task of interest in high energy physics: building a particle classifier to improve event selection at the particle detectors. The pipeline is built using tools from the “Big Data
-
Keeping your logs clean with Apache Tomcat 9, log4j2 and spring-boot
Keeping your logs clean with Apache Tomcat 9, log4j2 and spring-boot These last days I’ve been playing with Apache Tomcat 9 and log4j2. You may wonder why I want to change the good an old Tomcat JULI. Well, although I love the simplicity of its configuration and the fact that JULI works, log4j2 offers a
-
Testing web applications SSO with Keycloak
Testing web applications SSO with Keycloak Usually when you are developing a new feature or fixing an issue, you want to focus in your business logic. If your application delegates the authentication in some SSO system you usually mocks the response from this last one. However for integration tests, it is nice to be able