Blaze Log Collection: Overview of Log Aggregation and Troubleshooting

🔥 Blaze architecture consists of major components: Blaze grid manager, orchestrators, service Opie container, and data exchange framework.

📊 Blaze logs are important for log analysis and can be correlated with the task light DTM process and grid manager logs.

💡 Blaze tasks are the basic units of execution and are executed independently on different data nodes before being combined using the orchestrator.

🔥 Blaze logging flow: request sent to orchestrator, containers provided by resource manager, task logs created on data nodes.

📝 Task logs aggregated by Orchestrator and stored in HDFS for access.

⚙️ Blaze grid manager should be stopped when collecting aggregated logs.

🔥 Blaze log collection involves stopping the grid manager and other components, aggregating their logs, and uploading them to HDFS.

📁 The aggregated logs can be found in a specific directory in the HDFS filesystem, organized by application ID and task.

🔍 The job blaze monitor is used to access and analyze the logs during and after task execution.

🔍 The video explains how to view and fix logs using a UI called Blaze Log Collection.

📚 By accessing the task lights, you can see their status, sub mappings, and the data node where they were executed.

💻 The video also demonstrates how to fetch the log of a specific task and access the application log from the resource manager.

📌 The video demonstrates how to access the aggregated log location on the HDFS after stopping the Blaze grid manager.

🔎 Within the log location, there are different directories for service logs and task logs, which provide information about the execution of tasks and the data nodes on which they were executed.

🔍🔢 To troubleshoot issues, it is important to identify the specific DTM process on which a task failed by checking the DTM process log.

📝 The process of collecting blaze log for troubleshooting purposes involves retrieving the grid task log and the DTM process log.

🔍 By analyzing the application log, we can determine which data node the blaze did manager started on and check the logging and priority for each container.

🗺️ The mapping log provides information on the location of the blaze application log, URL of the blaze monitor, and the aggregated log location.

🔍 The video discusses the different types of log collection: live log collection, historical log collection, and aggregated log collection.

🖥️ The one-click log collector is a new feature that allows users to easily collect mapping logs, application logs, and task logs in a zip file, which can be downloaded from the browser.

📚 Starting from version 10.4, users can take advantage of the one-click log collector and find more information in the provided knowledge base articles.