I’ve recently noticed an automation job I setup that uses my PowerShell script to convert Azure Firewall logs in JSON format begin to fail on June 5, 2024 at the 23:00 hour. The following is my original post:
Converting Azure Firewall logs in JSON format created from Archive to a storage account diagnostic setting to CSV format
https://terenceluk.blogspot.com/2023/07/converting-azure-firewall-logs-in-json.html
After troubleshooting the error by manually executing the script, I realized that it was failing on the steps used to format content to conform to the JSON specification RFC 8259. Further investigation by opening the 22:00 and 23:00 hour logs show that the “category” key value pair is now placed at the bottom of the log:
This broke my RegEx expression using it as the beginning of the log entry rendering the process of formatting the JSON to conform with RFC 8529 to no longer work. I gave the format a bit of thought and was afraid that this may change again in the future so I updated the PowerShell script’s Step #2 where it adds a comma between log entries to parse each entry using the open bracket to mark the first log entry and going through each line until the ending bracket before inserting the comma. It has slowed the script down quite a bit but would catch any future changes.
A note I’d like to make about the Step #2 string parse and addition of the comma is that I originally used code to write to an accumulator string, which can be slow and intensive because strings are immutable in .NET, meaning that every time I append to a string using +=, a new string is constructed. This meant that the large JSON log files can cause the script to crash due to performance and memory issues so I decided to use .NET’s StringBuilder class that would be faster and more efficient for this kind of string modifications.
The updated script can be found here at my GitHub repo: https://github.com/terenceluk/Azure/blob/main/Azure%20Firewall/Bulk-Convert-Az-Firewall-Logs-JSON-to-CSV_v2.ps1
