A DevOps Approach to AEM Packages: Automating Creation, Configuration, and More

Adobe Experience Manager (AEM) packages are the unsung heroes of content management — powerful containers that bundle everything from code and configurations to critical content. But let’s face it: manually creating, configuring, and downloading these packages can feel like a tedious dance of clicks. What if you could automate this process with a few keystrokes, ensuring consistency, speed, reliability, and less heavy lifting? I will show you a Bash script that flips the script (pun intended!) on how AEM developers and admins work with the Package Manager API. Think about crafting packages in seconds, tailoring filters on the fly, and snagging backups with surgical precision — all before your coffee cools to that perfect sipping temperature. ☕ Before we dive in, a quick note: This article is a deep dive, meticulously detailed and unapologetically technical. We’ll dissect the script’s logic, explore AEM API intricacies, and troubleshoot edge cases. For developers eager to jump straight into the code, you can jump to the bottom of the article. But if you’re here to understand the how and why behind the automation, strap in — we’re going all the way down the rabbit hole. 🕳️ 1. Script Overview The create-remote-aem-pkg.sh script automates interactions with AEM’s Package Manager API, offering a structured approach to package creation, configuration, and distribution. Designed for developers and administrators, it replaces manual workflows with a command-line-driven process that emphasizes consistency and reliability. 1.1 Core Functionalities Package Validation: Checks for existing packages to avoid redundancy before initiating creation. Dynamic Filter Injection: Programmatically defines content paths (e.g., /content/dam, /apps) to include in the package. Build Automation: Triggers package compilation and downloads the output to a specified directory, appending a timestamp to filenames for version control. Error Handling: Validates HTTP responses, folder paths, and authentication to provide actionable feedback during failures. Authentication: Supports basic¹ credential-based authentication via curl. 1.2 Key Benefits Efficiency: Reduces manual steps required for package creation, configuration, and download. Consistency: Ensures uniform package structures and naming conventions across environments. Traceability: Detailed logging at each stage (creation, filtering, building, downloading) aids in auditing and troubleshooting. 1.3 Practical Applications Scheduled Backups: Integrate with cron jobs to regularly archive critical content paths. Environment Synchronization: Replicate configurations or content between AEM instances during deployments. Pre-Update Snapshots: Capture stable states of /etc or /apps before applying system updates. 1.4 Prerequisites Access to an AEM instance (credentials, server, port). Basic familiarity with Bash scripting and AEM’s Package Manager. Permission to create and download packages via the AEM API. 1.5 Example Usage ./create-remote-aem-pkg.sh admin securepass123 localhost 4502 backup-group "Content Backup" /backups /content/dam /etc/clientlibs This command creates a package named “Content Backup” under the group backup-group, including /content/dam and /etc/clientlibs, and saves the output to the /backups directory. 2. Script Breakdown Let’s dissect the create-remote-aem-pkg.sh script (you can find it at the bottom of the article) to understand how it orchestrates AEM package management. We’ll focus on its structure, key functions, and workflow logic—ideal for developers looking to customize or debug the tool. 2.1 Core Functions _log(): A utility function that prefixes messages with timestamps for clear audit trails. _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } Why it matters: Ensures every action (e.g., “Package built”) is logged with context, simplifying troubleshooting. check_last_exec(): Validates the success of prior commands by checking exit codes and API responses. check_last_exec () { # Checks $? (exit status) and $CURL_OUTPUT for errors if [ "$status" -ne 0 ] || [[ $output =~ .*success\":false* ]]; then _log "Error detected!"; exit 1; fi } Why it matters: Prevents silent failures by halting execution on critical errors like authentication issues or invalid paths. 2.2 Input Parameters The script accepts seven positional arguments followed by dynamic filters: USR="$1" # AEM username PWD="$2" # AEM password SVR="$3" # Server host (e.g., localhost) PORT="$4" # Port (e.g., 4502) PKG_GROUP="$5" # Package group (e.g., "backups") PKG_NAME="$6" # Package name (e.g., "dam-backup") BK_FOLDER="$7" # Backup directory (e.g., "/backups") shift 7 # Remaining arguments become filters (e.g., "/content/dam") Positional arguments ensure simplicity, while shift handles variable filter paths flexibly. 2.3 Package Validation & Creation Sanitize Names: Replaces spaces in PKG_NAME with underscores to avoid URL issues. PKG_NAME=${PKG_NAME// /_} Check Existing Packages: Uses curl to list packages via AEM’s API, avoiding redundant creations. if [ $(curl ... | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log "Package exists—skipping creation." else curl -X POST ... # Creates the package fi 2.4 Dynamic Filter Configuration Constructs a JSON array of filters from input paths: FILTERS_PARAM="" for i in "${!FILTERS[@]}"; do FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" # Adds commas between entries, but not after the last done Example output: [{"root": "/content/dam"}, {"root": "/apps"}] This JSON is injected into the package definition via AEM’s /crx/packmgr/update.jsp endpoint. 2.5 Build & Download Workflow Build the Package: Triggers compilation using AEM’s build command: curl -X POST … -F "cmd=build" Note: The script waits for the build to complete before proceeding. Download: Uses curl to fetch the .zip and save it with a timestamped filename: BK_FILE="$PKG_NAME-$(date +%Y%m%d-%H%M%S).zip" curl -o "$BK_FOLDER/$BK_FILE" ... 3. Error Handling, Security Notes, & Logging Robust error handling and logging are critical for unattended scripts like create-remote-aem-pkg.sh, ensuring failures are caught early and logged clearly. Here’s how the script safeguards against unexpected issues and provides actionable insights. 3.1 Logging Mechanism Timestamped Logs: The _log function prefixes every message with a [YYYY.MM.DD-HH:MM:SS] timestamp, creating an audit trail for debugging: _log "Starting backup process..." # Output: [2023.10.25-14:30:45] Starting backup process... Why it matters: Timestamps help correlate script activity with AEM server logs or external events (e.g., cron job schedules). Verbose Output: Critical steps, like package creation, filter updates, and downloads, are explicitly logged to track progress. 3.2 Error Validation Workflow Pre-Flight Checks: Validates the existence of the backup folder (BK_FOLDER) before proceeding: if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && exit 1 fi Sanitizes PKG_NAME to avoid URL issues (e.g., spaces replaced with underscores). API Response Validation: The check_last_exec function examines both shell exit codes ($?) and AEM API responses: check_last_exec "Error message" "$CURL_OUTPUT" $CURL_STATUS Exit Codes: Non-zero values (e.g., curl network failures) trigger immediate exits. API Errors: Detects success\":false JSON responses or "HTTP ERROR" strings in AEM output. 3.3 HTTP Status Verification: When downloading the package, the script checks for a 200 status code: if [ "$(curl -w "%{http_code}" ...)" -eq "200" ]; then # Proceed if download succeeds else _log "Error downloading the package!" && exit 1 fi 3.4 Common Failure Scenarios Invalid credentials: check_last_exec catches 401 Unauthorized responses and exits with a clear error message. Invalid filter path: AEM API returns success:false, the script logs "Error adding filters" and terminates. Disk full: Fails to write BK_FILE, checks file size with -s flag and alerts before exiting. AEM instance unreachable:curl exits with a non-zero code, the script logs "Error building the package". 3.5 Security Considerations SSL Certificate Bypass: The script uses curl -k for simplicity, which skips SSL verification. Recommendation for Production: Replace with --cacert to specify a CA bundle. Plaintext Passwords: Credentials are passed as arguments, which may appear in process logs. Mitigation: Use environment variables or a secrets vault (e.g., $AEM_PASSWORD). 3.6 Debugging Tips Enable Verbose Output: Temporarily add set -x at the script’s start to print executed commands. Test API Calls Manually: Isolate issues by running critical curl commands outside the script Inspect Logs: Redirect script output to a file for later analysis: ./create-remote-aem-pkg.sh ... >> /var/log/aem_backup.log 2>&1 4. Tailoring the Tool to Your Workflow The create-remote-aem-pkg.sh script is designed to be a starting point—a foundation you can modify to align with your team’s needs. Below are common customizations, along with implementation guidance, to extend its functionality or adapt it to specific use cases. 4.1 Adjusting the Backup Filename Format The default filename uses a timestamp ($PKG_NAME-$(date +%Y%m%d-%H%M%S).zip). Modify this to include environment names, project IDs, or semantic versioning: # Example: Include environment (e.g., "dev", "prod") BK_FILE="${PKG_NAME}-${ENV}-$(date +%Y%m%d).zip" # Example: Add Git commit SHA for traceability COMMIT_SHA=$(git rev-parse --short HEAD) BK_FILE="${PKG_NAME}-${COMMIT_SHA}.zip" Tip: Ensure date/time formats avoid characters forbidden in filenames (e.g., colons : on Windows). 4.2 Expanding or Modifying Filters The script accepts dynamic paths as filters but you can also hardcode frequently used paths or add exclusions: # Hardcode essential paths (e.g., "/var/audit") DEFAULT_FILTERS=("/content/dam" "/apps" "/var/audit") FILTERS=("${DEFAULT_FILTERS[@]}" "${@}") # Merge with command-line inputs # Add exclusion rules (requires AEM API support) FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": [{\"modifier\": \"exclude\", \"pattern\": \".*/test/*\"}]}" 4.3 Enhancing Security Avoid Plaintext Passwords: Use environment variables or a secrets manager to inject credentials: # Fetch password from environment variable PWD="$AEM_PASSWORD" # Use AWS Secrets Manager (example) PWD=$(aws secretsmanager get-secret-value --secret-id aem/prod/password --query SecretString --output text) Enforce SSL Validation:Replacecurl -k (insecure) with a trusted CA certificate: curl --cacert /path/to/ca-bundle.crt -u "$USR":"$PWD" ... 4.4 Adding Post-Build Actions Extend the script to trigger downstream processes after a successful download: # Example: Upload to cloud storage aws s3 cp "$BK_FOLDER/$BK_FILE" s3://my-backup-bucket/ # Example: Validate package integrity CHECKSUM=$(sha256sum "$BK_FOLDER/$BK_FILE" | cut -d ' ' -f 1) _log "SHA-256 checksum: $CHECKSUM" # Example: Clean up old backups (retain last 7 days) find "$BK_FOLDER" -name "*.zip" -mtime +7 -exec rm {} \; 4.5 Adding Notification Alerts Notify teams of success/failure via Slack, email, or monitoring tools: # Post to Slack on failure curl -X POST -H 'Content-type: application/json' \ --data "{\"text\":\"🚨 AEM backup failed: $(hostname)\"}" \ https://hooks.slack.com/services/YOUR/WEBHOOK/URL # Send email via sendmail if [ $? -ne 0 ]; then echo "Subject: Backup Failed" | sendmail admin@mycompany.com fi 5. Conclusion Managing AEM packages doesn’t have to be a manual, error-prone chore. With the create-remote-aem-pkg.sh script, you can transform package creation, filtering, and distribution into a streamlined, repeatable process. This tool isn’t just about saving time, it’s about enabling consistency, reliability, and scalability in your AEM operations. Key Takeaways Automation Wins: By eliminating repetitive GUI interactions, the script reduces human error and frees teams to focus on higher-value tasks. Flexibility Matters: Whether backing up critical content, syncing environments, or preparing for updates, the script adapts to diverse use cases with minimal tweaking. Resilience is Key: Built-in logging, error checks, and security considerations ensure the script behaves predictably, even when things go sideways. Great tools are born from real-world challenges. This script is a starting point; think of it as a foundation to build upon as your team’s needs grow. Whether you’re a solo developer or part of a large DevOps team, automation like this exemplifies how small investments in code can yield outsized returns in productivity and peace of mind. Ready to take the next step? 🛠️ Customize: Tailor the script using Section 6 as your guide. 🔍 Audit: Review your existing AEM workflows for automation opportunities. 🤝 Share: Mentor your team or write a blog post about your modifications. Thank you for following along — now go forth and automate! 🚀 Appendix Complete Code #!/bin/bash set -eo pipefail # The script will create a package thought the package manager api: # - The package is created, if not already present # - Package filters are populated accordingly to specified paths # - Package is builded # - Package is download to the specified folder _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } check_last_exec () { local message="$1" local output="$2" local status=$3 if [ "$status" -ne 0 ]; then echo && echo "$message" && echo exit 1 fi if [[ $output =~ .*success\":false* ]] || [[ $output =~ .*"HTTP ERROR"* ]]; then _log "$message" exit 1 fi } USR="$1" PWD="$2" SVR="$3" PORT="$4" PKG_GROUP="$5" PKG_NAME="$6" BK_FOLDER="$7" shift 7 # The following paths will be included in the package FILTERS=($@) BK_FILE=$PKG_NAME"-"$(date +%Y%m%d-%H%M%S).zip _log "Starting backup process..." echo "AEM instance: '$SVR':'$PORT' AEM User: '$USR' Package group: $PKG_GROUP Package name: '$PKG_NAME' Destination folder: $BK_FOLDER Destination file: '$BK_FILE' Filter paths: " printf '\t%s\n\n' "${FILTERS[@]}" if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && echo exit 1 fi PKG_NAME=${PKG_NAME// /_} check_last_exec "Error replacing white space chars from package name!" "" $? || exit 1 _log "Removed whitespaces from package name: '$PKG_NAME'" BK_FILE=$PKG_NAME.zip _log "Backup file: '$BK_FILE'" _log "Creating the package..." if [ $(curl -k -u "$USR":"$PWD" "$SVR:$PORT/crx/packmgr/service.jsp?cmd=ls" 2>/dev/null | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log " Package '$PKG_GROUP/$PKG_NAME' is already present: skipping creation." else curl -k --silent -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/.json/etc/packages/$PKG_GROUP/$PKG_NAME?cmd=create" \ -d packageName="$PKG_NAME" -d groupName="$PKG_GROUP" check_last_exec " Error creating the package!" "" $? _log " Package created" fi # create filters variable FILTERS_PARAM="" ARR_LEN="${#FILTERS[@]}" for i in "${!FILTERS[@]}"; do FILTERS_PARAM=$FILTERS_PARAM"{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" T=$((i+1)) if [ $T -ne $ARR_LEN ]; then FILTERS_PARAM=$FILTERS_PARAM", " fi done # add filters _log "Adding filters to the package..." CURL_OUTPUT=$(curl -k --silent -u "$USR":"$PWD" -X POST "$SVR:$PORT/crx/packmgr/update.jsp" \ -F path=/etc/packages/"$PKG_GROUP"/"$PKG_NAME".zip -F packageName="$PKG_NAME" \ -F groupName="$PKG_GROUP" \ -F filter="[$FILTERS_PARAM]" \ -F "_charset_=UTF-8") CURL_STATUS=$? # Pass the status to the check_last_exec function check_last_exec "Error adding filters to the package!" "$CURL_OUTPUT" $CURL_STATUS _log " Package filters updated successfully." # build package _log "Building the package..." CURL_OUTPUT=$(curl -k -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/script.html/etc/packages/$PKG_GROUP/$PKG_NAME.zip" \ -F "cmd=build") check_last_exec " Error building the package!" "$CURL_OUTPUT" $? _log " Package built." # download package _log "Downloading the package..." if [ "$(curl -w "%{http_code}" -o "$BK_FOLDER/$BK_FILE" -k --silent -u "$USR":"$PWD" "$SVR:$PORT/etc/packages/$PKG_GROUP/$PKG_NAME.zip")" -eq "200" ]; then if [ -f "$BK_FOLDER/$BK_FILE" ] && [ -s "$BK_FOLDER/$BK_FILE" ]; then _log " Package $BK_FILE downloaded in $BK_FOLDER." exit 0 fi fi _log " Error downloading the package!" exit 1 References [¹] Skipping SSL verification with curl -k is handy for testing, but you’ll want something sturdier in production (for example --cacert)! [²] AEM Package Manager Official Documentation Adobe Experience Manager (AEM) packages are the unsung heroes of content management — powerful containers that bundle everything from code and configurations to critical content. But let’s face it: manually creating, configuring, and downloading these packages can feel like a tedious dance of clicks. What if you could automate this process with a few keystrokes, ensuring consistency, speed, reliability, and less heavy lifting? I will show you a Bash script that flips the script (pun intended!) on how AEM developers and admins work with the Package Manager API. Think about crafting packages in seconds, tailoring filters on the fly, and snagging backups with surgical precision — all before your coffee cools to that perfect sipping temperature. ☕ perfect Before we dive in, a quick note: This article is a deep dive, meticulously detailed and unapologetically technical. We’ll dissect the script’s logic, explore AEM API intricacies, and troubleshoot edge cases. For developers eager to jump straight into the code, you can jump to the bottom of the article. But if you’re here to understand the how and why behind the automation, strap in — we’re going all the way down the rabbit hole. 🕳️ Before we dive in, a quick note : This article is a deep dive, meticulously detailed and unapologetically technical. We’ll dissect the script’s logic, explore AEM API intricacies, and troubleshoot edge cases. For developers eager to jump straight into the code, you can jump to the bottom of the article. But if you’re here to understand the how and why behind the automation, strap in — we’re going all the way down the rabbit hole. 🕳️ Before we dive in, a quick note Before we dive in, a quick note This article is a deep dive, deep dive, meticulously detailed and unapologetically technical. We’ll dissect the script’s logic, explore AEM API intricacies, and troubleshoot edge cases. For developers eager to jump straight into the code, you can jump to the bottom of the article. But if you’re here to understand the how and why behind the automation, strap in — we’re going all the way down the rabbit hole. 1. Script Overview The create-remote-aem-pkg.sh script automates interactions with AEM’s Package Manager API, offering a structured approach to package creation, configuration, and distribution. Designed for developers and administrators, it replaces manual workflows with a command-line-driven process that emphasizes consistency and reliability. create-remote-aem-pkg.sh 1.1 Core Functionalities Package Validation: Checks for existing packages to avoid redundancy before initiating creation. Dynamic Filter Injection: Programmatically defines content paths (e.g., /content/dam, /apps) to include in the package. Build Automation: Triggers package compilation and downloads the output to a specified directory, appending a timestamp to filenames for version control. Error Handling: Validates HTTP responses, folder paths, and authentication to provide actionable feedback during failures. Authentication: Supports basic¹ credential-based authentication via curl. Package Validation : Checks for existing packages to avoid redundancy before initiating creation. Package Validation Dynamic Filter Injection : Programmatically defines content paths (e.g., /content/dam , /apps ) to include in the package. Dynamic Filter Injection /content/dam /apps Build Automation : Triggers package compilation and downloads the output to a specified directory, appending a timestamp to filenames for version control. Build Automation Error Handling : Validates HTTP responses, folder paths, and authentication to provide actionable feedback during failures. Error Handling Authentication : Supports basic¹ credential-based authentication via curl . Authentication curl 1.2 Key Benefits Efficiency: Reduces manual steps required for package creation, configuration, and download. Consistency: Ensures uniform package structures and naming conventions across environments. Traceability: Detailed logging at each stage (creation, filtering, building, downloading) aids in auditing and troubleshooting. Efficiency : Reduces manual steps required for package creation, configuration, and download. Efficiency Consistency : Ensures uniform package structures and naming conventions across environments. Consistency Traceability : Detailed logging at each stage (creation, filtering, building, downloading) aids in auditing and troubleshooting. Traceability 1.3 Practical Applications Scheduled Backups: Integrate with cron jobs to regularly archive critical content paths. Environment Synchronization: Replicate configurations or content between AEM instances during deployments. Pre-Update Snapshots: Capture stable states of /etc or /apps before applying system updates. Scheduled Backups : Integrate with cron jobs to regularly archive critical content paths. Scheduled Backups Environment Synchronization : Replicate configurations or content between AEM instances during deployments. Environment Synchronization Pre-Update Snapshots : Capture stable states of /etc or /apps before applying system updates. Pre-Update Snapshots /etc /apps 1.4 Prerequisites Access to an AEM instance (credentials, server, port). Basic familiarity with Bash scripting and AEM’s Package Manager. Permission to create and download packages via the AEM API. Access to an AEM instance (credentials, server, port). Basic familiarity with Bash scripting and AEM’s Package Manager. Permission to create and download packages via the AEM API. 1.5 Example Usage ./create-remote-aem-pkg.sh admin securepass123 localhost 4502 backup-group "Content Backup" /backups /content/dam /etc/clientlibs ./create-remote-aem-pkg.sh admin securepass123 localhost 4502 backup-group "Content Backup" /backups /content/dam /etc/clientlibs This command creates a package named “Content Backup” under the group backup-group , including /content/dam and /etc/clientlibs , and saves the output to the /backups directory. backup-group /content/dam /etc/clientlibs /backups 2. Script Breakdown Let’s dissect the create-remote-aem-pkg.sh script (you can find it at the bottom of the article) to understand how it orchestrates AEM package management. We’ll focus on its structure, key functions, and workflow logic—ideal for developers looking to customize or debug the tool. create-remote-aem-pkg.sh 2.1 Core Functions _log(): A utility function that prefixes messages with timestamps for clear audit trails. _log() : A utility function that prefixes messages with timestamps for clear audit trails. _log() _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } Why it matters : Ensures every action (e.g., “Package built”) is logged with context, simplifying troubleshooting. Why it matters check_last_exec(): Validates the success of prior commands by checking exit codes and API responses. check_last_exec() : Validates the success of prior commands by checking exit codes and API responses. check_last_exec() check_last_exec () { # Checks $? (exit status) and $CURL_OUTPUT for errors if [ "$status" -ne 0 ] || [[ $output =~ .*success\":false* ]]; then _log "Error detected!"; exit 1; fi } check_last_exec () { # Checks $? (exit status) and $CURL_OUTPUT for errors if [ "$status" -ne 0 ] || [[ $output =~ .*success\":false* ]]; then _log "Error detected!"; exit 1; fi } Why it matters : Prevents silent failures by halting execution on critical errors like authentication issues or invalid paths. Why it matters 2.2 Input Parameters The script accepts seven positional arguments followed by dynamic filters: USR="$1" # AEM username PWD="$2" # AEM password SVR="$3" # Server host (e.g., localhost) PORT="$4" # Port (e.g., 4502) PKG_GROUP="$5" # Package group (e.g., "backups") PKG_NAME="$6" # Package name (e.g., "dam-backup") BK_FOLDER="$7" # Backup directory (e.g., "/backups") shift 7 # Remaining arguments become filters (e.g., "/content/dam") USR="$1" # AEM username PWD="$2" # AEM password SVR="$3" # Server host (e.g., localhost) PORT="$4" # Port (e.g., 4502) PKG_GROUP="$5" # Package group (e.g., "backups") PKG_NAME="$6" # Package name (e.g., "dam-backup") BK_FOLDER="$7" # Backup directory (e.g., "/backups") shift 7 # Remaining arguments become filters (e.g., "/content/dam") Positional arguments ensure simplicity, while shift handles variable filter paths flexibly. shift 2.3 Package Validation & Creation Sanitize Names: Replaces spaces in PKG_NAME with underscores to avoid URL issues. Sanitize Names : Replaces spaces in PKG_NAME with underscores to avoid URL issues. Sanitize Names PKG_NAME PKG_NAME=${PKG_NAME// /_} PKG_NAME=${PKG_NAME// /_} Check Existing Packages: Uses curl to list packages via AEM’s API, avoiding redundant creations. Check Existing Packages : Uses curl to list packages via AEM’s API, avoiding redundant creations. Check Existing Packages curl if [ $(curl ... | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log "Package exists—skipping creation." else curl -X POST ... # Creates the package fi if [ $(curl ... | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log "Package exists—skipping creation." else curl -X POST ... # Creates the package fi 2.4 Dynamic Filter Configuration Constructs a JSON array of filters from input paths: FILTERS_PARAM="" for i in "${!FILTERS[@]}"; do FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" # Adds commas between entries, but not after the last done FILTERS_PARAM="" for i in "${!FILTERS[@]}"; do FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" # Adds commas between entries, but not after the last done Example output : Example output [{"root": "/content/dam"}, {"root": "/apps"}] [{"root": "/content/dam"}, {"root": "/apps"}] This JSON is injected into the package definition via AEM’s /crx/packmgr/update.jsp endpoint. /crx/packmgr/update.jsp 2.5 Build & Download Workflow Build the Package: Triggers compilation using AEM’s build command: Build the Package : Triggers compilation using AEM’s build command: Build the Package build curl -X POST … -F "cmd=build" curl -X POST … -F "cmd=build" Note : The script waits for the build to complete before proceeding. Note Download: Uses curl to fetch the .zip and save it with a timestamped filename: Download : Uses curl to fetch the .zip and save it with a timestamped filename: Download curl .zip BK_FILE="$PKG_NAME-$(date +%Y%m%d-%H%M%S).zip" curl -o "$BK_FOLDER/$BK_FILE" ... BK_FILE="$PKG_NAME-$(date +%Y%m%d-%H%M%S).zip" curl -o "$BK_FOLDER/$BK_FILE" ... 3. Error Handling, Security Notes, & Logging Robust error handling and logging are critical for unattended scripts like create-remote-aem-pkg.sh , ensuring failures are caught early and logged clearly. Here’s how the script safeguards against unexpected issues and provides actionable insights. create-remote-aem-pkg.sh 3.1 Logging Mechanism Timestamped Logs: The _log function prefixes every message with a [YYYY.MM.DD-HH:MM:SS] timestamp, creating an audit trail for debugging: Timestamped Logs : The _log function prefixes every message with a [YYYY.MM.DD-HH:MM:SS] timestamp, creating an audit trail for debugging: Timestamped Logs _log [YYYY.MM.DD-HH:MM:SS] _log "Starting backup process..." # Output: [2023.10.25-14:30:45] Starting backup process... _log "Starting backup process..." # Output: [2023.10.25-14:30:45] Starting backup process... Why it matters : Timestamps help correlate script activity with AEM server logs or external events (e.g., cron job schedules). Why it matters Verbose Output: Critical steps, like package creation, filter updates, and downloads, are explicitly logged to track progress. Verbose Output : Critical steps, like package creation, filter updates, and downloads, are explicitly logged to track progress. Verbose Output 3.2 Error Validation Workflow Pre-Flight Checks : Pre-Flight Checks Validates the existence of the backup folder (BK_FOLDER) before proceeding: Validates the existence of the backup folder ( BK_FOLDER ) before proceeding: BK_FOLDER if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && exit 1 fi if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && exit 1 fi Sanitizes PKG_NAME to avoid URL issues (e.g., spaces replaced with underscores). Sanitizes PKG_NAME to avoid URL issues (e.g., spaces replaced with underscores). PKG_NAME API Response Validation : API Response Validation The check_last_exec function examines both shell exit codes ( $? ) and AEM API responses: check_last_exec $? check_last_exec "Error message" "$CURL_OUTPUT" $CURL_STATUS check_last_exec "Error message" "$CURL_OUTPUT" $CURL_STATUS Exit Codes: Non-zero values (e.g., curl network failures) trigger immediate exits. Exit Codes : Non-zero values (e.g., curl network failures) trigger immediate exits. Exit Codes curl API Errors: Detects success\":false JSON responses or "HTTP ERROR" strings in AEM output. API Errors : Detects success\":false JSON responses or "HTTP ERROR" strings in AEM output. API Errors success\":false 3.3 HTTP Status Verification : When downloading the package, the script checks for a 200 status code: 3.3 HTTP Status Verification 200 if [ "$(curl -w "%{http_code}" ...)" -eq "200" ]; then # Proceed if download succeeds else _log "Error downloading the package!" && exit 1 fi if [ "$(curl -w "%{http_code}" ...)" -eq "200" ]; then # Proceed if download succeeds else _log "Error downloading the package!" && exit 1 fi 3.4 Common Failure Scenarios Invalid credentials: check_last_exec catches 401 Unauthorized responses and exits with a clear error message. Invalid filter path: AEM API returns success:false, the script logs "Error adding filters" and terminates. Disk full: Fails to write BK_FILE, checks file size with -s flag and alerts before exiting. AEM instance unreachable:curl exits with a non-zero code, the script logs "Error building the package". Invalid credentials: check_last_exec catches 401 Unauthorized responses and exits with a clear error message. check_last_exec 401 Unauthorized Invalid filter path: AEM API returns success:false , the script logs "Error adding filters" and terminates. success:false Disk full: Fails to write BK_FILE , checks file size with -s flag and alerts before exiting. BK_FILE -s AEM instance unreachable: curl exits with a non-zero code, the script logs "Error building the package". curl 3.5 Security Considerations SSL Certificate Bypass: The script uses curl -k for simplicity, which skips SSL verification. Recommendation for Production: Replace with --cacert to specify a CA bundle. SSL Certificate Bypass : The script uses curl -k for simplicity, which skips SSL verification. Recommendation for Production : Replace with --cacert to specify a CA bundle. SSL Certificate Bypass curl -k Recommendation for Production --cacert Plaintext Passwords: Credentials are passed as arguments, which may appear in process logs. Mitigation: Use environment variables or a secrets vault (e.g., $AEM_PASSWORD). Plaintext Passwords : Credentials are passed as arguments, which may appear in process logs. Mitigation : Use environment variables or a secrets vault (e.g., $AEM_PASSWORD ). Plaintext Passwords Mitigation $AEM_PASSWORD 3.6 Debugging Tips Enable Verbose Output: Temporarily add set -x at the script’s start to print executed commands. Test API Calls Manually: Isolate issues by running critical curl commands outside the script Inspect Logs: Redirect script output to a file for later analysis: Enable Verbose Output : Temporarily add set -x at the script’s start to print executed commands. Enable Verbose Output set -x Test API Calls Manually : Isolate issues by running critical curl commands outside the script Test API Calls Manually curl Inspect Logs : Redirect script output to a file for later analysis: Inspect Logs ./create-remote-aem-pkg.sh ... >> /var/log/aem_backup.log 2>&1 ./create-remote-aem-pkg.sh ... >> /var/log/aem_backup.log 2>&1 4. Tailoring the Tool to Your Workflow 4. Tailoring the Tool to Your Workflow The create-remote-aem-pkg.sh script is designed to be a starting point—a foundation you can modify to align with your team’s needs. Below are common customizations, along with implementation guidance, to extend its functionality or adapt it to specific use cases. create-remote-aem-pkg.sh 4.1 Adjusting the Backup Filename Format The default filename uses a timestamp ( $PKG_NAME-$(date +%Y%m%d-%H%M%S).zip ). Modify this to include environment names, project IDs, or semantic versioning: $PKG_NAME-$(date +%Y%m%d-%H%M%S).zip # Example: Include environment (e.g., "dev", "prod") BK_FILE="${PKG_NAME}-${ENV}-$(date +%Y%m%d).zip" # Example: Add Git commit SHA for traceability COMMIT_SHA=$(git rev-parse --short HEAD) BK_FILE="${PKG_NAME}-${COMMIT_SHA}.zip" # Example: Include environment (e.g., "dev", "prod") BK_FILE="${PKG_NAME}-${ENV}-$(date +%Y%m%d).zip" # Example: Add Git commit SHA for traceability COMMIT_SHA=$(git rev-parse --short HEAD) BK_FILE="${PKG_NAME}-${COMMIT_SHA}.zip" Tip : Ensure date/time formats avoid characters forbidden in filenames (e.g., colons : on Windows). Tip : 4.2 Expanding or Modifying Filters The script accepts dynamic paths as filters but you can also hardcode frequently used paths or add exclusions: # Hardcode essential paths (e.g., "/var/audit") DEFAULT_FILTERS=("/content/dam" "/apps" "/var/audit") FILTERS=("${DEFAULT_FILTERS[@]}" "${@}") # Merge with command-line inputs # Add exclusion rules (requires AEM API support) FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": [{\"modifier\": \"exclude\", \"pattern\": \".*/test/*\"}]}" # Hardcode essential paths (e.g., "/var/audit") DEFAULT_FILTERS=("/content/dam" "/apps" "/var/audit") FILTERS=("${DEFAULT_FILTERS[@]}" "${@}") # Merge with command-line inputs # Add exclusion rules (requires AEM API support) FILTERS_PARAM+="{\"root\": \"${FILTERS[$i]}\", \"rules\": [{\"modifier\": \"exclude\", \"pattern\": \".*/test/*\"}]}" 4.3 Enhancing Security Avoid Plaintext Passwords : Avoid Plaintext Passwords Use environment variables or a secrets manager to inject credentials: # Fetch password from environment variable PWD="$AEM_PASSWORD" # Use AWS Secrets Manager (example) PWD=$(aws secretsmanager get-secret-value --secret-id aem/prod/password --query SecretString --output text) # Fetch password from environment variable PWD="$AEM_PASSWORD" # Use AWS Secrets Manager (example) PWD=$(aws secretsmanager get-secret-value --secret-id aem/prod/password --query SecretString --output text) Enforce SSL Validation : Replace curl -k (insecure) with a trusted CA certificate: Enforce SSL Validation curl -k curl --cacert /path/to/ca-bundle.crt -u "$USR":"$PWD" ... curl --cacert /path/to/ca-bundle.crt -u "$USR":"$PWD" ... 4.4 Adding Post-Build Actions Extend the script to trigger downstream processes after a successful download: # Example: Upload to cloud storage aws s3 cp "$BK_FOLDER/$BK_FILE" s3://my-backup-bucket/ # Example: Validate package integrity CHECKSUM=$(sha256sum "$BK_FOLDER/$BK_FILE" | cut -d ' ' -f 1) _log "SHA-256 checksum: $CHECKSUM" # Example: Clean up old backups (retain last 7 days) find "$BK_FOLDER" -name "*.zip" -mtime +7 -exec rm {} \; # Example: Upload to cloud storage aws s3 cp "$BK_FOLDER/$BK_FILE" s3://my-backup-bucket/ # Example: Validate package integrity CHECKSUM=$(sha256sum "$BK_FOLDER/$BK_FILE" | cut -d ' ' -f 1) _log "SHA-256 checksum: $CHECKSUM" # Example: Clean up old backups (retain last 7 days) find "$BK_FOLDER" -name "*.zip" -mtime +7 -exec rm {} \; 4.5 Adding Notification Alerts Notify teams of success/failure via Slack, email, or monitoring tools: # Post to Slack on failure curl -X POST -H 'Content-type: application/json' \ --data "{\"text\":\"🚨 AEM backup failed: $(hostname)\"}" \ https://hooks.slack.com/services/YOUR/WEBHOOK/URL # Send email via sendmail if [ $? -ne 0 ]; then echo "Subject: Backup Failed" | sendmail admin@mycompany.com fi # Post to Slack on failure curl -X POST -H 'Content-type: application/json' \ --data "{\"text\":\"🚨 AEM backup failed: $(hostname)\"}" \ https://hooks.slack.com/services/YOUR/WEBHOOK/URL # Send email via sendmail if [ $? -ne 0 ]; then echo "Subject: Backup Failed" | sendmail admin@mycompany.com fi 5. Conclusion Managing AEM packages doesn’t have to be a manual, error-prone chore. With the create-remote-aem-pkg.sh script, you can transform package creation, filtering, and distribution into a streamlined, repeatable process. This tool isn’t just about saving time, it’s about enabling consistency, reliability, and scalability in your AEM operations. create-remote-aem-pkg.sh Key Takeaways Automation Wins: By eliminating repetitive GUI interactions, the script reduces human error and frees teams to focus on higher-value tasks. Flexibility Matters: Whether backing up critical content, syncing environments, or preparing for updates, the script adapts to diverse use cases with minimal tweaking. Resilience is Key: Built-in logging, error checks, and security considerations ensure the script behaves predictably, even when things go sideways. Automation Wins: By eliminating repetitive GUI interactions, the script reduces human error and frees teams to focus on higher-value tasks. Automation Wins : By eliminating repetitive GUI interactions, the script reduces human error and frees teams to focus on higher-value tasks. Automation Wins Flexibility Matters: Whether backing up critical content, syncing environments, or preparing for updates, the script adapts to diverse use cases with minimal tweaking. Flexibility Matters : Whether backing up critical content, syncing environments, or preparing for updates, the script adapts to diverse use cases with minimal tweaking. Flexibility Matters Resilience is Key: Built-in logging, error checks, and security considerations ensure the script behaves predictably, even when things go sideways. Resilience is Key : Built-in logging, error checks, and security considerations ensure the script behaves predictably, even when things go sideways. Resilience is Key Great tools are born from real-world challenges. This script is a starting point; think of it as a foundation to build upon as your team’s needs grow. Whether you’re a solo developer or part of a large DevOps team, automation like this exemplifies how small investments in code can yield outsized returns in productivity and peace of mind. Ready to take the next step? Ready to take the next step? 🛠️ Customize: Tailor the script using Section 6 as your guide. 🔍 Audit: Review your existing AEM workflows for automation opportunities. 🤝 Share: Mentor your team or write a blog post about your modifications. 🛠️ Customize : Tailor the script using Section 6 as your guide. Customize Section 6 🔍 Audit : Review your existing AEM workflows for automation opportunities. Audit 🤝 Share : Mentor your team or write a blog post about your modifications. Share Thank you for following along — now go forth and automate! 🚀 Appendix Complete Code #!/bin/bash set -eo pipefail # The script will create a package thought the package manager api: # - The package is created, if not already present # - Package filters are populated accordingly to specified paths # - Package is builded # - Package is download to the specified folder _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } check_last_exec () { local message="$1" local output="$2" local status=$3 if [ "$status" -ne 0 ]; then echo && echo "$message" && echo exit 1 fi if [[ $output =~ .*success\":false* ]] || [[ $output =~ .*"HTTP ERROR"* ]]; then _log "$message" exit 1 fi } USR="$1" PWD="$2" SVR="$3" PORT="$4" PKG_GROUP="$5" PKG_NAME="$6" BK_FOLDER="$7" shift 7 # The following paths will be included in the package FILTERS=($@) BK_FILE=$PKG_NAME"-"$(date +%Y%m%d-%H%M%S).zip _log "Starting backup process..." echo "AEM instance: '$SVR':'$PORT' AEM User: '$USR' Package group: $PKG_GROUP Package name: '$PKG_NAME' Destination folder: $BK_FOLDER Destination file: '$BK_FILE' Filter paths: " printf '\t%s\n\n' "${FILTERS[@]}" if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && echo exit 1 fi PKG_NAME=${PKG_NAME// /_} check_last_exec "Error replacing white space chars from package name!" "" $? || exit 1 _log "Removed whitespaces from package name: '$PKG_NAME'" BK_FILE=$PKG_NAME.zip _log "Backup file: '$BK_FILE'" _log "Creating the package..." if [ $(curl -k -u "$USR":"$PWD" "$SVR:$PORT/crx/packmgr/service.jsp?cmd=ls" 2>/dev/null | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log " Package '$PKG_GROUP/$PKG_NAME' is already present: skipping creation." else curl -k --silent -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/.json/etc/packages/$PKG_GROUP/$PKG_NAME?cmd=create" \ -d packageName="$PKG_NAME" -d groupName="$PKG_GROUP" check_last_exec " Error creating the package!" "" $? _log " Package created" fi # create filters variable FILTERS_PARAM="" ARR_LEN="${#FILTERS[@]}" for i in "${!FILTERS[@]}"; do FILTERS_PARAM=$FILTERS_PARAM"{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" T=$((i+1)) if [ $T -ne $ARR_LEN ]; then FILTERS_PARAM=$FILTERS_PARAM", " fi done # add filters _log "Adding filters to the package..." CURL_OUTPUT=$(curl -k --silent -u "$USR":"$PWD" -X POST "$SVR:$PORT/crx/packmgr/update.jsp" \ -F path=/etc/packages/"$PKG_GROUP"/"$PKG_NAME".zip -F packageName="$PKG_NAME" \ -F groupName="$PKG_GROUP" \ -F filter="[$FILTERS_PARAM]" \ -F "_charset_=UTF-8") CURL_STATUS=$? # Pass the status to the check_last_exec function check_last_exec "Error adding filters to the package!" "$CURL_OUTPUT" $CURL_STATUS _log " Package filters updated successfully." # build package _log "Building the package..." CURL_OUTPUT=$(curl -k -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/script.html/etc/packages/$PKG_GROUP/$PKG_NAME.zip" \ -F "cmd=build") check_last_exec " Error building the package!" "$CURL_OUTPUT" $? _log " Package built." # download package _log "Downloading the package..." if [ "$(curl -w "%{http_code}" -o "$BK_FOLDER/$BK_FILE" -k --silent -u "$USR":"$PWD" "$SVR:$PORT/etc/packages/$PKG_GROUP/$PKG_NAME.zip")" -eq "200" ]; then if [ -f "$BK_FOLDER/$BK_FILE" ] && [ -s "$BK_FOLDER/$BK_FILE" ]; then _log " Package $BK_FILE downloaded in $BK_FOLDER." exit 0 fi fi _log " Error downloading the package!" exit 1 #!/bin/bash set -eo pipefail # The script will create a package thought the package manager api: # - The package is created, if not already present # - Package filters are populated accordingly to specified paths # - Package is builded # - Package is download to the specified folder _log () { echo "[$(date +%Y.%m.%d-%H:%M:%S)] $1" } check_last_exec () { local message="$1" local output="$2" local status=$3 if [ "$status" -ne 0 ]; then echo && echo "$message" && echo exit 1 fi if [[ $output =~ .*success\":false* ]] || [[ $output =~ .*"HTTP ERROR"* ]]; then _log "$message" exit 1 fi } USR="$1" PWD="$2" SVR="$3" PORT="$4" PKG_GROUP="$5" PKG_NAME="$6" BK_FOLDER="$7" shift 7 # The following paths will be included in the package FILTERS=($@) BK_FILE=$PKG_NAME"-"$(date +%Y%m%d-%H%M%S).zip _log "Starting backup process..." echo "AEM instance: '$SVR':'$PORT' AEM User: '$USR' Package group: $PKG_GROUP Package name: '$PKG_NAME' Destination folder: $BK_FOLDER Destination file: '$BK_FILE' Filter paths: " printf '\t%s\n\n' "${FILTERS[@]}" if [ ! -d "$BK_FOLDER" ]; then _log "Backup folder '$BK_FOLDER' does not exist!" && echo exit 1 fi PKG_NAME=${PKG_NAME// /_} check_last_exec "Error replacing white space chars from package name!" "" $? || exit 1 _log "Removed whitespaces from package name: '$PKG_NAME'" BK_FILE=$PKG_NAME.zip _log "Backup file: '$BK_FILE'" _log "Creating the package..." if [ $(curl -k -u "$USR":"$PWD" "$SVR:$PORT/crx/packmgr/service.jsp?cmd=ls" 2>/dev/null | grep "$PKG_NAME.zip" | wc -l) -eq 1 ]; then _log " Package '$PKG_GROUP/$PKG_NAME' is already present: skipping creation." else curl -k --silent -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/.json/etc/packages/$PKG_GROUP/$PKG_NAME?cmd=create" \ -d packageName="$PKG_NAME" -d groupName="$PKG_GROUP" check_last_exec " Error creating the package!" "" $? _log " Package created" fi # create filters variable FILTERS_PARAM="" ARR_LEN="${#FILTERS[@]}" for i in "${!FILTERS[@]}"; do FILTERS_PARAM=$FILTERS_PARAM"{\"root\": \"${FILTERS[$i]}\", \"rules\": []}" T=$((i+1)) if [ $T -ne $ARR_LEN ]; then FILTERS_PARAM=$FILTERS_PARAM", " fi done # add filters _log "Adding filters to the package..." CURL_OUTPUT=$(curl -k --silent -u "$USR":"$PWD" -X POST "$SVR:$PORT/crx/packmgr/update.jsp" \ -F path=/etc/packages/"$PKG_GROUP"/"$PKG_NAME".zip -F packageName="$PKG_NAME" \ -F groupName="$PKG_GROUP" \ -F filter="[$FILTERS_PARAM]" \ -F "_charset_=UTF-8") CURL_STATUS=$? # Pass the status to the check_last_exec function check_last_exec "Error adding filters to the package!" "$CURL_OUTPUT" $CURL_STATUS _log " Package filters updated successfully." # build package _log "Building the package..." CURL_OUTPUT=$(curl -k -u "$USR":"$PWD" -X POST \ "$SVR:$PORT/crx/packmgr/service/script.html/etc/packages/$PKG_GROUP/$PKG_NAME.zip" \ -F "cmd=build") check_last_exec " Error building the package!" "$CURL_OUTPUT" $? _log " Package built." # download package _log "Downloading the package..." if [ "$(curl -w "%{http_code}" -o "$BK_FOLDER/$BK_FILE" -k --silent -u "$USR":"$PWD" "$SVR:$PORT/etc/packages/$PKG_GROUP/$PKG_NAME.zip")" -eq "200" ]; then if [ -f "$BK_FOLDER/$BK_FILE" ] && [ -s "$BK_FOLDER/$BK_FILE" ]; then _log " Package $BK_FILE downloaded in $BK_FOLDER." exit 0 fi fi _log " Error downloading the package!" exit 1 References [¹] Skipping SSL verification with curl -k is handy for testing, but you’ll want something sturdier in production (for example --cacert )! curl -k --cacert [²] AEM Package Manager Official Documentation AEM Package Manager Official Documentation AEM Package Manager Official Documentation