I have several file systems mounted on /mnt/hdd0, /mnt/hdd1, /mnt/hdd2, etc. on my Manjaro Linux system. Let’s call these file systems fs0, fs1, fs2, and so on. Currently, all of these file systems have the same directory structure and contain files.

My goal is to reorganize and redistribute all the files across the drives based on the alphabetical ordering of their relative file paths, following these rules:

  1. The files with the earliest alphabetical relative paths should be moved to the first mounted file system in alphabetical order (e.g. /mnt/hdd0) which contains the appropriate folder structure.
  2. The files with the latest alphabetical relative paths should be moved to the last mounted file system in alphabetical order (e.g. /mnt/hddN) which contains the appropriate folder structure.
  3. The remaining files should be distributed across the remaining mounted file systems (/mnt/hdd1, /mnt/hdd2, etc.) in alphabetical order based on their relative paths.

The process should fill up each file system in alphabetical order (/mnt/hdd0, /mnt/hdd1, etc.) until the given free space threshold is reached before moving on. While reorganizing, I want to ensure each file system maintains a specified amount of free space remaining. If a file system does not have enough free space for the files being moved to it, I want to move enough existing files from that file system to another file system with the most available space to make room.

I need step-by-step instructions, including commands, scripts, or tools in Manjaro Linux, to automate this entire process of reorganizing and redistributing the files while following the rules mentioned above.

  • @The_Lemmington_PostOPM
    link
    1
    edit-2
    4 months ago

    To accomplish this task, we can use a combination of Bash scripting and various Linux utilities such as find, mv, du, and sort. Here’s a step-by-step approach:

    1. Create a bash script file, let’s call it reorganize.sh.
    #!/bin/bash
    
    # Define the mount points and the desired free space threshold
    declare -a MOUNT_POINTS=("/mnt/hdd0" "/mnt/hdd1" "/mnt/hdd2" ...)
    FREE_SPACE_THRESHOLD=20  # In gigabytes
    
    # Function to get the available space on a file system
    get_available_space() {
        local mount_point="$1"
        df --output=avail "$mount_point" | tail -n 1 | awk '{print $1}'
    }
    
    # Create an associative array to keep track of the available space on each file system
    declare -A AVAILABLE_SPACE
    for mount_point in "${MOUNT_POINTS[@]}"; do
        AVAILABLE_SPACE["$mount_point"]=$(get_available_space "$mount_point")
    done
    
    # Function to find the mount point with the most available space
    find_mount_point_with_most_space() {
        local max_space=0
        local max_mount_point=""
    
        for mount_point in "${!AVAILABLE_SPACE[@]}"; do
            if [ "${AVAILABLE_SPACE[$mount_point]}" -gt "$max_space" ]; then
                max_space="${AVAILABLE_SPACE[$mount_point]}"
                max_mount_point="$mount_point"
            fi
        done
    
        echo "$max_mount_point"
    }
    
    # Function to redistribute files based on alphabetical order
    redistribute_files() {
        local source_dir="$1"
        local target_dir="$2"
    
        # Get a list of files sorted alphabetically by relative path
        local files=$(find "$source_dir" -type f | sort)
    
        # Move files to the target directory
        while read -r file; do
            local relative_path=$(realpath --relative-to="$source_dir" "$file")
            local file_size=$(du -k "$file" | cut -f 1)
    
            # Update available space on source and target directories
            AVAILABLE_SPACE["$source_dir"]=$((AVAILABLE_SPACE["$source_dir"] + file_size))
            AVAILABLE_SPACE["$target_dir"]=$((AVAILABLE_SPACE["$target_dir"] - file_size))
    
            mkdir -p "$target_dir/$(dirname "$relative_path")"
            mv "$file" "$target_dir/$relative_path"
        done <<< "$files"
    }
    
    # Function to move files from one file system to another if the free space is below the threshold
    move_files_if_needed() {
        local source_dir="$1"
        local temp_dir=$(mktemp -d --tmpdir="$(find_mount_point_with_most_space)")
    
        if [ "${AVAILABLE_SPACE[$source_dir]}" -lt "$((FREE_SPACE_THRESHOLD * 1024 * 1024))" ]; then
            # If the available space on the source directory is below the threshold,
            # move the files to a temporary directory on the mount point with the most available space
            redistribute_files "$source_dir" "$temp_dir"
            rm -rf "$source_dir"/*
            # Move the files back to the source directory, preserving the alphabetical order
            redistribute_files "$temp_dir" "$source_dir"
            rmdir "$temp_dir"
        fi
    }
    
    # Main script
    for ((i=0; i<${#MOUNT_POINTS[@]}; i++)); do
        local mount_point="${MOUNT_POINTS[$i]}"
        # Check if the current mount point needs to be cleared to meet the free space threshold
        move_files_if_needed "$mount_point"
        # Redistribute files from the current mount point to the remaining mount points in alphabetical order
        for ((j=i+1; j<${#MOUNT_POINTS[@]}; j++)); do
            redistribute_files "$mount_point" "${MOUNT_POINTS[$j]}"
        done
    done
    
    1. Save the script file and make it executable:
    chmod +x reorganize.sh
    
    1. Run the script:
    ./reorganize.sh
    
    • @The_Lemmington_PostOPM
      link
      1
      edit-2
      3 months ago

      This wouldn’t work because of the way it deletes everything when making space on a filesystem. It should just move enough files to make space for the ones it’s moving there. Even if this script worked, I would need a lot of testing before I were confident using it to reorganize my actual data.

      Also, to copy the directory structure I use:

      rsync -av -f"+ */" -f"- *" "/mnt/hdd1" "/mnt/hdd2" &