Bash: FileBot get path of processed files (without history)?

[noviceiii]

Dear all

I have media files coming into a directory. This media files shall be renamed according the media type.
This works fine so far with below script (bash). Thanks to the file bot team for such a great tool! Amazing!

I'd like to enhance the script by the following logiv:
- delete the source directory, no matter whats in it. Which is easy possible with by rm -r $dir
- process the target directory any further

Questions:
- how do I know, when the source folder processing has been fully completed?
I could add rm -r $dir or clean=y (more or less solved as it is now).
- how do I get the path of the newly created destination folder?
There I would need the folder once, no matter how many files are in it.

From research I have learned, that a few people achieve this by using history.xml. I'd like to avoid this solution as it could lead to race conditions if the script is executed more than once in the same time. Which isn't vey much likely but it could happen.

Kind regards
n

Code: Select all

filebot.sh -script fn:amc --output "$otdir" --action copy --conflict override -non-strict \
    --def movieFormat="$otdir/Kino/{ny}/{ny}{fn =~ /3D/ ? '.[3D]-[H-SBS]' : ''}.{lang =~ // ? 'NOLANG' : lang}" \
    seriesFormat="$otdir/Serien/{n}/{fn}" animeFormat="$otdir/Anime/{n}/{fn}" musicFormat="$otdir/Audio/{n}/{fn}" \
    --def clean=y unsorted=y music=y artwork=n \
    "ut_label=None" "ut_state=128" "ut_title={n}" "ut_kind=multi" "ut_file=$file" "ut_dir=$dir"

[rednoah]

1.
Processing is complete when the command returns. The FileBot process itself ensures that only one process is actively processing files by locking the log file. Your command does not specify --log-file amc.log so you might end up with multiple instances of FileBot executing in parallel which is not ideal.

I recommend always specifying a log file, so that FileBot will lock that log file and prevent multiple processes running at the same time:

Code: Select all

--log-file amc.log

2.
You might be able to use the --def exec option for that. You can pass in a command-template and the amc script will execute it after processing files.

e.g.

Code: Select all

--def exec='ls -l {quote folder}'

[noviceiii]

Hi redNoah

Thank you for your kind explanation!

I understand, that there is no way to access the history variables (such as $to; $from; $date) directly.

Indeed, I don't use the log file by intention as I have no use for - unless debugging. I'd be fine to have filebot in multible instances running. It won't access the same folder more than once but could access others meanwhile.

However, if there is no other way, I'll follow your suggestion by enabling the log. Appreciated!

Just to complete the picture what I had in mind in pseudo code (maybe also as a feature request?):

Code: Select all

$returnvalues = filebot .... --def exitinfo=${from} --def exitinfo=${to} --def exitwith=${topath}
# Gives an array in $returnvalues with "/sourcepath/Avatar.2009.DTS.Stuff.yeahyeah.mkv", "/targetpath/Avatar (2009)/Avatar (2009).mkv", "movie"

Certainly different approaches are welcome.

Kind regards
n3

[rednoah]

Sorry, but exporting bash variables on exit is not something I've ever seen any tool do before, assuming that it's even possible from a process that is merely called by your bash script. This is the kind of information you'd just grep from the filebot console output, which is something bash is exceptionally good at.

Alternatively, you can do something like this:

Code: Select all

--def exec="store-exit-info {quote f, original, type, json}"

where store-exit-info is your custom script that is given all the information you want and then simply does it's own thing.


Note that you'll get the output path, and the original filename, but not the original file path. If you need that, you'd have to grep it from the history script output.

Maybe you could make a separate thread on what exactly it is you're trying to do, like original situation, what you want, what steps in between you have issues with, etc. The more information and context I have, the better I can think of a good solution.

[noviceiii]

There are several program that return somehow values in shell in the way shown (?).
Like

Code: Select all

$a='ls -l'

Nevertheless, I guess I got your point! thx.
That will work. I can do it with

Code: Select all

--exec myprogram.sh parameter1 parameter2 parameterN

And I probably understand your confusion: I have filebot is within a loop in my current script. I do therefore not know the file paths nor the files within a directory or in what order they are processed:

Code: Select all

inotifywait -m -r -e close_write,move -q --format '%w%f' $mydir | while read PATHS
do
 
  # TODO: IF MEDIA FILE; do FileBot; delete DIR; CALL --exec SUB SCRIPT for further processing of output
    file=$(basename "${PATHS}")
    dir=$(dirname "${PATHS}") 

    filebot.sh -script fn:amc --output "$otdir" --action copy --conflict override -non-strict \
    --def movieFormat="$otdir/Kino/{ny}/{ny}{fn =~ /3D/ ? '.[3D]-[H-SBS]' : ''}.{lang =~ // ? 'NOLANG' : lang}" \
    seriesFormat="$otdir/Serien/{n}/{fn}" animeFormat="$otdir/Anime/{n}/{fn}" musicFormat="$otdir/Audio/{n}/{fn}" \
    --def clean=y unsorted=y music=y artwork=n \
    "ut_label=None" "ut_state=128" "ut_title={n}" "ut_kind=multi" "ut_file=$file" "ut_dir=$dir"
    
    # TODO --exec SCRIPT 
    
  # ENDIF
done

[rednoah]

1.
'...' and `...` is not the same:

Code: Select all

$a='ls -l'
$a

This will result in ls -l being called.

Code: Select all

$a=`ls -l`
echo $a

This will print the output of ls -l which has previously been stored in a variable. You can do the same with FileBot, which is what I mean when I say you can just use grep, cut, awk, etc to parse the output and the the information you need from there.

When you call a command, the only thing you get back is (1) the numeric exit code (0 for success, and 1-255 for failure) and (2) the console output (stdout and stderr respectively). This holds true for any command from echo to filebot.


2.
If you use inotifywait then I'd make sure that new folders/files are added to the watch folder in an atomic single operation (and use a staging folder if you need to physically move files from another filesystem for some reason beforehand).

Here's something somewhat similar, very simple (and thus somewhat limited) I wrote some time ago:
https://github.com/filebot/plugins/blob ... ot-watcher

By not using -r you always now exactly which single file or folder has been added to the watch folder.


3.
As far as I can see, you end up calling FileBot for every single file in sequence, so there can't be any race conditions or concurrency problems. It's also very slow to call FileBot for individual files. More importantly, a few things like multi-part movie detection won't work if files aren't processed as part of the same batch.

[noviceiii]

1. yes

2.
Thank you for the example. I'll follow up on that.

3.
Any yes; multipart files wouldn't work with my script.
.. maybe not even music files if the directory would have been deleted after the first detection of media file

.. I shall spend more than 2 minutes for the conceptional work of my files

[noviceiii]

Hi

I have re-made the script as discussed. Please find it below. Feedback welcome. (don't be to harsh )
.. and mind the line brakes when copy/paste.

Greetings,
n3

Code: Select all

#!/bin/bash

# Watches a directory for new files.
# Fires up filebot to rename directory and files.
# Eventually transfers to a remote server.
# Requires FileBot, inotifywait and a valid
# ssh certifcate from the remote server. Google for "ssh-copy-id".

# v 0.3, December 12th, 2017 by n3
# thanks to rednoa from filebot forum
# not tested on TV Shows

#  TODO:
#  - target server handling
#  - testing with tv shows


# ------------< program settings >----------------- #
dirwtc="/opt/incoming"                              # directory where files come in
dirtmp="/opt/tmp"                                   # directory to for temporary files. should be different from above.
logfil="/opt/watchlog.log"                          # full path to log file
vmimes="directory|video|audio|empty|octet-stream"   # valid mime types to process

dstmvi="/destination/movie/"                        # movie direcotry in destination
dstshw="/destination/tvshow/"                       # show direcotry in destination
dstani="/destination/anime/"                        # anime direcotry in destination
dstaud="/destination/music/"                        # audio (music) direcotry in destination

dstusr="USERNAME"                                   # Username of target server. Login is done with certifcate. no passwort.
dstsrv="192.168.100.100"                            # Server address or name of target server

waitwt=15                                           # how long to wait before proceeding if a new file is ready. In seconds. Rather high for TV Shows.
waitof=900                                          # how long to wati if destination server isn't available. In seconds.
waitat=4                                            # number of attemps to check if server is available before terminate program.



# ------------< program >----------------- #
deb=""                                              # init debug var. No need to change something.
err=""                                              # init err var. No need to change something.
n=0                                                 # init integer. No need to change.
donotproceed="false"                                # init do not proceed. (bash doesn't know boolean haha)


inotifywait -m "$dirwtc" -e close_write,move --format '%w%f' | stdbuf -oL uniq | while read -r FILE;
do

    deb="[----] START\n"
    deb="$deb [INFO] Got triggered on file >$FILE< in directory >$dirwtc<.\n"
    deb="$deb [INFO] See if destination server >$dstsrv< is available.\n"


    for n in {0..$waitat}
    do
         # Check: see if server is available
        if [ $? -ne 0 ];
        then
            # dirty path: server not available
            let "n=n+1"

            err="$err [ERROR] Destination server >$dstsrv< is not available.\n"
            err="$err [ERROR] This is attempt no. >$n<. Waiting for >$waitof< seconds.\n"

            sleep "$waitof"
            donotproceed="true"
        else
            # happy path: server is available
            donotproceed="false"
            deb="$deb [INFO] Server >$dstsrv< is available. Checked for >$n< attemps. Go from this side.\n"

            #set n = wait to exit for-loop
            n=$waitat
        fi
    done

    # Check: see if file has been changed before we had the chance to process
    deb="$deb [INFO] See if >$FILE< has been touched longer than 1 second ago.\n"
    if [ `find "$FILE" -type f -newermt "1 second ago" -print -quit` ];
    then
        # dirty path
                err="$err [ERROR] File >$FILE< has been touched before cooldown of >$waitwt< seconds. Terminate program.\n"
        donotproceed="true"
    else
        # happy path
        deb="$deb [INFO] Glad to tell that we can proceed with >$FILE<.\n"
    fi

    # Check: see if source directory exist. We never know
    dir=$(dirname "${FILE}")

    # Check: if source directory exists
    if [ -d "$dir" ];
    then
        # happy path
        deb="$deb [INFO] Glad to tell that the source directory >$dir< exists.\n"
    else
       err="$err [ERROR] Source directory >$dir< of file >$FILE< dosn't exist.\n"
       donotproceed="true"
    fi


    # Check: if mime type is something we like
    if file --mime-type "$FILE" | egrep "$mimesty";
    then
        # happy path
        deb="$deb [INFO] Happy to say that we have a valid file of mime type >$FILE<.\n"
    else
       err="$err [ERROR] The file >$FILE< dosn't seem to have a mime type we atmire.\n"
       donotproceed="true"
    fi


    deb="$deb [INFO] Befor we start: value of DoNotProceed is >$donotproceed<.\n"

    # after all checking, proceed, if all checking was fine
    if [ $donotproceed = "false" ];
    then

        deb="$deb [INFO] Proceed with file bot and file >$FILE<. Yayy.\n"

        # if filebot isn't working, use full path to filebot.sh
        ret=`filebot -script fn:amc --log-file "$logfil" \
        --output "$dirtmp" --action copy \
        --conflict override -non-strict \
        --def \
        movieFormat="$dirtmp/movies/{ny}/{ny}{fn =~ /3D/ ? '.[3D]-[H-SBS]' : ''}{'.'+lang}" \
        seriesFormat="$dirtmp/shows/{n}/Season {s00}/{s00e00} - {t}" \
        animeFormat="$dirtmp/animes/{n}/{fn}" \
        musicFormat="$dirtmp/audio/{n}/{fn}" \
        --def \
        clean=y unsorted=y music=y artwork=n \
        "$FILE"`

        deb="$deb [INFO] Filebot returns:\n\n>$ret<.\n\n"


        # get the new path of the new file name from the return values of filebot
        newpath=`echo $ret | grep -o -P '(from \[.*\]) (to \[)(\K.*)(\/)'`

        deb="$deb [INFO] The path of the processed file is >$newpath<, from >$FILE<.\n"

        # see if that path exists, that filebot created
        if [ -d "$newpath" ];
        then
            deb="$deb [INFO] The path >$newpath< has been successfully created.\n"

            # removing files in watched directory
            drm=`rm -r "$FILE"`
            deb="$deb [INFO] Removed >$FILE< with status >$drm<.\n"

            # handle according tmp directory

            # temp path is for movies
            if [ -d "$dirtmp/movies/" ];
            then
                # copy to server: movies
                deb="$deb [INFO] Copy MOVIE >$newpath< to $dstusr@$dstsrv:$dstmvi.\n"
                sc=`scp -r -v "$newpath" "$dstusr"@"$dstsrv":"$dstmvi"`
                deb="$deb [INFO] The return value of SCP is >$sc<.\n"
                
                # TODO: check if destination exists/ if file has been transfered
                rm -r "$dirtmp/movies/"
            fi

            # temp path is for tv shows
            if [ -d "$dirtmp/shows/" ];
            then
                # copy to server: movies
                deb="$deb [INFO] Copy TV Shows >$newpath< to $dstusr@$dstsrv:$dstshw.\n"
                sc=`scp -r -v "$newpath" "$dstusr"@"$dstsrv":"$dstshw"`
                deb="$deb [INFO] The return value of SCP is >$sc<.\n"

                # TODO: check if destination exists/ if file has been transfered
                rm -r "$dirtmp/shows/"
            fi

            # temp path is for animes
            if [ -d "$dirtmp/animes/" ];
            then
                # copy to server: movies
                deb="$deb [INFO] Copy Animes >$newpath< to $dstusr@$dstsrv:$dstani.\n"
                sc=`scp -r -v "$newpath" "$dstusr"@"$dstsrv":"$dstani"`
                deb="$deb [INFO] The return value of SCP is >$sc<.\n"

                # TODO: check if destination exists/ if file has been transfered
                rm -r "$dirtmp/animes/"
            fi

            # temp path is for audio files
            if [ -d "$dirtmp/audio/" ];
            then
                # copy to server: movies
                deb="$deb [INFO] Copy Audio files >$newpath< to $dstusr@$dstsrv:$dstaud.\n"
                sc=`scp -r -v "$newpath" "$dstusr"@"$dstsrv":"$dstaud"`
                deb="$deb [INFO] The return value of SCP is >$sc<.\n"

                # TODO: check if destination exists/ if file has been transfered
                rm -r "$dirtmp/audio/"
                            fi

            # temp path is unknown/ unsorted files
            if [ -d "$dirtmp/Unsorted/" ];
            then
                # copy to server: movies
                err="$err [ERROR] Unsorted! Did not proceed files of >$newpath<. \n"
            fi

                err="$err [OK] Finished for >$newpath<. \n"

        else
            # dirty path
            err="$err [ERROR] The path >$newpath< is missing.\n"
            # donotproceed="true"
        fi

    else
        err="$err [ERROR] Filebot did not start. Value of donotproceed says >$donotproceed<.\n"
    fi

    # delete log
    echo "" > $logfil
    
    # output

    echo "- DEBUG VALUES ------- "
    echo "$deb";

    echo "- ERROR VALUES ------- "
    echo "$err"


done # end of while loop

[noviceiii]

maybe to add. My initial question was how to get the new file name.
I do this now by cutting (regexp) the screen output.
1/ get all return values into a variable $ret
2/ cut out the from/to values into $newpath

Honestly, I am not too happy about my solution. I regard it as conceptually wrong.
Preferably I'd get an array back (or similar) from filebot (maybe something for a feature request?) for batch processing.

However, it works this way for now. And filebot is an amazing tool!

Code: Select all

#1/ get result into variable
ret=`filebot -script fn:amc --log-file "$logfil" \
        --output "$dirtmp" --action copy \
        --conflict override -non-strict \
        --def \
        movieFormat="$dirtmp/movies/{ny}/{ny}{fn =~ /3D/ ? '.[3D]-[H-SBS]' : ''}{'.'+lang}" \
        seriesFormat="$dirtmp/shows/{n}/Season {s00}/{s00e00} - {t}" \
        animeFormat="$dirtmp/animes/{n}/{fn}" \
        musicFormat="$dirtmp/audio/{n}/{fn}" \
        --def \
        clean=y unsorted=y music=y artwork=n \
        "$FILE"`

        # 2/ get the new path of the new file name from the return values of filebot
        newpath=`echo $ret | grep -o -P '(from \[.*\]) (to \[)(\K.*)(\/)'`  
 

[rednoah]

Re-organize previously organized files using local xattr metadata