Raspberry Pi Forums

Posted: **Wed Nov 21, 2018 3:59 pm**

I'm updating an electronic picture frame I made a long time ago and need to find or create a script to avoid duplicate file names but don't know where to start. All drives and directories are mounted and accessible, slide show script is running, and I've automated adding new pictures to the frame. It hangs on the living room wall and runs 4 hours per night, which makes adding pictures inconvenient when it's not running. So I have a directory on a NAS drive where I put pictures to be added to the frame, and they get copied over when it boots. The issue is the frame is over a decade old so now has several thousand pictures and I'm running into file name duplication issues. What I have is:
Directory 1 - Pictures on the frame that are displayed randomly
Directory 2 - Pictures to be added to the frame, on the NAS that get copied to directory 1 when the frame boots.
Directory 3 - Pictures that have been copied to the frame, on the NAS (just a way to check to make sure it's working)
Directory 4 - Pictures moved from directory 2 because they had the same file name as a picture in directory 1, that have not been copied to the frame. I'll manually rename these and put back into directory 2

What I'm trying to do is compare file names in D2 to D1. When there is a match move that file (or those files) from D2 to D4, then copy the remaining non-matching files from D2 to D1 and D3, then delete the contents of D2.

I haven't been able to find a script to do this and creating one is beyond my skill level. Any advice would be appreciated. I'm not a programmer so this modification may be a lost cause. Thanks in advance.

Posted: **Mon Nov 26, 2018 12:44 am**

What scripting language are you using?
It would be pretty simple to do a brute-force check in Python.

Posted: **Mon Nov 26, 2018 2:49 pm**

rzusman wrote: ↑
Mon Nov 26, 2018 12:44 am
What scripting language are you using?
It would be pretty simple to do a brute-force check in Python.

Since I started the thread I've been watching Python tutorials trying to figure out how to do this. 5 days later and I'm still not an expert, go figure...
But it's interesting and I'm making progress. I found this similar script and I'm trying to change it to do what I need. This one deletes duplicates, I want to move them to a separate directory, and you have to enter directory paths when you run this one, while my directory paths won't change so I need to hard code them in which will allow for automated running on boot. Also, I'm just interested in duplicate file names and don't want it to search recursively -

Code: Select all

import os
import filecmp
import argparse


def sub_files(folder):
    relpath = os.path.relpath
    join = os.path.join
    for path, _, files in os.walk(folder):
        relative = relpath(path, folder)
        for file in files:
            yield join(relative, file)


def remove_duplicates(folder_a, folder_b, shallow=False, dry_run=True):
    folders = [folder_a, folder_b]
    files = [set(sub_files(folder)) for folder in folders]
    duplicates, *_ = groups = [files[0] & files[1]]
    if not shallow:
        duplicates, *_ = groups = filecmp.cmpfiles(*folders, duplicates)
    print('\n\n'.join(
        '{}:\n{}'.format(name, '\n'.join('    {}'.format(file)
                                         for file in files))
        for files, name in zip(groups, ('Duplicates', 'Non-Duplicates', 'Errors'))
        if files
    ))
    if not dry_run:
        join = os.path.join
        remove = os.remove
        for file in duplicates:
            remove(join(folder_a, file))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('folder_a', help='folder to delete from')
    parser.add_argument('folder_b', help='folder to find duplicates in')
    parser.add_argument(
        '-d', '--dry', help='dry-run the program', action="store_true")
    parser.add_argument('-s', '--shallow', help='only check file names for duplicates',
                        action="store_true")
    args = parser.parse_args()

    remove_duplicates(args.folder_a, args.folder_b,
                      shallow=args.shallow, dry_run=args.dry)

I've removed argparse and references to it and hard coded directory paths. It no longer prints duplicates but doesn't return any errors. Printing them is just a step in my clumsy development "process". Once I know it's identifying duplicates I'll figure out how to move them from e:\test\source to e:\test\duplicates, then move the remaining files in source to e:\test\target. Here's what I have so far, any hints or advice is greatly appreciated -

Code: Select all

import os
import filecmp


def sub_files(folder):
    relpath = os.path.relpath
    join = os.path.join
    for path, _, files in os.walk(folder):
        relative = relpath(path, folder)
        for file in files:
            yield join(relative, file)


def remove_duplicates(folder_a, folder_b, shallow=True, dry_run=True):
    folders = [r'E:\test\source', r'E:\test\target']
    files = [set(sub_files(folder)) for folder in folders]
    duplicates, *_ = groups = [files[0] & files[1]]
    if not shallow:
        duplicates, *_ = groups = filecmp.cmpfiles(*folders, duplicates)
    print('\n\n'.join(
        '{}:\n{}'.format(name, '\n'.join('    {}'.format(file)
                                         for file in files))
        for files, name in zip(groups, ('Duplicates', 'Non-Duplicates', 'Errors'))
        if files
    ))
    if not dry_run:
        join = os.path.join
        remove = os.remove
        for file in duplicates:
            remove(join(folder_a, file))

Posted: **Mon Nov 26, 2018 2:56 pm**

Could you put your code in a code box eg

Code: Select all

import os
import filecmp
import argparse


def sub_files(folder):
      relpath = os.path.relpath

It will be much easier to understand and maintain the indentation.

Posted: **Mon Nov 26, 2018 3:20 pm**

gordon77 wrote: ↑
Mon Nov 26, 2018 2:56 pm
Could you put your code in a code box eg

It will be much easier to understand and maintain the indentation.

I was wondering how that was done, now I know. I edited the post above. Good advice, thanks gordon.

Posted: **Wed Nov 28, 2018 10:20 am**

Try this.. Backup any files first !! Change dir_1 etc as required

Code: Select all

#!/usr/bin/env python3
import os
import shutil

dir_1 = "/home/pi/Pictures/"
dir_2 = "/home/pi/Documents/"
dir_3 = "/home/pi/Downloads/"
dir_4 = "/home/pi/Templates/"

# read list of dir_2 Files
dir_2_File  = []
dir_2_File  = (os.listdir(dir_2))

# check for duplicates in dir_1 and dir_2, move matches to dir_4 then delete in dir_2
for counter in range (0,len(dir_2_File)):
   if os.path.exists(dir_1 + dir_2_File[counter]):
      shutil.copy(dir_2 + dir_2_File[counter], dir_4 + dir_2_File[counter])
      os.remove(dir_2 + dir_2_File[counter])

# read list of dir_2 Files
dir_2_File  = []
dir_2_File  = (os.listdir(dir_2))
    
# move files from dir_2 to dir_1 and dir_3     
for counter in range (0,len(dir_2_File)):
   shutil.copy(dir_2 + dir_2_File[counter], dir_1 + dir_2_File[counter])
   shutil.copy(dir_2 + dir_2_File[counter], dir_3 + dir_2_File[counter])
   os.remove(dir_2 + dir_2_File[counter])

Posted: **Wed Nov 28, 2018 9:31 pm**

Thank you so much Gordon! It works like a charm. And also thank you for adding the comments that made it understandable and a great learning experience!

Posted: **Thu Nov 29, 2018 8:23 pm**

Shorter version...

Code: Select all

#!/usr/bin/env python3
import os
import shutil

dir_1 = "/home/pi/Pictures/"
dir_2 = "/home/pi/Documents/"
dir_3 = "/home/pi/Downloads/"
dir_4 = "/home/pi/Templates/"

# read list of dir_2 Files
dir_2_File  = []
dir_2_File  = (os.listdir(dir_2))

for counter in range (0,len(dir_2_File)):
   # check for duplicates in dir_1 and dir_2, move matches to dir_4 then delete in dir_2
   if os.path.exists(dir_1 + dir_2_File[counter]):
      shutil.copy(dir_2 + dir_2_File[counter], dir_4 + dir_2_File[counter])
      os.remove(dir_2 + dir_2_File[counter])
   # if no match then move file to dir_1 and dir_3, and delete from dir_2
   else:
      shutil.copy(dir_2 + dir_2_File[counter], dir_1 + dir_2_File[counter])
      shutil.copy(dir_2 + dir_2_File[counter], dir_3 + dir_2_File[counter])
      os.remove(dir_2 + dir_2_File[counter])

Raspberry Pi Forums

Picture frame project, managing files

Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files

Re: Picture frame project, managing files