Add support for multiple levels of folders. #199 (#205)

* Use original name from EXIF instead of parsing assumed file name format. #107
* Updates to docs and code

Readme.md

@@ -122,7 +122,8 @@ I work tirelessly to make sure your photos are always sorted and organized so yo
 You don't love me yet but you will.
 
 I only do 3 things.
-* Firstly I organize your existing collection of photos.
+
+* Firstly I organize your existing collection of photos into a customizable folder structure.
 * Second I help make it easy for all the photos you haven't taken yet to flow into the exact location they belong.
 * Third but not least I promise to do all this without a yucky propietary database that some friends of mine use.
 
@@ -152,16 +153,15 @@ Updating EXIF of photos from the command line.
 
 I'm most helpful when I'm fully utilized to keep your photos organized.
 
-Here's an example of a very asynchronous setup.
+Here's an example of how I can create 3 geographically distributed copies of your meticulously organized photo library.
+
 * Specify a folder in your Dropbox/Google Drive to store the organized photo library.
-* Set up a Hazel rule to notify me when photos arrive in `~/Downloads` so I can import them.
+* Set up a cron job to import photos in `~/Ready-To-Upload`.
-  * The rule waits 1 minute before processing the photo which gives you a chance to move it elsewhere if it's not something you want in the library.
+* Add photos to `~/Ready-To-Upload` and wait for your cron job to trigger.
-* Use AirDrop to transfer files from any iPhone to your laptop. That goes to `~/Downloads` for the Hazel rule to process.
-  * AirDrop is fast, easy for anyone to use and once the transfer is finished your don't have to stick around. I'll move it to Dropbox/Google Drive and Dropbox/Google Drive will sync it to their servers.
 * Periodically recategorize photos by fixing their location or date or by adding them to an album.
 * Have a Synology at home set to automatically sync down from Dropbox/Google Drive.
 
-This setup means you can quickly get photos off your or anyone's phone and know that they'll be organized and backed up in 3 locations by the time you're ready to view them.
+This setup means you can quickly get photos off your phone or dSLR and know that they'll be organized and backed up in 3 locations by the time you're ready to view or share them.
 
 <p align="center"><img src ="creative/workflow-simplified-white-bg.png" /></p>
 
@@ -209,33 +209,60 @@ OK, so what if you don't like the folders being named `2015-07-Jul/Mountain View
 
 You can add a custom folder structure by editing your `config.ini` file. This is what I include in the sample config file.
 
+#### Custom folder examples
+
+Sometimes examples are easier to understand than explainations so I'll start there. If you'd like to understand my magic I explain it in more detail below these examples. You customize your folder structure in the `Directory` section of your `config.ini`.
+
 ```
-[Directory]
+location=%city, %state
-date=%Y-%m-%b
+year=%Y
-location=%city
+full_path=%year/%location
+
+# 2015/Sunnyvale, California
+
+location=%city, %state
+month=%B
+year=%Y
+full_path=%year/%month/%location
+
+# 2015/December/Sunnyvale, California
+
+location=%city, %state
+month=%m
+year=%Y
+date=%year-%month
 full_path=%date/%location
+
+# 2015-12/Sunnyvale, California
+
+full_path=%country/%state/%city
+
+# US/California/Sunnyvale
+
 ```
 
-There needs to be 2 levels of folders and you can construct them using the date and location. Use `full_path` to determine how the 2 levels are nested. If for some reason your config is not correct I will use the default formatting which is found in `config.ini-sample`.
+#### How folder customization works
 
-The default formatting from the above config looks like `2015-07-Jul/Mountain View`.
+You can construct your folder structure using a combination of the location and dates. Under the `Directory` section of your `config.ini` file you can define placeholder names and assign each a value. For example, `date=%Y-%m` would create a date placeholder with a value of YYYY-MM which would be filled in with the date from the EXIF on the photo.
 
-#### Customizing the date format
+The placeholders can be used to define the folder structure you'd like to create. The example above happens to be the default structure and would look like `2015-07-Jul/Mountain View`.
 
-You can use any of [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior) to create your ideal structure.
+I have a few built-in location placeholders you can use.
 
-* To have `201601`, use `date=%Y%m`
+* `%city` the name of the city the photo was taken. Requires geolocation data in EXIF.
-* For `Sunday, 01 January 2016`, use `date=%A, %d %B %Y`
+* `%state` the name of the state the photo was taken. Requires geolocation data in EXIF.
-* Python also has some pre-built formats. So you can get `Sun Jan 01 12:34:56 2016`, by using `%c`
+* `%country` the name of the country the photo was taken. Requires geolocation data in EXIF.
 
-#### Customizing the location format
+I also have some date placeholders you can customize. You can use any of [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior) to customize the date format to your liking.
 
-I use the [Open Street Maps Nominatim reverse geocoding API](http://wiki.openstreetmap.org/wiki/Nominatim#Example) provided by MapQuest. You can use `city`, `state` and `country` to construct the folder name.
+* `%day` the day the photo was taken.
+* `%month` the month the photo was taken.
+* `%year` the year the photo was taken.
 
-* To have `Sunnyvale`, use `location=%city`
+In addition to my built-in and date placeholders you can combine them into a single folder name using my complex placeholders.
-* To have `Sunnyvale-CA`, use `location=%city-%state
 
-Sometimes a location may not have all of the values available. If your format is `%city-%state` and `city` was not returned then the folder name will be `%state`. Take note that I'll strip out extra characters so you don't end up with folders name `-%state` when `city` is not found.
+* `%location` can be used to combine multiple values of `%city`, `%state` and `%country`. For example, `location=%city, %state` would result in folder names like `Sunnyvale, California`.
+* `%date` can be used to combine multiple values from [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior). For example, `date=%Y-%m` would result in folder names like `2015-12`.
 
 ### Reorganize by changing location and dates
 

elodie/compatability.py

@@ -26,8 +26,15 @@ def _decode(string, encoding=sys.getfilesystemencoding()):
 
 
 def _copyfile(src, dst):
-    # Python 3 hangs using open/write method
+    # shutil.copy seems slow, changing to streaming according to
+    # http://stackoverflow.com/questions/22078621/python-how-to-copy-files-fast  # noqa
+    # Python 3 hangs using open/write method so we proceed with shutil.copy
+    #  and only perform the optimized write for Python 2.
     if (constants.python_version == 3):
+        # Do not use copy2(), it will have an issue when copying to a
+        #  network/mounted drive.
+        # Using copy and manual set_date_from_filename gets the job done.
+        # The calling function is responsible for setting the time.
         shutil.copy(src, dst)
         return
 

elodie/filesystem.py

@@ -114,7 +114,7 @@ class FileSystem(object):
         # First we check if we have metadata['original_name'].
         # We have to do this for backwards compatibility because
         #   we original did not store this back into EXIF.
-        if(metadata['original_name'] is not None):
+        if('original_name' in metadata and metadata['original_name']):
             base_name = os.path.splitext(metadata['original_name'])[0]
         else:
             # If the file has EXIF title we use that in the file name
@@ -163,18 +163,18 @@ class FileSystem(object):
 
         config_directory = config['Directory']
 
-        path_parts = re.search(
+        # Find all subpatterns of full_path that map to directories.
-                         '\%([^/]+)\/\%([^/]+)',
+        #  I.e. %foo/%bar => ['foo', 'bar']
+        path_parts = re.findall(
+                         '\%([a-z]+)',
                          config_directory['full_path']
                      )
 
-        if not path_parts or len(path_parts.groups()) != 2:
+        if not path_parts or len(path_parts) == 0:
             return self.default_folder_path_definition
 
-        path_part_groups = path_parts.groups()
         self.cached_folder_path_definition = [
-            (path_part_groups[0], config_directory[path_part_groups[0]]),
+            (part, config_directory[part]) for part in path_parts
-            (path_part_groups[1], config_directory[path_part_groups[1]]),
         ]
         return self.cached_folder_path_definition
 
@@ -188,25 +188,21 @@ class FileSystem(object):
         path = []
         for path_part in path_parts:
             part, mask = path_part
-            if part == 'date':
+            if part in ('date', 'day', 'month', 'year'):
                 path.append(time.strftime(mask, metadata['date_taken']))
-            elif part == 'location':
+            elif part in ('location', 'city', 'state', 'country'):
-                if(
+                place_name = geolocation.place_name(
-                    metadata['latitude'] is not None and
+                    metadata['latitude'],
-                    metadata['longitude'] is not None
+                    metadata['longitude']
-                ):
+                )
-                    place_name = geolocation.place_name(
+
-                        metadata['latitude'],
+                location_parts = re.findall('(%[^%]+)', mask)
-                        metadata['longitude']
+                parsed_folder_name = self.parse_mask_for_location(
-                    )
+                    mask,
-                    if(place_name is not None):
+                    location_parts,
-                        location_parts = re.findall('(%[^%]+)', mask)
+                    place_name,
-                        parsed_folder_name = self.parse_mask_for_location(
+                )
-                            mask,
+                path.append(parsed_folder_name)
-                            location_parts,
-                            place_name,
-                        )
-                        path.append(parsed_folder_name)
 
         # For now we always make the leaf folder an album if it's in the EXIF.
         # This is to preserve backwards compatability until we figure out how
@@ -217,11 +213,6 @@ class FileSystem(object):
             elif(len(path) == 2):
                 path[1] = metadata['album']
 
-        # if we don't have a 2nd level directory we use 'Unknown Location'
-        if(len(path) < 2):
-            path.append('Unknown Location')
-
-        # return '/'.join(path[::-1])
         return os.path.join(*path)
 
     def parse_mask_for_location(self, mask, location_parts, place_name):
@@ -338,11 +329,6 @@ class FileSystem(object):
             shutil.move(_file, dest_path)
             os.utime(dest_path, (stat.st_atime, stat.st_mtime))
         else:
-            # Do not use copy2(), will have an issue when copying to a
-            # network/mounted drive using copy and manual
-            # set_date_from_filename gets the job done
-            # shutil.copy seems slow, changing to streaming according to
-            # http://stackoverflow.com/questions/22078621/python-how-to-copy-files-fast  # noqa
             compatability._copyfile(_file, dest_path)
             self.set_utime(media)
 
@@ -352,7 +338,7 @@ class FileSystem(object):
         return dest_path
 
     def set_utime(self, media):
-        """ Set the modification time on the file base on the file name.
+        """ Set the modification time on the file based on the file name.
         """
 
         # Initialize date taken to what's returned from the metadata function.

elodie/geolocation.py

@@ -19,6 +19,7 @@ from elodie import log
 from elodie.localstorage import Db
 
 __KEY__ = None
+__DEFAULT_LOCATION__ = 'Unknown Location'
 
 
 def coordinates_by_name(name):
@@ -115,10 +116,14 @@ def get_key():
 
 
 def place_name(lat, lon):
+    lookup_place_name_default = {'default': __DEFAULT_LOCATION__}
+    if(lat is None or lon is None):
+        return lookup_place_name_default
+
     # Convert lat/lon to floats
-    if not isinstance(lat, float):
+    if(not isinstance(lat, float)):
         lat = float(lat)
-    if not isinstance(lon, float):
+    if(not isinstance(lon, float)):
         lon = float(lon)
 
     # Try to get cached location first
@@ -132,19 +137,18 @@ def place_name(lat, lon):
 
     lookup_place_name = {}
     geolocation_info = lookup(lat=lat, lon=lon)
-    if(geolocation_info is not None):
+    if(geolocation_info is not None and 'address' in geolocation_info):
-        if('address' in geolocation_info):
+        address = geolocation_info['address']
-            address = geolocation_info['address']
+        for loc in ['city', 'state', 'country']:
-            for loc in ['city', 'state', 'country']:
+            if(loc in address):
-                if(loc in address):
+                lookup_place_name[loc] = address[loc]
-                    lookup_place_name[loc] = address[loc]
+                # In many cases the desired key is not available so we
-                    # In many cases the desired key is not available so we
+                #  set the most specific as the default.
-                    #  set the most specific as the default.
+                if('default' not in lookup_place_name):
-                    if('default' not in lookup_place_name):
+                    lookup_place_name['default'] = address[loc]
-                        lookup_place_name['default'] = address[loc]
 
     if('default' not in lookup_place_name):
-        lookup_place_name = {'default': 'Unknown Location'}
+        lookup_place_name = lookup_place_name_default
 
     if(lookup_place_name is not {}):
         db.add_location(lat, lon, lookup_place_name)

elodie/tests/filesystem_test.py

@@ -248,6 +248,48 @@ full_path=%date/%location
 
     assert path == os.path.join('2015-12-05','United States of America-California-Sunnyvale'), path
 
+@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir())
+def test_get_folder_path_with_with_more_than_two_levels():
+    with open('%s/config.ini-location-date' % gettempdir(), 'w') as f:
+        f.write("""
+[Directory]
+year=%Y
+month=%m
+location=%city, %state
+full_path=%year/%month/%location
+        """)
+
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    filesystem = FileSystem()
+    media = Photo(helper.get_file('with-location.jpg'))
+    path = filesystem.get_folder_path(media.get_metadata())
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    assert path == os.path.join('2015','12','Sunnyvale, California'), path
+
+@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir())
+def test_get_folder_path_with_with_only_one_level():
+    with open('%s/config.ini-location-date' % gettempdir(), 'w') as f:
+        f.write("""
+[Directory]
+year=%Y
+full_path=%year
+        """)
+
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    filesystem = FileSystem()
+    media = Photo(helper.get_file('plain.jpg'))
+    path = filesystem.get_folder_path(media.get_metadata())
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    assert path == os.path.join('2015'), path
+
 def test_get_folder_path_with_location_and_title():
     filesystem = FileSystem()
     media = Photo(helper.get_file('with-location-and-title.jpg'))
@@ -658,3 +700,47 @@ full_path=%date/%location
     ]
     if hasattr(load_config, 'config'):
         del load_config.config
+
+@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir())
+def test_get_folder_path_definition_with_more_than_two_levels():
+    with open('%s/config.ini-location-date' % gettempdir(), 'w') as f:
+        f.write("""
+[Directory]
+year=%Y
+month=%m
+day=%d
+full_path=%year/%month/%day
+        """)
+
+    if hasattr(load_config, 'config'):
+        del load_config.config
+    filesystem = FileSystem()
+    path_definition = filesystem.get_folder_path_definition()
+    expected = [
+        ('year', '%Y'), ('month', '%m'), ('day', '%d')
+    ]
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    assert path_definition == expected, path_definition
+
+@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir())
+def test_get_folder_path_definition_with_only_one_level():
+    with open('%s/config.ini-location-date' % gettempdir(), 'w') as f:
+        f.write("""
+[Directory]
+year=%Y
+full_path=%year
+        """)
+
+    if hasattr(load_config, 'config'):
+        del load_config.config
+    filesystem = FileSystem()
+    path_definition = filesystem.get_folder_path_definition()
+    expected = [
+        ('year', '%Y')
+    ]
+    if hasattr(load_config, 'config'):
+        del load_config.config
+
+    assert path_definition == expected, path_definition