Python cheat sheets

I'm out of hibernation, time I posted something.

As much as I use Python these days, there's a few things I find myself looking up regularly. At one point I just made a small crib sheet and stuck it to my monitor. I have examples on it for list comprehensions, filter, and map.

List Comprehensions
These are useful for creating modified lists from existing data without a lot of fuss. They aren't all that hard to remember, but the syntax was a bit alien to me for awhile. They're basically an expression followed by a for clause.

The below example takes an existing list, my_list and builds a new list with only the elements that are greater than 2. In this case the result is assigned right back to my_list.

my_list = [x for x in my_list if x > 2]

Filter
Using filter is a powerful way to remove undesired elements from a list. You pass a function as the first argument, which generally returns True/False based on some criteria. The second argument is the sequence to be filtered (or any iterable object). Only the elements that return True when passed to that function will remain in the newly returned list.

Filtering is often done with a lambda as the first function argument. A lambda is a one-off function that's defined and used in the same place. Since it's only used once, it doesn't need a name. It's so common to see filter and lambda together, the fact they were seperate didn't occur to me when I was learning the language.

In this example, we have a list of filenames, my_files, and we want to remove any that aren't Python scripts, ending in '.py'.

my_files = filter(lambda f: f.endswith('.py'), my_files)

That is the shorter equivalent of:

def is_py_filename(filename):
   return filename.endswith('.py')

my_files = filter(is_py_filename, my_files)

With filter, passing None as the first argument instead of a function automatically removes any elements that don't evaluate to True. That includes integers or floats that are zero, as well as occurrences of False or None.

my_files = filter(None, my_files)

Map
Mapped functions let you apply a function to every element in a sequence.

def add_ten(x):
   return x+10

result = map(add_ten, [1,2,3,4,5])

The value of result would be [11, 12, 13, 14, 15]. Of course you could also use a lambda here, too:

result = map(lambda x: x+10, [1,2,3,4,5])

So what's on your cheat sheet?

Read-only Windows files with Python

How do you use Python to get or change read-only/writeable access on files in Windows? The Python docs don't answer this in a direct manner. Here's one option using only the standard library.

import os, stat
myFile = r'C:\stuff\grail.txt'

fileAtt = os.stat(myFile)[0]
if (not fileAtt & stat.S_IWRITE):
   # File is read-only, so make it writeable
   os.chmod(myFile, stat.S_IWRITE)
else:
   # File is writeable, so make it read-only
   os.chmod(myFile, stat.S_IREAD)

You may prefer the pywin32 extensions for this sort of thing...

import win32api, win32con
myFile = r'C:\stuff\grail.txt'

fileAtt = win32api.GetFileAttributes(myFile)
if (fileAtt & win32con.FILE_ATTRIBUTE_READONLY):
   # File is read-only, so make it writeable
   win32api.SetFileAttributes(myFile, ~win32con.FILE_ATTRIBUTE_READONLY)
else:
   # File is writeable, so make it read-only
   win32api.SetFileAttributes(myFile, win32con.FILE_ATTRIBUTE_READONLY)

Or, more concisely with win32:

roAtt = win32api.GetFileAttributes(myFile) & win32con.FILE_ATTRIBUTE_READONLY
win32api.SetFileAttributes(myFile, ~roAtt)

Using win32 you can also set other Windows file attributes (unlike os.chmod), but read/write is usually all I care about.

ColladaMax for 3ds Max 2009 64-bit

I was unable to find the ColladaMax importer/exporter plugin for 3ds Max 2009 64-bit, so I built one. It's from the 3.05b source.

Update 02/03/09 - I made a new build that depends on an older version of the DirectX SDK. It should fix the "failed to initialize" error some of you were getting, without the need to install anything else.

Feel free to grab it if you like:
ColladaMax_2009x64.rar (709 KB)

Photoshop scripting with Python

Photoshop natively supports scripting with AppleScript, JavaScript and VBScript. While Python is notably absent from that list, it can still be used to automate nearly anything in Photoshop. This is thanks to the extensive COM interface Photoshop provides.

The methods here are similar to those used in my GDC 2008 Python lecture, about driving 3ds Max via Python. You start by dispatching the Photoshop COM server, using Python as the client:

import win32com.client
psApp = win32com.client.Dispatch("Photoshop.Application")

This connects to your already-opened Photoshop session, or opens one if none are running. The root COM object is then assigned to psApp, and you're ready to do some cool stuff. Here's a quick example:

psApp.Open(r"D:\temp\blah.psd")         # Opens a PSD file
doc = psApp.Application.ActiveDocument  # Get active document object
layer = doc.ArtLayers[0]                # Get the bottom-most layer
layer.AdjustBrightnessContrast(20,-15)  # Bright +20, Contrast -15
doc.Save()                              # Save the modified PSD

Here's a more complex example. This script recursively scans a folder for PSD files, exporting various textures contained inside. One PSD can have specifically-named Layer Groups, each of which is written to a separate PNG file with a specific suffix. If a Group contains several layers, they're flattened when exported, allowing you to keep all your layered effects intact in the PSD.

In the example below, a group named "diffuse" is exported as "psdname_D.png", the "normal" group as "psdname_N.png", and so on. The exportType dictionary determines the name/suffix pairs.

# Recursively scans a folder (psdRoot) for Photoshop PSD files.
# For each, exports various 24-bit PNG textures based on layer
# groups found in the PSD.
# Requires the Win32 Extensions:
# http://python.net/crew/mhammond/win32/

import win32com.client
import os

# Change to match your root folder
psdRoot = r'C:\ArtFiles\PSD'

# Map of layer group names and the suffixes to use when exporting
exportTypes = {'diffuse':'_D', 'normal':'_N', 'specular':'_S'}

if (__name__ == '__main__'):
   # COM dispatch for Photoshop
   psApp = win32com.client.Dispatch('Photoshop.Application')

   # Photoshop actually exposes several different COM interfaces,
   # including one specifically for classes defining export options.
   options = win32com.client.Dispatch('Photoshop.ExportOptionsSaveForWeb')
   options.Format = 13   # PNG
   options.PNG8 = False  # Sets it to PNG-24 bit

   # Get all PSDs under root dir
   psdFiles = []

   for root, dir, files in os.walk(psdRoot):
      for thisFile in files:
         if (thisFile.lower().endswith('.psd')):
            fullFilename = os.path.join(root, thisFile)
            psdFiles.append(fullFilename)

   # Loop through PSDs we found
   for psdFile in psdFiles:
      doc = psApp.Open(psdFile)
      layerSets = doc.LayerSets

      if (len(layerSets) > 0):
         # First hide all root-level layers
         for layer in doc.Layers:
            layer.Visible = False
         # ... and layerSets
         for layerSet in layerSets:
            layerSet.Visible = False
           
         # Loop through each LayerSet (aka Group)
         for layerSet in layerSets:
            lsName = layerSet.Name.lower()

            if (lsName in exportTypes):
               layerSet.Visible = True  # make visible again

               # Make our export filename
               pngFile = os.path.splitext(psdFile)[0] + exportTypes[lsName] + '.png'

               # If PNG exists but older than PSD, delete it.
               if (os.path.exists(pngFile)):
                  psdTime = os.stat(psdFile)[8]
                  pngTime = os.stat(pngFile)[8]
        
                  if (psdTime > pngTime):
                     os.remove(pngFile)

               # Export PNG for this layer Group
               if (not os.path.exists(pngFile)):
                  doc = psApp.Open(psdFile)
                  doc.Export(ExportIn=pngFile, ExportAs=2, Options=options)
                  print 'exporting:', pngFile
               else:
                  print 'skipping newer file:', psdFile
                 
               # Make LayerSet invisible again
               layerSet.Visible = False

         # Close PSD without saving
         doc.Close(2)

It only exports when the PNG is missing or older than the PSD. This makes it good for running a batch texture export on your project's entire texture tree.

Here is a ZIP containing the above script and a sample PSD file to try it on: exportTextureLayers.zip (143 KB)

I imagine you can do all of the above with the native Photoshop scripting. I just think it's cool being able to use Python instead of rooting through a language I'm less familiar with. Dinosaurs were roaming the earth the last time I tried anything in VB.

If you dig this, I'd recommend reading the Photoshop CS5 Scripting Guide and Photoshop CS5 VBScript Reference found in the Adobe Photoshop Developer Center. While the above wasn't VBScript, the COM interface we used is nearly identical.

Checksums in 3ds Max (part 2 of 2)

In Part 1 I showed how to calculate checksums inside 3ds Max. Here's how to do something useful with them.

Any TA that's crossed paths with 3ds Max can tell you it doesn't do the best job of managing scene materials. Due to scene object merges/imports and other typical operations, it's common for a given material to be copied several times in one Max scene. Meaning, it's not instanced across several objects, but actually copied several times in memory. This can lead to increased memory usage and potentially inefficiencies in your game engine (depending how your exporter deals with this).

What's worse, you usually can't rely on similar material names to find duplicates by hand. To do a thorough search with MaxScript, you would need to loop through every material in the scene and compare every property in it to every other material's property. This would be a slow process in C, and a complete horror-show with MaxScript.

Enough grim talk. Here's a walkthrough of a MaxScript that uses checksums to make short work of this. To summarize, the script loops through the materials in your scene, creating a checksum for each as it goes. It uses that checksum to do a quick compare on previous material checksums it found, to see if they're actually property-identical. If it finds a dupe, that object is given the original material instead, effectively deleting the duplicated material.

The script is divided into three functions and a short bit of main code.

getChecksum() is the first function, taken from my previous post. It calls the Python COM object we registered, which returns a checksum to the MaxScript. If you can't (or don't want to) set-up the COM object, you can use the MaxScript implementation I listed in that blog post instead... it's just less robust than the MD5 checksums used by the Python method.

Next is the getPropsString() and getMaterialChecksum() functions:

------------------------------------------------------------
-- (str)getPropsString (material)mat
--
-- Description:
-- Builds a string representing the property names/values
-- of the supplied Max material.
------------------------------------------------------------
fn getPropsString mat = (
   myStr = "" as stringStream
   if (mat == undefined) then (
      format "undefined" to:myStr
   ) else (
      -- Start our string w/the classname
      format (classOf mat as string) to:myStr
      if (classof mat == ArrayParameter) then (
         -- Array, so recursively add strings for each element
         for element in mat do (
            format (getPropsString element) to:myStr
         )
      ) else (
         -- Not an array, so see if it has properties
         propNames = undefined
         try (
            propNames = getPropNames mat
         ) catch ()
         if (classOf mat == BitMap) then (
            try (  -- Add bitmap's filename
               format mat.filename to:myStr
            ) catch ()
         ) else if (propNames == undefined) then (
            format (mat as string) to:myStr
         ) else (
            format (propNames as string) to:myStr
            -- Loop through properties, adding their names
            -- and values to our string to be checksummed
            for i in 1 to propNames.count do (
               format (i as string) to:myStr
               p = propNames[i]
               val = getProperty mat p
               format (i as string) to:myStr
               format (getPropsString val) to:myStr
            )
         )
      )
   )
   (myStr as string)
)

------------------------------------------------------------
-- (str)getMaterialChecksum (material)mat
--
-- Description:
-- Takes a Max material (or multi-sub material) and
-- calculates a checksum value from it, for use as a
-- hashtable key, or whatever you like.
------------------------------------------------------------
fn getMaterialChecksum mat = (
   str = ""
   if (classof mat == Multimaterial) then (
      for id in mat.materialIDList do (
         -- Add material IDs as factors
         str += id as string
      )
      for subMat in mat.materialList do (
         -- Get string representing each submaterial
         str += (getPropsString subMat)
      )
      ) else (
         -- Get string representing this material
         str += getPropsString mat
      )
   -- Add string length as a factor
   str += str.count as string

   -- Get checksum from our base string
   -- 99991 = largest prime number under 10k
   (getChecksum str)
)

The above functions work together to generate a string of data representing the supplied 3ds Max material (or Multi-Sub material). Once it has that string, it's passed to getChecksum().

The main code block loops through the entire 3ds Max scene, doing the above for every material found on geometry objects:

----------
-- MAIN
----------
-- Set up a few things first.

timeStart = timestamp()  -- Time we started process
removedCount = 0  -- Counters for printing info below
uniqueCount = 0

-- Array of two synced arrays, first with the material
-- checksums, second with materials themselves.
-- Basically a poor-man's hashtable.
csMatArr = #(#(), #())

format "Scanning scene materials...\n"

-- Loop through all geometry
for obj in geometry do (
   mat = obj.material

   alreadyDone = (findItem csMatArr[2] mat) != 0

   if (not alreadyDone) and (mat != undefined) then (
      -- First get this material's checksum
      csum = getMaterialChecksum mat

      idx = findItem csMatArr[1] csum

      if (idx != 0) then (
         -- Dupe material found, so remove it by
         -- assigning the first mat to this object
         format "Replacing material '%' with '%'\n" mat.name csMatArr[2][idx].name
         obj.material = csMatArr[2][idx]
         removedCount += 1
      ) else (
         -- New checksum, so add it to our table,
         -- along with the material itself.
         append csMatArr[1] csum
         append csMatArr[2] mat
         uniqueCount += 1
      )
   )
)

gc()  -- Remind Max to take out the trash

-- Done, print some results
format "-- DONE in % secs --\n" ((timestamp() - timeStart) / 1000.0)
format "Old material count = %\n" (uniqueCount + removedCount)
format "New material count = %\n" uniqueCount
format "Duplicates removed = %\n" removedCount

That's it. At the end a summary is printed to the MaxScript Listener.

In the Max scene I was working with today this script cut the root material count from 533 to 261. That's 51% fewer materials! It also reduced the file load time from 136 seconds to 102 seconds.

You can download the complete script above here: RemoveDupeMaterials.zip (4 KB)
It includes the Python script to register the COM server, and the alternate MaxScript checksum method.

Update 7/25/08: I modified getPropsString() to better handle bitmap values, and generally run faster. The ZIP file above is updated as well. Thanks to MoonDoggie/Colin on CGTalk for the feedback!