Sunday, December 6, 2009

MaxScript DotNet Sockets with Python

I create and work with several Python tools that manipulate scenes in 3ds Max. These are usually floating dialogs linked to the main 3ds Max window that send MaxScript commands via COM. This works well until I need the tool's UI to update when something happens in 3ds Max. Like refresh an object list when the selection in Max changes.

You would think doing a COM connection the other way would work. However, since Python-registered COM servers run separately in their own instance of the interpreter, there's no native connection to the original tool.

Nathaniel Albright, a fellow TA at Volition, recently created a Python COM server that communicated to his Python tool via TCP/IP socket. So it went 3ds Max -> Python COM server -> TCP/IP -> Python tool. This works well, but I wondered if the DotNet facilities in MaxScript offered a direct way to use sockets.

I had yet to touch DotNet in MaxScript, so this seemed like a good opportunity to learn a few things. After a lot of searching online I only turned up a few scraps of info, no complete recipe. However, I did find enough to get MXS DotNet sockets working, and assemble a comprehensive example.

I created a little Python tool that displays the names of all selected objects in the 3ds Max scene. As the scene selection changes, the list of names automatically updates. I won't go over all the code in this post, but the full working example tool is included in the zipfile below.

There's three main points of interest in the example:

1. Using DotNet in MaxScript to communicate via TCP/IP socket
2. Listening on a socket in a background thread in Python
3. Creating and posting custom wxPython events

The MaxScript Client

I made a MaxScript struct called "mxs_socket". The code follows, and also included in the zipfile below.

struct mxs_socket (
   ip_address = "", -- "localhost" also valid
   port       = 2323,        -- default port

   -- <dotnet>connect <string>ip_string <int>port_int
   -- Description:
   -- Takes IP address, port and connects to socket listener at that
   -- address
   fn connect ip_string port_int = (
      socket = dotNetObject "System.Net.Sockets.Socket" ( dotnetclass "System.Net.Sockets.AddressFamily" ).InterNetwork ( dotnetclass "System.Net.Sockets.SocketType" ).Stream ( dotnetclass "System.Net.Sockets.ProtocolType" ).Tcp
      socket.Connect ip_string port_int

      socket   -- return

   -- <int>send <string>data
   -- Description:
   -- Converts a string (or any object that can be converted
   -- to a string) to dotnet ASCII-encoded byte sequence and
   -- sends it via socket. Uses ip_address and port defined
   -- in struct above, or set by client.
   -- Returns integer of how many bytes were sent.
   fn send data = (
      -- Convert string to bytes
      ascii_encoder = dotNetObject "System.Text.ASCIIEncoding"
      bytes = ascii_encoder.GetBytes ( data as string )

      -- Connect, send bytes, then close
      socket = connect ip_address port
      -- result is # of bytes sent
      result = socket.Send bytes

      result  -- return # of bytes sent
Using this, I can send bytes to any socket listener on port 5432 by doing the following:
socket = mxs_socket port:5432
socket.send "Hello, World!"
The connect method was pretty simple in the end. The only twist turned out to be converting the socket integer into a DotNet socket object.

The send method converts the string into an ASCII-encoded DotNet bytes object, connects to the listener, sends the bytes, then closes the connection. The value returned is the number of bytes sent.

The last lines of the above code sets up a MaxScript callback that fires when the object selection changes in the scene. That uses mxs_socket to send a string containing the names of all the selected objects to any tool that's listening on that port.

Now I just need to make my Python tool listen.

The Python Server

My Python server/listener (also in the zipfile below) is a typical wxPython frame, but with two added qualities... It uses a background thread to listen on a socket, and posts a custom wx.Event when data is received. I had never used either of these techniques before, but it was fun getting it working.

Since a typical wxPython app sits in its main loop waiting for user input, I created a Socket_Listen_Thread class, a subclass of threading.Thread. This does the listening in a background thread while the main UI thread waits on the user. The run method here does the real work:
def run( self ):
   self.running = True

   while ( self.running ):
      # Starting server...
      # Listen for connection.  We're in non-blocking mode so it can
      # check for the signal to shut down from the main thread.
         client_socket, clientaddr = self.socket.accept( )
         data_received = True
      except socket.error:
         data_received = False

      if ( data_received ):
         # Set new client socket to block.  Otherwise it will
         # inherit the non-blocking mode of the server socket.
         client_socket.setblocking( True )

         # Connection found, read its data then close
         data = client_socket.recv( self.buffer_size )
         client_socket.close( )

         # Create wx event and post it to our app window
         event = self.event_class( data = data )
         wx.PostEvent( self.window, event )
This listening thread runs quietly in the background until it receives data. At that point I need a way to break into the main UI thread. There's other ways to do this, but using a custom wx.Event seemed to be the best fit here.

First, when the wx.Frame is opened, I create a custom wxPython event.
(Max_Update_Event, EVT_3DS_MAX_UPDATE) = wx.lib.newevent.NewEvent()
Calling NewEvent returns both a new Event class and an object to bind the event handler to. I pass the event class to the listener thread, bind an event handler to it, and that's all.

When data comes in over that TCP/IP port from our MaxScript tool, the listening thread receives it and posts our custom event to the main wx.Frame. That in turn fires the event handler to update the UI.

My example MaxScript client and Python listener described above can be found in the following ZIP file. Drop me a line if you do something useful with them, I'd love to hear about it. (4 KB)

Thanks to Nate Albright and everyone contributing to the "dotNet + MXS" and "Python + MXS" threads on the CGSociety forums. Those long-running threads have been very inspiring, and contain several tips that were key in getting the MaxScript DotNet socket stuff hammered out.

Thursday, August 6, 2009

Regular Expressions Coaching

Try as I might, I've never had instant recall on the details of regular expressions. For whatever reason (infrequent use, old age) the syntax just slides out of my head and onto the floor mere minutes after using it.

Awhile back a co-worker introduced me to The Regex Coach, and I've used it regularly ever since. Paste in a snippet of text, type a regex pattern, and it highlights matches in the text as you type. Super easy way to (re)learn or explore regex.

It's targeted at Perl-style regex, which for me has proved completely compatible with Python's re module. If you like it, don't forget to donate.

A bonus tip for the regex-challenged like myself...
If you memorize only one thing: (.*)

Saturday, June 27, 2009

Hidden HiddenDOSCommand details

Twice in recent months I've been bitten by MaxScript's HiddenDOSCommand.

It was added in 3ds Max 2008 as a way to issue DOS commands without bringing up an ugly command prompt. Sounds great but what the docs don't tell you is that the optional "startpath:" argument is actually not optional at all. If you leave it out you'll receive a cryptic error like this:.

HiddenDOSCommand "notepad %temp%\\cmdout.tmp" prompt:"Waiting..."
-- Error! CreateProcess(cmd /e:on /d /c "notepad %temp%\hiddencmdout.tmp") failed!
Note, that command was pasted from Example Usage in the MaxScript docs for HiddenDOSCommand. It will not work, nor will the other examples listed there unless you include "startpath"....
HiddenDOSCommand "notepad %temp%\\cmdout.tmp" prompt:"Waiting..." startpath:"C:\\"
This may have been addressed in the helpfile for 3ds Max 2010, I haven't checked. This can be Google fodder in the meantime.

Sunday, May 10, 2009

What we do with Python

There's a great thread going at called What do you do with Python? The other day I posted a few of the things our studio has done with Python in the past year or two...

  • Measure start/stop times of various processes, logging data to SQL database. For instance, how long it takes 3ds Max to start up, so we can spot bad trends when new tools are published.
  • System for logging errors and tools usage data to central database, with optional emailing of errors/callstack. Works for Python tools as well as MaxScript (via COM).
  • A non-linear GUI editor for an otherwise complex/table-driven cutscene pipeline.
  • Build graphical user interfaces (generally with wxPython) that integrate with in-house and off-the-shelf C applications. For example, floating Python dialogs that link to app windows as children, or as docking task panes.
  • Tool that communicates with game C code (via socket) running on consoles to do in-game realtime lighting.
  • Embed Python interpreter into editor framework for next-gen development tools. This is the one I spend lots of time on these days... works like MaxScript in 3ds Max, but for our custom editors.
  • One Exporter that writes out various data files from 3ds Max, Photoshop, and imports/categorizes them in our asset system.
  • Logs me into Outlook's webmail without manually entering my creds every time. I guess that was a home project. :)
  • At 3ds Max startup, scan folders for MaxScripts, building a MacroScript .mcr file for all of them.
  • At 3ds Max startup, builds list of texture map folders for a given project, sorts them by user's discipline and adds them to Max's bitmap paths list.
  • Profile rendering performance of art assets recently submitted to Perforce, recording data to SQL database.
  • Searches web-based bug tracker database for entries assigned to you and displays data in a Vista Sidebar gadget.
  • Creates makefiles with dependencies, for distributed build processes in Incredibuild/XGE.
  • Wavelet transform calculations for content-based image comparison tools. For finding textures that are too similar, or comparing rendered output of one shader vs. another.
  • Takes zipcode or lat/long as input, gathers geo-survey data from various online sources and creates the road/terrain network inside our world editor.
  • Tons of data mining uses. Like searching various exported XML files for instances of X material, mesh, etc. in game world.
  • Tool for bridging various apps with COM interfaces in other tools. Like firing MaxScripts in 3ds Max from Ultraedit, or taking current Python script in Wing and running it in our editor's embedded interpreter.
  • Custom scripts for integrating our tools/processes into Wing (the Python IDE we use).

P.S. Call your mom today.

Tuesday, March 24, 2009

Volition at GDC 2009

Hey if you're at this year's Game Developers Conference, be sure to check out the talks from my co-workers. Volition has a strong showing this year, with plenty of great tech artist material.

Blowing Up the Outside World: Destruction Done the Next Gen Way
by Eric Arnold and Jeff Hanna
This session presents an in-depth look at the tools and technologies used to make a truly destructible world for RED FACTION: GUERRILLA. The presenters will share the lessons they learned and the problems they had to overcome in order to have destruction in their game.

Technical Artist Roundtable
by Jeff Hanna
This roundtable will be an animated group discussion about being an effective technical artist. Topics of discussion will include what skills a technical artist should possess, how the role differs from company to company, scripting content creation applications, shader development, asset management, and improving production pipelines.

Technical Art Techniques Panel: Tools and Pipeline
Robert Galanakis, Jeff Hanna, Seth Gibson, Christopher Evans and Ross Patel
As game pipelines, their tools, and content become more complex, technical artists have become the developers of choice for much of the planning, overseeing, and implementation of pipelines. Technical artists from BioWare, Bungie, Microsoft, and Volition discuss their solutions and practices for tools and pipeline.

Breathing LIFE into an Open World
by Scott Phillips
Examine the history of populating open worlds and a detailed description and post-mortem of the LIFE system developed by Volition for SAINTS ROW 2. Attendees will learn about the inspiration, organization, methodology, successes and failures of the LIFE system used to add life to the open world city of Stilwater.

Universal Character System in SAINTS ROW 2
by Chris Fortier
Attendees will learn how SAINTS ROW 2's character customization and random NPC generator work. Some things that will be discussed include the universal body mesh, character morphing, normal map blending, layered clothing, shader-based customization features, how we assemble NPCs and how all this character variation affects animation.

Tuesday, March 17, 2009

Python cheat sheets

I'm out of hibernation, time I posted something.

As much as I use Python these days, there's a few things I find myself looking up regularly. At one point I just made a small crib sheet and stuck it to my monitor. I have examples on it for list comprehensions, filter, and map.

List Comprehensions
These are useful for creating modified lists from existing data without a lot of fuss. They aren't all that hard to remember, but the syntax was a bit alien to me for awhile. They're basically an expression followed by a for clause.

The below example takes an existing list, my_list and builds a new list with only the elements that are greater than 2. In this case the result is assigned right back to my_list.

my_list = [x for x in my_list if x > 2]
Using filter is a powerful way to remove undesired elements from a list. You pass a function as the first argument, which generally returns True/False based on some criteria. The second argument is the sequence to be filtered (or any iterable object). Only the elements that return True when passed to that function will remain in the newly returned list.

Filtering is often done with a lambda as the first function argument. A lambda is a one-off function that's defined and used in the same place. Since it's only used once, it doesn't need a name. It's so common to see filter and lambda together, the fact they were seperate didn't occur to me when I was learning the language.

In this example, we have a list of filenames, my_files, and we want to remove any that aren't Python scripts, ending in '.py'.
my_files = filter(lambda f: f.endswith('.py'), my_files)
That is the shorter equivalent of:
def is_py_filename(filename):
   return filename.endswith('.py')

my_files = filter(is_py_filename, my_files)
With filter, passing None as the first argument instead of a function automatically removes any elements that don't evaluate to True. That includes integers or floats that are zero, as well as occurrences of False or None.

my_files = filter(None, my_files)
Mapped functions let you apply a function to every element in a sequence.
def add_ten(x):
   return x+10

result = map(add_ten, [1,2,3,4,5])
The value of result would be [11, 12, 13, 14, 15]. Of course you could also use a lambda here, too:
result = map(lambda x: x+10, [1,2,3,4,5])
So what's on your cheat sheet?