eduardo simioni

Looking around the web it seemed fairly simple to work with Google Python API to update a spreadsheet. But alas, it is not. Installation is super straightforward (with Python 2.7.3 at least), just download and run installer. Using it is another world.

Accessing and reading data from a spreadsheet seems to be fully working, but once you start to try other stuff you bump into many problems. I wanted to just create a new spreadsheet (or worksheet) and feed text data into it but had to adapt my way through to get it working. Although it’s in version 3, there are some serious limitations to it. Such as:

• There’s no support yet to delete spreadsheets.
• Error messages are quite cryptic, like “500 internal server error”.
• Client.InsertRow when used only on a brand new worksheet gives a ’400 bad request’ error, but works if the worksheet has a header row created by hand on the web UI.
• Deleteworksheet, althought it exists, always gives an ‘Internal server’ error
• InsertRow and update cell are called on the client, to update the spreadsheet online on each command, so if you are adding 2000 rows it might take a while. Making an instance of a cell or row feed, updating it locally and sending it to the server might work, but I haven’t tried it. I’m not sure there is a way to send a feed actually.
• There are many authentication methods, and although all except OAuth2 are deprecated, only OAuth1 and ClientLogin works with the old spreadsheet service framework, the only one I was able to use.
• Official documentation is scarce. Seems like there’s better support for Java and .NET API’s, while Python’s lagging behind.

Mind you, this is just a quick run to try out the API. Things could have been working better if I used OAuth2 instead of ClientLogin, but I didn’t have time right now to meddle more into this stuff. I’m not developing an app, just trying to quickly use spreadsheets with Python, so I’m not even sure if it’s worth going through all the bureocracy of registering a “client secret”. Anyway, without further ado, here’s the code (puns intended):

import gdata.spreadsheet.service

def getSpreadsheetKeys(sheetName="eventIDMap", worksheetName="Sheet1"):
    '''
    returns client, spreadsheet key and worksheet key from
    given names with an empty worksheet
    '''

    # connects to the service
    client = gdata.spreadsheet.service.SpreadsheetsService()
    client.ssl = True
    client.email = 'your@gmail.com'
    client.password = 'password'
    client.ProgrammaticLogin()

    # goes through the list of spreadsheets available in your@gmail.com google drive
    # looking for one named with sheetName
    spreadsheetFeed = client.GetSpreadsheetsFeed()
    sheet = None
    for spreadsheet in spreadsheetFeed.entry:
        if spreadsheet.title.text == sheetName:
            sheet = spreadsheet

    # if not found would create one. if client.CreateResource worked...
    #if sheet == None:
        #sheet = gdata.data.Resource(type='spreadsheet', title=sheetName)
        #sheet = client.CreateResource(sheet)

    # gets sheet key. it's actually in the url if you open it on a browser.
    sheetKey = sheet.id.text.rsplit('/', 1)[1]

    # gets a list of worksheets present in the spreadsheet
    worksheetFeed = client.GetWorksheetsFeed(sheetKey)
    for worksheetEntry in worksheetFeed.entry:
        if worksheetEntry.title.text == worksheetName:
            #worksheetEntry.title.text = ( "deleteme_" + ''.join(random.choice(string.ascii_lowercase + string.digits) for x in range(3)) )
            #client.UpdateWorksheet(worksheetEntry)
            #client.DeleteWorksheet(worksheetEntry) # this gives a 500 Internal Server Error
            #worksheet = client.AddWorksheet(worksheetName, 4000, 5, sheetKey)
            worksheet = worksheetEntry
        else:
            worksheet = client.AddWorksheet(worksheetName, 4000, 5, sheetKey)

    # gets worksheet key, also found in the url
    worksheetKey = worksheet.id.text.rsplit('/', 1)[1]

    return client, sheetKey, worksheetKey

main():

    # gather stuff you want to send to the spreadsheet into
    # a list or something, this case eventTable

    client, sheetKey, worksheetKey = getEventIDSpreadsheetKeys()

    # what happens here is that for each entry in eventTable
    # a dictionary is created. each label in the dictionary
    # correspond to the names in the first row on the worksheet
    # so A1 is eventid, B1 is eventname, and so on.
    for event in eventTable:
        eventDict = {'eventid':event[0],
                     'eventname':event[1],
                     'state':event[2],
                     'asd':event[3],
                     'wada':event[4],
                     'anothercollumn':event[5],
                     }
        client.InsertRow(rowDict, sheetKey, worksheetKey)

The biggest hurdle with the example above is that the spreadsheet “eventIDMap” must exist in your Google Drive and the first row must have the exact same elements as the dictionary, and they need to be in lower case. There’s no way to add that first row, not with import gdata.spreadsheet.service anyway.

Conclusion is, with Python, if you want to add data to a spreadsheet, it might be easier to output a csv file from whatever source you have and import it into a spreadsheet. On the other hand, reading data from a spreadsheet can be a couple of lines code.

As Jason Parks noted, Wing is definitely a great IDE to work with Python. With OpenMaya and pymel auto-completion together with code debugging it’s everything you might want to script on Maya. Eric Pavey has a great tutorial on how to setup everything, I will just try to resume it with some tips on getting it working on Linux. Some considerations:

• Sending code works as with Eclipse. You open a port on Maya and send it through a socket. Locally or remotely.
• Autocompletion works and in Maya 2012 is quite easy to setup.
• Debug works by importing wingdbstub in your script. Maya to Wing communication function basically the same way as sending code, but on a different port.

Each point is independent from one another and you have to setup each one differently.

Sending code from Wing to Maya.

1. Open port on Maya
2. Copy Eric Pavey’s scripts to their folders
3. Setup hotkeys on Wing’s keymap.normal

To open a port on Maya create a shelf button or add to userSetup.mel either:

import maya.cmds as cmds
    if cmds.commandPort(':7720', q=True) !=1:
        cmds.commandPort(n=':7720', eo = False, nr = True)

Or

// userSetup.mel
commandPort -name "127.0.0.1:7720" -echoOutput;
commandPort -name ":7720" -echoOutput;

You need then two .py files:

wingHotkeys.py – goes into your /.wingide4/scripts in your /home. This saves what you have on Wing on a temp file, opens a socket and communicates with Maya, calling:
executeWingCode.py – which goes under your /home//maya/scripts. It’s called inside Maya by the previous script to run the temp file.

Lastly, you need to bind one or more wrappers into a hotkey in Wing, which calls the functions in wingHotkeys.py. You should do this at /usr/lib/wingide4.0/keymap.normal. Just change one key to ‘Ctrl-P’: ‘python_to_maya()’ or ‘Ctrl-P’: ‘all_python_to_maya()’ for example. The scripts provided above are slightly edited versions of Eric Pavey’s version to work on Linux. The only changes are the socket line, the temp file and the send all to python wrappers.

Auto-completion

1. Add pi files to Wing Source Analysis

On Wing, under Edit/Preferences/Source Analysis/Advanced/Interface File Path add the directory: /usr/autodesk/maya/devkit/other/pymel/extras/completion/pi. That’s it. For older versions of Maya you might have to find or download the pi files from somewhere.

Debug

1. Copy wingdbstub.py from Wing to /maya/scripts
2. Edit wingdbstub.py and set kEmbedded=1
3. Enable Passive Listen and Kill Externally Launched on Wing
4. When you want to debug, import wingdbstub in your script

To set this up copy /usr/lib/wingide4.0/wingdbstub.py to /maya/scripts in your /home. Open it and search for kEmbedded and change from =0 to =1.

On Wing, under Edit/Preferences/Debugger/External/Remote activate ‘Enable Passive Listen’ and ‘Kill Externally Launched’. With this you are good to go, but when you want to debug something you need to add this to the beginning of the code you want to debug:

import wingdbstub
wingdbstub.Ensure()

This procedure is also on Maya’s help.

On Windows the folders are different, but these procedures are exactly the same. Both wingHotkeys.py and executeWingCode.py should also work on Windows, the location of the temp file and the socket line depends on the system you are running everything (left two different if’s there just for clarity.)

If you know where to find the information it becomes quite straight forward if you are not doing anything complex. With python you can use xml.dom.minidom, while with maxscript you can use either .NET Object “System.Xml.XmlDocument” or Class “System.Xml.XmlReader“.

With python you create a root element and append child elements from there. A very good sample can be found on: http://www.postneo.com/projects/pyxml/ To read you have getElementsByTagName on both python and dotnet XmlDocument. XmlReader might be a bit faster, but you have to parse elements by yourself.

There’s an issue with minidom’s toprettyprintxml, it adds whitespace and tabs between tags, around textNodes. And they are obviously read afterwards. A couple of different solutions are discussed on Ron Rothman’s blog, the easiest one using xml.dom.ext.PrettyPrint.

It’s quite easy. Basically you need to:

1. plot to the skeleton;
2. save one animation;
3. turn on Mirror Animation for the character;
4. plot back to the control rig, which then mirrors the animation;
5. rotate the Character Reference model 180 degrees;
6. plot back to the skeleton;
7. save mirrored animation.

The tricky part is just number 5 where you need to do some matrix rotation. Python for this would be something like:

from pyfbsdk import *
app = FBApplication()
char = app.CurrentCharacter
savePath = r"C:\" # could be FB application path or project path
filename = app.FBXFileName
skeleton = FBCharacterPlotWhere.kFBCharacterPlotOnSkeleton
ctrlrig = FBCharacterPlotWhere.kFBCharacterPlotOnControlRig

# plot to skeleton, see bellow
plotAnim(char, skeleton)

# save left animation
sOptions = FBFbxOptions(False) # false = save options
sOptions.SaveCharacter = True
sOptions.SaveControlSet = False
sOptions.SaveCharacterExtension = False
sOptions.ShowFileDialog = False
sOptions.ShowOptionsDialog = False
app.SaveCharacterRigAndAnimation(savePath + "\\" + filename + "_L", char, sOptions)

# activate mirror and plot
char.MirrorMode = True
plotAnim(char, ctrlrig)

# get reference model
refModel = FBFindModelByName("Character_Ctrl:Reference")

# rotating 180, the tricky part
# http://www.j3d.org/matrix_faq/matrfaq_latest.html#Q28
rotateY180 = FBMatrix()
rotateY180[0] = math.cos((180*0.017453292519943295769236907684886))
rotateY180[2] = math.sin((180*0.017453292519943295769236907684886))
rotateY180[8] = -math.sin((180*0.017453292519943295769236907684886))
rotateY180[10] = math.cos((180*0.017453292519943295769236907684886))

refMT = FBMatrix()
refModel.GetMatrix(refMT)

refModel.SetMatrix( MatrixMult(rotateY180, refMT) )
scene.Evaluate()

# plot back to skeleton
plotAnim(char, skeleton)

# save again
app.SaveCharacterRigAndAnimation(savePath + "\\" + filename + "_R", char, sOptions)

The plot and multiplication functions are:

# This is from Neil3d: http://neill3d.com/mobi-skript-raschet-additivnoj-animacii?langswitch_lang=en
def MatrixMult(Ma, Mb):
    res = FBMatrix()

    for i in range(0,4):
        for j in range(0,4):
            sum=0
            for k in range(0,4):
                sum += Ma[i*4+k] * Mb[k*4+j]

            res[i*4+j] = sum
    return res

def plotAnim(char, where):
    if char.GetCharacterize:
        switchOn = char.SetCharacterizeOn(True)

    plotoBla = FBPlotOptions()
    plotoBla.ConstantKeyReducerKeepOneKey = True
    plotoBla.PlotAllTakes = True
    plotoBla.PlotOnFrame = True
    plotoBla.PlotPeriod = FBTime( 0, 0, 0, 1 )
    #plotoBla.PlotTranslationOnRootOnly = True
    plotoBla.PreciseTimeDiscontinuities = True
    #plotoBla.RotationFilterToApply = FBRotationFilter.kFBRotationFilterGimbleKiller
    plotoBla.UseConstantKeyReducer = False
    plotoBla.ConstantKeyReducerKeepOneKey  = True

    if (not char.PlotAnimation(where, plotoBla)):
        FBMessageBox( "Something went wrong", "Plot animation returned false, cannot continue", "OK", None, None )
        return False

    return char

If you are exporting the character to a game, pay attention to the root node rotation. If you used it in the characterization, changes are it might also be rotated, and you might not want that to happen. You can also establish a convention on the name of the files, and easily detect if the animation currently open ends with _L or _R, and adapt the filename correctly.

FBApplication().SaveCharacterRigAndAnimation() is the equivalent of the Save Character Animation on the Character Controls window, which saves only the animation, without mesh. I prefer to use this when exporting only the animation from Motionbuilder to some other software, since it’s cleaner, faster and file sizes are smaller, but you could use any other function also.

It’s quite straight forward, once you have the correct package. Here is two places worth looking for 64 bits modules:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

http://www.activestate.com/activepython/downloads

If you install the module it rests under C:\Python26\Lib\site-packages. You just need to copy it to a sys.path from Motionbuilder’s python. A good choice is: C:\Program Files\Autodesk\Autodesk MotionBuilder 2011 64-bit\bin\x64\python\lib\plat-win

Made some improvements to the script. After almost blowing my brains out trying to debug again all the matrix math, it seems the additive results were not being added correctly to the control rig’s bind pose.

Anyway, I switched from getting the bind pose from the control rig for the actual “stance pose” for the bone skeleton. This is the pose stored when you “add the character”. I get it’s matrix and paste it on a frame on another layer at take03. It shouldn’t be needed if the bones have zeroed out rotations though.

From all that debugging I’ve learned a couple of things:

- How to install numpy and any other modules/libraries to Mobu’s python;
- That Mobu crashes if I try to extend a list of FBMatrix() to another list (but append works);
- That to get the stance pose I need to necessarily do a .SetMatrix then .AnimationNode.KeyCandidate(). And obviously a .GetMatrix before that to find the stance matrix.

I will post more about the modules thingy on the next days.

I think a couple more words for new animators might be helpful.

Retargeting is just the process of “copying” the animation from one skeleton to the other. As you probably know, the simple cut and paste of keyframes between characters do not work. The joints might have different names, might have different rotations, different zeroed rotations, positions, etc etc. Some real time engines are able to do this with very strict rules of how the skeletons must be made, most of the time the difference are just and only the proportions, where just the rotation of the joints (except the root node) are used between skeletons that are mostly equal.

When characters have different skeletons, even with different hierarchies, you have to use some sort of tool, like Motionbuidler. The way it does is to have two characters in the scene, and copy it from “one control rig to the other”. If you haven’t, you should familiarize yourself completely with Motionbuilder’s characterize tools, since they are essential to fully use the software. Characters in Motionbuilder can have many inputs, like an Actor from mocap software, the control rig, which is used to animate or edit the animation of a skeleton, or another character, to “retarget” the animation from this character to the current one.

Getting into more detail, to do this, you import or merge both skeleton hierarchies into one scene. You characterize each one of them, correctly. They need both to be on tpose, and ideally should have all their bones above ground (above 0 y). Refer to Motionbuilder help for this, it’s thoroughly explained there. And this is one of the tricky parts, to use the retargeter script, each animation to be imported must be on tpose on frame 0. You don’t actually need to create a control rig for each one of them, after both are characterized, you just need to select the input type of the new character to the old character, and activate it. Animation on both characters should then be synched, regardless of differences in hierarchies or proportions.

The retargeter script just automates this process, loading a folder containing .fbx or .bvh animations over the new character. It does characterize the animations if needed, but they need to follow either Motionbuilder nomenclature or 3dsMax Biped to be correctly characterized. If they are not, you can just edit the Motionbuilder one inside the script, changing the right column to match the names on your skeleton. You don’t need to change all of them, just the ones present on your skeleton (if you don’t have finger animations don’t bother, for example).

You can, for example, get all or some of the 2600+ mocap files from Carnegie Mellon University Motion Capture Database at cgspeed.com, which has .bvh’s and .fbx’s with a tpose on frame 0, and with the retargeter script quickly retarget them over your character.

Finally added some new features, some polish on the code and updated the script on github. You can use it to retarget any amount of animations over your characters without much hassle. You can download it here.

You just need a scene with an already characterized character and a folder with animations in .fbx. The animations need to be either already characterized or not. If they are not, they need to be on a tpose on frame 0 and have either Motionbuilder (with or without a prefix) or 3dsMax Biped nomenclature. They need to be characterized to be retargeted from one control rig to the other, so that’s why it needs a tpose. For a custom bone mapping you can easilly add a new one or edit the mobu one.

I might add bvh support in the future (maybe with FBFileBatch()?) when I find more free time again. Or feel free to add it yourself.

* edit: Ok, added support for bvh, namespace, and custom skeletons. You still need to edit the bone mapping if you have custom skeleton though.

Ok, done! Tested with EMotionFX at work and it worked great! It will probably work with both Unity, UDK, or any other animation system, but I’d love to hear from anyone that tries it out.

You can download the .py over at github, or all ziped files here. The readme files explains it all. You just need a characterized character, an animation ploted on take01, a pose ploted on take02 to be subtracted and an empty take03.

If you find any bugs or have any question, ideas or feature requests just send them my way.

Edit: added an example file using the Gremlin sample scene. Just open it and run the script. You can get a good idea of how it works by studying the content on the takes, it’s quite simple and straight forward.

Uploaded a tentative version to github. I got it working with a full character, but will have to try it out on EMotionFX on monday at work.

I need to add the pasting of the tpose automatically also.