Hi,
recently I had to improve our postjobscript system: I did some code rewrite, now much more elegant. Here I post an example script, that converts the current job's output to half sized exr, and a mov.
These are the changes since the old script that is referred in my old post:
- No more batch file! Running nuke directly.
- Automatically finding the installed nuke from deadline list
- Handling multiple views (currently left-right as viewnames are hardcoded, I need to change that)
- Handling multiple job output sequences
But as usual there can be bugs, so if you find one, please report.
The process to use:
- Put the 2 scripts in a folder somewhere in the deadline repository (for example I use this: DeadlineRepository\scripts\PrePostTaskJobScripts\JobNukeConvert\)
- Edit the JobNukeConvert.py: change the path in line (65) that starts with:
arg= (" -x -F " + framesList + ... to point to the current location of the other python script
- Set the JobNukeConvert.py as a postjobscript from deadline monitor (job properties)
- That's it. It should spit out a half-res. converted exr sequence, and a mov for each job output sequence (will put beside the rendered sequence)
- Of course you need to modify the glt_deadlinepostjobscript_convert.py according to your needs (what kind of formats to convert to etc.), but if you have some knowledge of nuke-python, that can't be that hard. Or maybe I can help...
Let me know if doesn't work for some reason. Or if it does work :) You can donwnload it from here.
I hope someone finds it useful,
Gabor
Hi Gabor,
ReplyDeleteBased off of the development you've done with Nuke/Deadline integration I think you may be able to answer a serious problem I'm having. At the studio where I work, I'm trying to set up Nuke for the renderfarm. All of our Nuke boxes are setup with an environment variable pointing to our global menu.py/init.py and scripts folder on the server. So all Nuke machines are running this based off of NUKE_PATH = "xxx" in the Windows environment variable per user. So, the problem I'm running into is that when a render node launches via command line, it bypasses the windows environment variables that are apparently only being initialized when the GUI launches.
Anyways, I hope this makes sense as I am in dire need of a fix. My coding skills really aren't much but I have a basic understanding of how to integrate scripts and make my way around them. I'm just wondering WHAT and WHERE I would edit something in Deadline to preLoad this for the render so I can still maintain a global scripts location. It works if I copy the folder to the render node locally. I have also tested the node locally by launching nuke gui and the custom scripts/menus appear. Just not in command line renders.
I found this thread detailing the exact same issue but in Muster, not Deadline. http://forums.thefoundry.co.uk/phpBB2/viewtopic.php?t=1931&postdays=0&postorder=asc&start=0
If you can help, please contact me via email at travis@travisbutton.com. I can not find ANY documentation on this specific problem. Thanks so much for your time and sorry for the long post!!!
-Travis
Hi Travis,
Deletejust sent a mail with some tips.
Hope it helps,
Gabor
As per Gabor's suggestions. Here is a fix I have gotten to work.
ReplyDeleteInstead of relying solely on a Windows based ENV variable for "NUKE_PATH", Gabor recommended going into each init.py file per render node in Nuke's Default root directory and point to your global scripts/menu.py/init.py that Nuke users are sharing from the server.
Go to:
C:\Program Files\Nuke6.x\plugins\init.py
Open this file and search for the line:
nuke.pluginAddPath("./user", addToSysPath=False)
Copy and paste this line to the one below it and change the "./user" to your global plugins path on the server. In my case it was:
nuke.pluginAddPath(W:/scripts/nuke", addToSysPath=False)
HINT* - If you copy this path from Windows explorer, you need to change "\" to "/" or it will not work! I actually then to automate the process a little bit, wrote a .bat file that automatically went to each of 50 render nodes and replaced the init.py from a master file.
ex:
copy "W:\source\file\location\init.py" "C:\Program Files\Nuke6.x\plugins\init.py"
Save this as a replaceFile.bat and run it!
Thanks again Gabor for the help! Hope this is useful for someone else.
Hi,
ReplyDeleteGreat work, this is just what i was looking for! However i get this error in deadline when it runs the script:
Error executing post job script: Python Error: index out of range: 0 (System.IndexOutOfRangeException)
(System.Exception)
I noticed my frame range is set to 1-5 but in the Error report in deadline the frame range shows 99999-99999, not sure if this is related?
Thanks in advance for any help.
Hi, no that's not a problem, post job scripts are always created with this number, to be the last task. Doesn't it write the error line number, or which script has the error? It could be in either one. Do you have output that is recognized by deadline? Which plugin do you use? If the renderplugin is nuke, do you have multiple views, other than 'left-right'?
ReplyDeleteHi Gabor, no line number unfortunately but it seems to be when running the JobNukeConvert.py. The job is a maya batch render saving out .exrs, the glt_deadlinepostscript_convert.py is your original version untouched. Once its up and running the plan is to resave the .exrs with zip1 compression (which Maya still cant do!)
DeleteThanks for your time, much appreciated.
Here it the full error log:
General deadline convert postjobscript is started!
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Error executing post job script: Python Error: index out of range: 0 (System.IndexOutOfRangeException)
(System.Exception)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Exception Details
RenderPluginException -- Error executing post job script: Python Error: index out of range: 0 (System.IndexOutOfRangeException)
(System.Exception)
RenderPluginException.Cause: JobError (2)
Exception.Data: ( )
Exception.TargetSite: Void RenderCurrentTask()
Exception.Source: deadline
Exception.StackTrace:
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()
Hi, yes it seems for some reason the script didn't get the list of outputdirectories. Please go to the job folder in deadline repository (find it by deadlinejobid) and open the job file (named something like "999_080_999_5cabebbb.job")
DeleteIn that xml-like file please check is there an entry, and then the actual outputdirectories, and then with filenames after it. As I suspect, there won't be any of this, or the script doesn't get it properly for some reason. Check this please.
Oh, which version of deadline are you using? We are still on 5.0, maybe that could be the problem, if you are on 5.1?
OutputDirectories has the correct path but there are no filenames, OutputFileNames tag is just one line and contains this:
DeleteIm using deadline 4.1.. maybe thats the issue?
(opentag)OutputFileNames /(closetag)
DeleteYes, I think that missing filename is the problem. I assume it should be there even in 4.1. Are you using some external renderer? If yes try with simple maya software or mental ray. Check the maya renderglobals output filename field also, is it properly named?
DeleteHey Gabor,
ReplyDeleteWanted to know if you've ever done anything like this with ffmpeg or ffmbc? I've been trying to find a solution to the Windows Nuke x64 ffmpegWriter to no avail. So, my other approach was Deadline. Ideally, a Nuke script would be executed and distributed via Deadline to render out an image sequence. Upon completion, that rendered image sequence would become the input on an ffmpeg or ffmbc commadline to generate pretty much any movie format you can think of. Only issue is that the process would not be distributed over multiple nodes but ran on one node utilizing most of the threads.
Deadline 5.0 currently doesn't support image sequences but 5.1 does, but won't be upgrading for a while.
Example commandlines below for commandline (ffmbc - which is based on ffmpeg). NOTE - The input jpg sequence numbering/padding is indicative of an image sequence that starts with xxx.1001.jpg.
Also threw in a little Nuke (afterrender) python script, but it's not written to deal with the initial render job, which is being rendered on multiple render nodes, and forcing the afterreder to render only on one node. The initial afterrender script was written by (http://www.richardgreenwood.ca/), I just modified it to work with different paddings:
------------------------------------------------------
FFMBC:
H264
"\\work\app_config\release\system\ffmbc.exe" -y -f image2 -r 23.976 -i "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-jpg\hfo_222_020-010_plate_el1_v001_wje.1%03d.jpg" -crf 10 -vcodec libx264 -preset slow -profile baseline -r 23.976 -color_primaries bt709 -color_transfer bt709 -color_matrix bt709 -pix_fmt yuv420p -threads 8 "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-mov\commandline_ffmbc_h264VHQ.mov"
PRORES
"\\work\app_config\release\system\ffmbc.exe" -y -f image2 -r 23.976 -i "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-jpg\hfo_222_020-010_plate_el1_v001_wje.1%03d.jpg" -vcodec prores -profile hq -r 23.976 -pix_fmt yuv422p10le -threads 8 "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-mov\commandline_ffmbc_proresHQ.mov"
DNXHD
"\\work\app_config\release\system\ffmbc.exe" -y -f image2 -r 23.976 -i "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-jpg\hfo_222_020-010_plate_el1_v001_wje.1%03d.jpg" -vcodec dnxhd -b 36M -r 23.976 -color_primaries bt709 -color_transfer bt709 -color_matrix bt709 -pix_fmt yuv422p -threads 8 "W:\newshowname_nsn-0000\222\020-010\elements\plates\plate_el1\v001\proxy-mov\commandline_ffmbc_dnxhd_36_8bit.mov"
Best,
Dan
Here's the afterrender nuke script I mentioned. Not enough room in initial post.
ReplyDeleteNUKE SETUP:
import nuke, nukescripts, os, re, shlex, subprocess
# Send rendered image sequence to shell avconv call
def sendToffmbc(codec = 'dnxhd'):
# Configuration
renderSlug = False
vcodec = {
'x264' : 'libx264 -pre baseline',
'dnxhd' : 'dnxhd -b 36M -r 23.976 -color_primaries bt709 -color_transfer bt709 -color_matrix bt709 -pix_fmt yuv422p -threads 8',
}
extension = '.mov'
# set some variables
fps = nuke.root().knob('fps').value()
firstFrame = int(nuke.root().knob("first_frame").getValue())
ss = 0 if renderSlug == True else secondsToStr(firstFrame/fps)
# grabs the write node's file value and makes sure the path uses printf style filenames
imgSeqPath = nukescripts.replaceHashes(nuke.filename(nuke.thisNode()))
imgSeqPath = imgSeqPath.replace(".%04d.",".1%03d.")
# generate mov path
base, ext = os.path.splitext(os.path.basename(imgSeqPath))
movPath = os.path.dirname(os.path.dirname(imgSeqPath)) + '/' + re.sub('\.?%0\d+d$', "", base) + extension
# make shell command
enc = 'ffmbc.exe -y -f image2 -r %s -i \'%s\' -vcodec %s \'%s\'' % (fps, imgSeqPath, vcodec[codec], movPath)
path = re.sub('/','\\\\', enc)
path = path.replace('\'','\"')
subprocess.Popen(shlex.split(enc), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# returns HH:MM:SS.SSS formatted string
# http://code.activestate.com/recipes/511486/
def secondsToStr(t):
rediv = lambda ll,b : list(divmod(ll[0],b)) + ll[1:]
return "%02d:%02d:%02d.%03d" % tuple(reduce(rediv,[[t*1000,],1000,60,60]))
I sent you a mail.
DeleteThis may be a really dumb question (fairly new to programming languages)
ReplyDeleteI am getting the error
"RenderPluginException -- Exception during render: An error occurred in RenderScript(): Python Error: Local variable 'counterLength' referenced before assignment. (IronPython.Runtime.UnboundLocalException)"
Am I missing something that is obvious?
Hi, what type of counter do you use in nuke (write node)? It should be something like %04d, or #### (could be different length of course). Seems that the counter type changer function doesn't get either type, that's why it's throwing this error. Let me know of the type of the counter in write node and see what can we do.
DeleteThis comment has been removed by the author.
ReplyDeleteAh, have fixed it thanks. Sorry for the inconvenience
ReplyDeleteWhat was the solution?
ReplyDeleteHi Gabor,
ReplyDeleteThanks for this! Really great to have this resource.
I'm getting the following error from Deadline:
Post job script "//vfxls/DeadlineRepository7/custom/scripts/Submission/Nuke/JobNukeConvert.py": TypeError : __main__() takes no arguments (1 given) (FranticX.Scripting.PythonNetException)
Any suggestions?
Thanks,
-Jake
try __main__(*args)
DeleteHello Gabor! I would love to try this out and learn from it. I cannot seem to access your link however..
ReplyDeleteAny chance you could re-upload it?
Thanks so much in advance!
Alican Sesli
Hi,
DeleteI think the link went dead due to the change in dropbox sharing system. I updated the link above, so you can access the file, hope it helps.
Cheers,
Gabor
Thank you for sharing this great work. I'm currently testing this out with my pipeline. I am running into the following error. Was wondering if anyone has a solution? Thank you.
ReplyDeleteAttributeError : type object 'SlaveUtils' has no attribute 'GetCurrentJobValue' (FranticX.Scripting.PythonNetException)
line 22, in __main__
name = SlaveUtils.GetCurrentJobValue( "Names" )
I'm on version 10. Can you provide an updated version? Thank you very much Gabor!
DeleteHi there - appreciate your script for me start coding in Deadline. But when I ran the script I get the following error message ..?
ReplyDelete=======================================================
Error
=======================================================
Error: Post task script "/prod/Deadline/scripts/PrePosPaskJobScritpts/JobNukeConvert/post_script_call_seq_to_mp4.py": NameError : global name 'GetConfigEntryWithDefault' is not defined (FranticX.Scripting.PythonNetException)
File "none", line 56, in __main__
at Deadline.Plugins.ScriptPlugin.RenderTasks (System.String taskId, Int32 startFrame, Int32 endFrame, System.String& outMessage, FranticX.Processes.AbortLevel& abortLevel) [0x00000] in :0
=======================================================
Type
=======================================================
RenderPluginException
=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.Plugin.RenderTask (System.String taskId, Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0
Thanks
/Biju
'GetConfigEntryWithDefault' is not defined (FranticX.Scripting.PythonNetException) ..?
DeleteHi, do you have the imports in the beginning of the script?
ReplyDeleteLike:
from System.IO import *
from deadline.scripting import * ...