Obstacles and Tips for Cloud-based Rendering
The cloud is a topic that is on a lot of people's minds, from how to get started to questions on security and costs.
Recently Broadcast TECH sat down with our CEO, Richard Lewis to get his take on the cloud. You can read the article here, but for the full interview, have a look below.
Q: At what point (size of project) does Cloud-based rendering become viable?
Cloud rendering is totally viable in a variety of situations, but project size isn’t the main determinant. The real sweet spots for cloud rendering are:
- When you only need it for a short time - A broadcast or post-production firm with an unusually large project can manage all of the extra rendering in the cloud without having to build facilities or purchase, setup and maintain significant additional compute resources.
- When you are late - If you’ve misjudged the rendering requirement of a project or the scope has expanded beyond your internal rendering capabilities, then the cloud to the rescue!
- When you’ve maxed out your infrastructure - If you are already using all your dedicated render nodes, all of your desktops when they are free or can’t rent any more machines either to add to your data center or in a local private cloud, then look to the cloud.
An independent artist or broadcast TV station could use the cloud for rendering under any of the above conditions, as could a small, medium or large post house or VFX studio, an architecture firm or an ad agency. It’s not really the size of the project that drives the benefits of the cloud, as much as the situation.
Q: How does it compare to a studio hosting its own on-premises hardware, and using a render farm manager like Qube! What are the benefits of each approach?
Since all professional rendering work needs to be managed by a render manager (like Qube!), I see rendering resources falling into four main categories.
- Dedicated render nodes - If a company is in the business of visual effects, post-production or computer generated imagery, they will have on-premise dedicated rendering resources that are used 90% or more of the year.
- Content creation desktops used when available - Artist desktops are powerful rendering machines and can be harnessed to add significant render power to a pipeline when well managed. Some of our customers announce at 8pm that it’s time to log out and everyone’s desktop becomes a rendering machine. Others have a logout policy that states when an artist leaves their desk for the day, they need to logout so it can be included in the render farm.
- Contracted private cloud render nodes, which are widely available in some markets and not at all in others - Canada enjoys a large percentage of all visual effects work and the industry supporting that work is quite mature. Our Canadian customers are able to contract servers for rendering in private clouds for extended periods of time to match their project requirements. While this is certainly a very secure, fast and cost-effective strategy, it may not be available to you simply due to your company’s geographic location.
- Public cloud render nodes - Prior to 2015 there were three main roadblocks to public cloud rendering: security, software availability and data transfer latency. Only recently have these been seriously addressed. Services like Google Cloud Platform’s VPN service, the availability of rental software licenses for both rendering and render management and data caching appliances from Avere Systems for rent in the Amazon EC2 compute cloud and Google Compute Engine have made cloud rendering an option for nearly everyone.
So our customers typically render on their render nodes in-house, their desktops when they are free, contracted private cloud servers they’ve rented for the duration of a project and cloud nodes for the rest.
Q: Local and cloud rendering can be combined, but how does this handover usually work? What’s the normal workflow?
When companies start rendering in the cloud, having a mature render management system in place is extremely important. A mature render manager is more than just a job dispatcher. It should have the capability to define groups or clusters of machines and offer priority queuing to insure the important work gets done first. Cloud rendering requires administrative control to make sure that jobs are dispatched to the appropriate resource. For example short-running NUKE renders that are unable to exploit Avere cache coherence in the cloud, might better be rendered locally while long-running 3D renders (V-Ray, Arnold, RenderMan, etc…) could be routed to the cloud. A NUKE render will typically be very quick and produce large frames that would need to be transferred back to the studio before preview or use. Professional render management systems can automatically send jobs to the right resource based on the type of job, or other metrics that help keep costs down.
A common workflow we see is to utilize our “Preview Frames” feature that sets the first, middle and last frame of a job to the highest priority in the queue. This allows an artist to see three representative frames of their sequence as soon as possible and before using significant render resources, especially when using cloud resources. If the sequence needs revision there is the opportunity to cancel the job, make the changes and re-submit before paying for rendering that won’t be used.
Q: Do contracts with cloud rendering services have to be in place first?
Yes, but that’s pretty straightforward these days. Major cloud providers offer free trial periods (Google Cloud Platform offers 60 days of rendering, worth up to $300 right now for free). Sign up is quick and trial accounts can be easily converted to paid accounts. If a customer wants to test the Cloud BEFORE an emergency, they can go here. TEST NOW!
The real ramp up is configuring your render pipeline to include the cloud as a resource. That ramp up includes setting up a VPN to the cloud provider, defining your cloud worker software stack, profiling your pipeline and deciding which jobs should go where and potentially contracting caching appliances to reduce data transmission to and from the cloud.
Q: What possible pitfalls do people need to be aware of? How can people avoid them?
- VPN - The VPN setup can be a little tricky and may take a few days and some tech support to get right.
- Storage - Storage can present challenges depending on how you approach it. If you keep all your data on premise things are more straightforward but some studios stage their scene files to cloud storage first in order to go wide quickly. It can take quite a few storage nodes to spread the data out enough to feed a large number of cloud render nodes. So there needs to be some tweaking to match on-premise render times by cloud nodes.
- Wasted Cloud Render Minutes - Another pitfall can be wasted cloud render minutes. Artists can be used to being able to render and re-render at will without a financial penalty if there is enough local resources. Once a studio is on the meter in the cloud it becomes more important than ever to make sure the data that is being rendered is really ready.
- Rendering Large Data - Last is trying to render everything in the cloud. Large data sets that render quickly and return large frame files are not good candidates for cloud rendering. Smaller data sets that require long render times are ideal.
Q: What about file security, especially with embargoed/industry sensitive projects?
There have been significant improvements to cloud security in the last year (in fact, two of the toughest VFX houses have recently approved the Google Cloud Platform). Cloud providers are quite secure but the transfer mechanism to and from has not always been so secure. Providers are offering secure VPN connections now to address these concerns for most customers.
Hosting your data in-house and only rendering in the cloud is another measure that can improve project security.
Q: What high-profile projects have used Qube! recently and how did it help?
Here are a couple recent customer examples:
- MPC London created unique station IDs for a major European broadcaster.
Q: How do you see cloud-based rendering services evolving over the next 12-24 months? In general, what additions and improvements to services can users expect? Why do you think this?
Services will become cheaper and easier to use. Users can expect better tools to help them configure their use of the cloud. Right now it is doable, but not easy.
Software vendors will release usage based licensing for render management and rendering. Some will be true “pay as you go” license models and some will require prepayment for blocks of hours.
- Do you have questions on how to setup your on-premise and cloud render farm?
- Do you want to test rendering in the Google Cloud?
As an Authorized Google Cloud partner, PipelineFX can help. Contact us today to get answers to your questions by submitting your request here.
