What went wrong with the Spark Sport's Rugby World Cup live-stream of the first All Blacks game and its quest for redemption.
How terrible would that be?
Pretty terrible. Devastating for true basketball fans during the US' March Madness, the two weeks of tournaments that determine the champions of college basketball. The scenario above is taken directly from a case study for Las Vegas-based iStreamPlanet, which Spark Sport contracted to deliver the Rugby World Cup to hundreds of thousands of paying Kiwis.
In 2017, iStreamPlanet’s director of software engineering Dan Penn proudly recounted how it successfully delivered millions of streams to basketball fans all over the US: “I am extremely happy to say that throughout the entire tournament there were no major problems… and the few minor bumps introduced by external factors were handled by the system and exposure was mitigated."
If only the same could be said for Saturday night’s video streaming failure during the All Blacks-Springboks Rugby World Cup pool game, which affected at least 6,000 subscribers. It wasn’t quite as catastrophic as iStreamPlanet’s hypothetical scenario, but given the importance of the game, it came pretty close.
Points of failure
Spark Sport isn’t saying if iStreamPlanet was responsible for the failure, which saw subscribers plagued with buffering issues, pixelation or the picture cutting out completely.
“We’re not naming partners. The fact remains that it’s a Spark Sport service and regardless of who our partners might be, it’s our responsibility to get this right for our customers,” Spark Sport spokesperson Ellie Cross told NOTED.
Spark did confirm that iStreamPlanet’s recent use of cloud computing to run its services, including for March Madness, wasn’t responsible for the issues. Operating services from cloud computing networks allow iStreamPlanet to scale up server capacity as needed, using the data centres of cloud computing companies. But there are risks in handing off to third-party infrastructure, as iStreamPlanet’s CEO Mio Babic noted following this year’s March Madness.
“The cloud is maturing very rapidly, but there are still, sort of, things that occasionally just happen in the cloud environment that can be concerning,” he said in an interview on the company’s website.
“You have to have the right tooling, processes, and architecture to make sure that you’re not exposed to some of those things.”
Another candidate potentially responsible for the technical failure is Akamai, a major player in internet content distribution with 240,000 servers internationally, nearly 800 of which are used for media delivery and storage in New Zealand alone.
It specialises in providing server capacity so companies can offer customers spread far and wide quick access to their content. All of the big players, from Youtube and the BBC, to ESPN and Apple, use Akamai or its competitors to achieve global reach.
“Our distributed architecture means we're closer to your audience, and can effectively bypass common interruptions to provide a consistent, high-quality online media experience to your audience...the kind they expect to have,” the company claims.
Akamai and iStreamPlanet regularly team up to work on streaming big US sports events, so Spark Sport was opting for some gold-plated partners when it signed them up to deliver the Rugby World Cup.
So what went wrong?
Spreading the load
“To explain what happened in a little more detail: the provider had set up their international network to bring Spark Sport traffic into and through New Zealand through multiple points,” explains Cross.
“What became apparent during the New Zealand vs South Africa match is that the traffic wasn’t distributed in the most efficient way – the spread had not been optimised. There was not a lack of capacity, but the load was not spread efficiently.”
That explains why on Saturday night, some subscribers had a flawless experience while others, like me, connecting via my Freeview SmartVu dongle in Wellington, were left with constant drop-outs. Those with fibre connections plugged straight into their computer saw the stream buffering, while others on barely viable copper connections had a flawless feed. It really was the luck of the draw – how well-balanced the load of ones and zeros you were receiving happened to be.
The video content encoded and prepared by iStreamPlanet to play in the Spark Sport app was sent to Akamai for distribution to its servers in New Zealand and on through our various broadband providers to viewers’ devices. Spark says the problem wasn’t related to how Akamai’s local servers were set up, but the way the video was configured leaving the US.
Spark Sport had more than enough capacity to handle the 132,000 concurrent streams that were watched at one point during the game. In fact, the network was apparently designed to handle up to 500,000 streams.
But network capacity means nothing if you don’t manage the flow of traffic across that network’s nodes properly. Spark Sport video met unexpected stability issues in a live environment where literally every millisecond counts.
These are the fine margins video streaming companies are working for when it comes to live events. Netflix, Youtube and other streaming services use large content delivery networks to store cached copies of video content at points all over the world to spread the load and speed delivery to internet users. Optimising them is a huge priority as it can mean the difference between the annoying buffering ‘ring of death’ on your screen and a smooth, high-definition or even ultra high-definition (4K) video feed.
These days, that generally works fine when you are calling up an episode of Stranger Things on Netflix. Multiple copies are already stored on a server close by and are ready to stream. For a live rugby game, multiple copies go to different servers on the network, but they have to be handed straight on to viewers in a race against time. Any significant delay causes problems and with record numbers of people demanding the content at the same time, any mistakes are amplified massively.
This is what makes live content delivery over the internet such a white knuckled gamble. Consider the logistics that Spark Sport faces in getting the RWC games to our screens. The video feed from cameras in the stadiums leaves Japan and heads to TVNZ’s studio in Auckland.
There, adverts and studio content, such as the game build-up commentary, are added before the ‘source feed’ is sent to iStreamPlanet in the US for video encoding to meet the needs of the Spark Sport app. Then it is sent back to New Zealand for distribution to the broadband network via those Akamai servers.
Every millisecond counts
It is a dizzying trip around the Pacific which explains the 40-second delay between live TV coverage on TVNZ and Spark Sport’s internet feed. Ten years ago, attempting to live-stream an international event here at such scale was unthinkable. The core networking wasn’t there, nor were the broadband connections for us to receive the video.
But our server infrastructure and international connectivity have improved to the point where live-streaming of big international events is a viable prospect – if everything is configured correctly. Companies that can deliver the experience stand to win a share of the lucrative subscription and ad revenue that previously was locked up by Sky TV.
The saving grace on Saturday night was Spark’s relatively quick move to get broadcasting partner TVNZ to begin screening the game on its Duke channel, live and free to everyone.
That call saved Spark Sport the full wrath of angry customers, who took to Twitter and Spark Sport’s Facebook page to vent their displeasure at the poor streaming quality. At least those of us affected were able to watch an uninterrupted second-half of rugby, which revealed the All Blacks to be at their disciplined and clinical best.
But the incident has no doubt dented confidence in live-streaming. The extra complexity involved in connecting the right device over broadband was always going to push the viewing audience to their limit anyway. The extra effort involved in homes, clubs and pubs all over the country to accommodate that was only ever going to be tolerated if Spark Sport delivered an experience as good as free-to-air TV, or what could be expected from the company everyone loves to hate – Sky TV.
A shot at redemption
Spark is now one more botched live-stream away from its sports streaming business aspirations evaporating completely. The games shown since Saturday have been delivered without incident, though the audience numbers have been much lower. The next big test will likely be this Sunday’s Wales vs Australia game, which may again see over 100,000 people log on.
With nearly 40 games left to show, Spark Sport has a shot at redemption. It can rescue its Rugby World Cup fortunes. But the fact remains that the crucial test of matching or bettering the conventional means of delivery we all know hasn’t been confidently proven yet.
Deputy Prime Minister Winston Peters reflected that view when he angrily described the streaming problems as an “abject disaster”. Rugby is a cultural touchstone for us; All Blacks tests are events that stay in the public’s consciousness for decades. Imagine if Stephen Donald’s fateful penalty kick for victory in the 2011 World Cup final had been interrupted by a buffering symbol.
There was low-level discontent across social media about varying screen quality on Spark Sport, well before the RWC started. I’ve noticed it myself – fleeting fuzziness and barely noticeable glitches that take you out of the viewing experience. It’s tolerable but begs the question – why not just use the tried and tested Freeview network we’ve all subsidised as taxpayers?
Eventually, live-streaming of big events will include interactive elements that the internet is best placed to deliver, such as multiple camera angles, in-game statistics, social media feeds and personalised ads tailored to the viewer’s tastes.
Watching a game online will have all the richness we’ve come to associate with smartphone apps and social media.
Get the basics right
But all of that is still years away. To get to that point, our streaming providers first have to prove they can get the basics right for live-streaming – a reliable, high-quality and uninterrupted stream to hundreds of thousands of people.
If Spark Sport can’t use the rest of this tournament to cement the first step in its grand plan for streaming, it should withdraw and let the usual suspects, Sky TV and the free-to-air broadcasters, reclaim responsibility – and the sports broadcasting rights that go with it.
For now, Spark Sport is sticking with its plan, confident it has a permanent fix in place.
“Since identifying the issue, the provider has made a number of changes to optimise the paths so that Spark Sport traffic flows correctly into and through New Zealand,” says Cross.
Spark’s streaming ambitions now rely on it.