Webster’s Dictionary defines ‘latency’ as “the quality or state of being present and capable of emerging or developing but not now visible, obvious, or active.” And now that I’ve finally gotten to use that hackneyed writing trope in my career, let’s dive into what latency means in the world of video games and how the engineers working on Xbox Series X are working to reduce it.
In its most simple terms, latency is the measurement of how long it takes for a signal or piece of data to travel from one point to another in a system. For this article, we’ll be talking about input latency, which measures how long it takes a signal to go from your controller to your console, or from your console to your display. Nearly always measured in miilliseconds (ms), latency is a key element of the responsiveness and feel of a game.
There are many components that contribute to end-to-end input latency, in a long pipeline from controller input to display. For Xbox Series X, the team finely tuned each and every one of them to ensure you have the most responsive and precise controls possible.
Last week, we got a tour of that pipeline from Andrew Goossen, Technical Fellow on Xbox Series X at Microsoft, to take a closer look at the improvements the team made, starting with the controller.
In your controller, stick and trigger movement are examples of analog input. Previous controllers sampled inputs and transmitted the results every 8ms, regardless of when games actually needed the input data. So even if you quickly adjusted the analog stick or trigger in a slightly different way, the game may have received this button state change too late and, for that instant, may use your previous button state.
While this was a smart solution for both wired and wireless controllers, since it provided consistency along with good wireless battery life, it wasn’t perfect. Since it wasn’t in sync with the game’s input loop, there would be occasional latency since the game may have asked for the current button presses from the controller, but the platform delivered it later than expected.
For Xbox Series X, the team devised a solution in which the new controller samples and transmits the most up-to-date input just before the game asks for it. The result is “just in time” delivery that they call “Dynamic Latency Input (DLI)”. The system now precisely keeps the controller in sync with the game for both analog and digital inputs. The most update-to-date input arrives at the console just before it’s needed by the game. The result: significantly lower controller latency. What’s more, it’s all done without sacrificing anything in the way of battery life.
Speaking of digital inputs, it should be noted that a button press is a digital input because it has only a 0 or 1 state. Previous controllers sampled digital inputs at the same rate as the analog inputs. Again, this was consistent, but it meant that button press information was only as good as the sample rate. The team found that while games generally interpolate and anticipate analog input, they can’t do that for digital inputs. To that end, digital inputs like button presses need to be as precise as possible.
With the Xbox Series X, the controller is now constantly monitoring for and transmitting button state changes. Games can access all of the button state changes that occurred since the last time they checked. While wireless is better than ever, when a controller is wired, the team implemented the ultimate solution: as soon as a digital state changes, the data gets transmitted. No more blaming the controller when you don’t get off that final hit in a fighting game.
As for how the input gets to your console wirelessly, the team decided that our Xbox Wireless radio protocol for the controller provided too many advantages (especially in terms of compatibility with Xbox One accessories, high bandwidth, low latency, support for up to 8 controllers, and audio/voice data transmission) to consider switching to another protocol. The team also improved transmission performance by finely tuning the radio design and placement for optimal connections and reduced retries.
So that’s the controller. On the console side, the team completely redesigned the input stack on the console itself, ensuring games will get input faster and be able to access it quicker than before while also eliminating delays that could previously happen if the game itself was very busy and taxing the console.
The team also did work to empower game developers to improve and optimize game engine latency. Thanks to a new mechanism the team added, the system can now know the input state that corresponds to every frame sent over HDMI. For the first time developers can easily measure and tune their end-to-end latency during game development without the use of a high-speed camera. By decoupling rendering and input pipelines, Xbox Series X also supports 120 FPS, so games have the option of doubling their frame-rate from 60 FPS, which halves their internal latency.
One of the biggest elements in all of this is the data that flows through your HDMI cable. Microsoft is an active member of the HDMI consortium, and the team worked hard to shape the HDMI 2.1 standard to support features important for Xbox Series X in a number of areas, including:
- Transmission time to the TV: HDMI 2.1 supports up to 4k at 120 Hz which allowed the team to reduce frame transmission time from 16.6ms to 8.3ms, without sacrificing on resolution. This is an automatic 8.3ms latency improvement on TVs that support it and benefits games at all framerates.
- Variable refresh rate (VRR): When games miss their frames, VRR will show the result with the lowest possible latency. 120 Hz TVs that support VRR typically have very large timing windows which makes above 40 FPS performance free of screen tearing.
- In-TV latency: Xbox Series X invokes Automatic Low Latency Mode (ALLM) on capable TVs to automatically enable Game Mode, removing any work previously required of gamers.
- Ultra High Speed HDMI Cable: An Ultra High Speed HDMI cable is included with every Xbox Series X to ensure that players receive the premium gaming experience if they have an HDMI 2.1 compatible display.
Influencing the HDMI standard was one thing, but we also needed to work closely with leading display manufacturers to adopt them in a consistent manner to deliver the best gaming experience. The team knew the TV ecosystem needed to be ready to support these features in time for the Xbox Series X launch, but they had to answer one question: How do we get TVs to support these key next-gen features years before the first console to support them?
They answered that question by adding those features a generation early with Xbox One X and Xbox One S.This drove adoption of key features such as VRR and ALLM in TVs released in 2019 and later as opposed to having to wait for these features to start arrive in select displays beginning in 2020 or 2021. So that’s the deal with latency. While it may be difficult to notice the improved latency from any one of these improvements alone, when they all add up, it makes for a profoundly more responsive experience for players.
For more on Xbox Series X, check out our features taking a closer look at the tech powering the Xbox Series X, our interview with a member of the team that designed the new Xbox Wireless Controller, and our glossary of next-gen terms.