The Refactor Factor - Pursuing Patterns

Refactoring

I've spent the past few months re-factoring the TennisVisuals.com codebase to support a range of new features and data sources that will surface in 2016.  It may not appear that a great deal is happening, and I certainly haven't produced what I thought I would produce in the timelines I've envisioned, but from a cold start less than a year ago I'm finally producing code that I'm willing to share with the public on Github.

The "Match Radar" had a significant overhaul and was my first use of a "reusable" and "updating" approach to D3 chart design.  The release of the source for the "reusable, updating radar chart" was picked up by "Building Widgets" and resulted in an R derivative of the chart, "d3radarR".

More significant for the future of TennisVisuals.com, however, was the redevelopment of the Points-to-Set Chart, which resulted in the creation of a "Universal Match Object" (UMO).  The UMO "understands" the structure of tennis matches and serves as a "validating" container for point data from which numerous "views" of a match may be generated. Points-to-Set is one such "view".

I recently used the UMO to validate 96,000+ matches with point-by-point data which were provided by Jeff Sackmann. (You can read about the results of that analysis here).  As I refactor my codebase the UMO has become central to the integration of disparate data sources.

A great deal of my focus of late has been re-visiting the data which Jeff Sackmann and his team of volunteers have been generating over at the Match Charting Project.  The number of charted matches has been growing by leaps and bounds.  When I started TennisVisuals.com I believed I would be far more focused on data that can be captured using applications such as Pro Tracker Tennis, but once I expanded the project to support multiple data sources the draw of the larger (and growing) MCP dataset began to exert more influence, especially as there is a broader audience for visualizations of professional tennis matches, and a greater opportunity for conversations and collaborations with others who are interested in analyzing the data.

Patterns of Play

Last week Nikita Taparia and Jeff Sackmann launched the Tennis Data Storytelling Challenge.  Since one of the goals of TennisVisuals.com is to showcase visualizations of tennis data, I elected to be one of the sponsors of the challenge.  I will showcase the top three visualizations that result from the challenge and endeavor to present them in such a way that they will be able to utilize the growing database of matches in real time.

To help promote the challenge I wrote a piece for The Tennis Notebook: "Pondering Point Patterns", and to further facilitate the exploration of MCP data I've been busy creating some MCP-specific tools.  The UMO has been central to this effort.  At this point .CSV files generated by the MCP spreadsheet are being validated by the UMO, which is then providing a standard API for querying and manipulating match data.

One of the most significant features of the Match Charting Project is that it is possible to capture not only point progressions and rally lengths, but also some details about specific strokes and shot placements during rallies.  Working with MCP data is driving the development of the UMO to a point where it will not only "understand" the structure of matches, but also the layout of the court.

Over at HeavyTopspin.com Jeff has written a number articles based on analysis of shot-specific data. One of the first articles that caught my eye (there are some color charts) examines the effectiveness of "Return-of-Serve" placement; another article looks at the tactical importance of "finding" an opponents backhand.  On the whole, however, there hasn't been many explorations of this rich portion of the MCP data.  At TennisVisuals.com I've not even touched it, until now...

Rally Tree: Point Distribution and Win Percentage

Tennis is an "intermittent" sport.  The level of intensity can vary greatly with the rally length of points and the time taken between points (among other factors including surface, ball type, sex and level of play).  When rallies are visualized they are typically depicted temporally from the first point to the last, which gives a jagged chart where it is difficult to discern any pattern at all.  "Rally Tree" is an attempt to bring a different perspective to the analysis of rallies.

 "Rally Tree" depicts the distribution of points across various rally lengths, beginning at the top with rally lengths of Zero, which indicate either Aces, Serve Winners, or Double Faults. Color coding differentiates errors where balls were "netted" vs. hit long.

 

There are several available views. The default view displays all points for a single match or selection of matches. You can filter by player to display only points served by either player (or composite of opponents).  Notice the number of winners among the points won for servers vs. those receiving.

Additionally there is an overlay depicting the percentage chance that a point was won for any given rally length. The offset vertical lines represent 50% either side of center (0%).

For the "served points" views, this gives a graphic representation of the Persistence of Server Advantage, which varies greatly among players. Please note that this is not the same as percentage of points won for a given rally length.

The "Rally Tree" graphics in this post are of Novak Djokovic's matches at Wimbledon in 2014 and 2015. The last two images depict the persistence of Djokovic's server advantage on the left and a composite of his opponents' server advantage on the left. Djokovic's dominance is obvious.  Apart from rallies of seven, he had a greater than 50% chance of winning all points with rallies up to sixteen.  His opponents' composite server advantage only extended to rallies of five.

You can play around with a live version of Rally Tree and explore your favorite players at TennisVisuals.com

In the near future "Rally Tree" will be integrated with "Game Tree" and other TAVA components so that selections in one component can drive views in another.  For instance, a "Point Progression" from 0-0 to 0-15 can be selected in "Game Tree" to view the distribution of points in the "Rally Tree" or "Points-to-Set".  From this point it will be possible to explore whether there are certain points in a match when rally lengths increase...

To read more about "Persistence of Server Advantage" please follow the link to Jeff Sackmann's blog post on the topic.

Game Tree: Point Progression

"Game Tree" is a depiction of Point Progression for a selection of games within a tennis match or across a series of tennis matches; it is a Sankey Diagram and possesses the "Markov property", meaning that the set of future "states" that are possible are constrained by the current "state", the point score at any moment in a game. Here is a nice interactive explanation.

"Markov Chains" have often been applied to tennis games.  Google "Markov Tennis" and you'll find a large number of articles on statistics which use Tennis to explore probability.  A few of the results use Data Flow Diagrams to depict Point Progression: Wolfram Alpha has an attractive visualization (see above) which was reproduced in an article on Predictive Modeling; and NC State University produced a YouTube video, as part of an online course titled "Introduction to Finite Math", with a whiteboard explanation of state transitions in tennis games.  When visualizations are provided they are usually arranged like the GameFish in TennisVisuals.com, with either horizontal or vertical orientation:

As far as I can tell, the Game Tree design created by Damien Saunder and David Webb at GameSetMap.com is the first time a Sankey Diagram (or Harness Flow Map) was applied to Point Progression in Tennis. The primary innovation was to apply the idea of "quantitive flow lines" to the possible point paths through the tree such that the width of each line represents the frequency with which games passed through each possible "state" for the score, but the real power of the design comes from its interactive nature.  SVG (Scalable Vector Graphics) are used to:

1. animate exploration of the data when it is filtered by selecting individual games, groups of games, or constraining the games to only service games for a chosen player
2. provide contextual information when "hovering" over specific elements

The original implementation of Game Tree, presented as a celebration of Nadal's 2013 comeback, used match data downloaded in XML format from the William Hill Sports betting website.

In the TennisVisuals version of Game Tree, data is retrieved in JSON format from the Mongo database which underpins TennisVisuals.com.  That data, in turn, is presently sourced from Jeff Sackmann's Match Charting Project (many other data sources will come online soon).

The inspiration for the Game Tree design seems to have been the same frustration that drove the development of the Points-to-Set chart: the final score of a tennis match reveals very little about how close a match actually may have been. Even a match with a 6-0, 6-0 score may have been "hotly contested".  Traditional stats miss the story every time. Percentage of Points Won for a 6-0, 6-0 match, for instance, provides only a very crude view of match intensity - ranging from 100% for complete dominance by one player to 62.5% for a match in which every game reached Deuce once and only once - it relates nothing of the drama and is of very little use for constructive analysis.

In the following Game Tree visualization of Nadal's service games in a match against Wawrinka at the 2013 Madrid Masters, it is easy to see that Nadal won the first point of his service games 77.8% of the time.  When he did lose the first point in a match, 100% of the time he won the second point.

With Game Tree it is possible to see how often Deuce was reached during a match; the thickness of flow lines even indicates how often game scores ricocheted between Deuce and Advantage.  In the match with Wawrinka, for games both served and received, Nadal lost only one game that reached Deuce:

In the Saunder/Webb implementation of Game Tree, the "Nodes" of the tree are color-coded to indicate momentum.  Dark nodes represent positive momentum while Red nodes represent negative momentum.  In the TennisVisuals version of Game Tree these representations still hold true, but momentum is always viewed from the perspective of the primary player; when filtering for the opponent's service game, the tree is not "flipped", as occurs in the Saunder/Webb version of the Nadal-Djokovic Roland Garros 2014 final.

The relative importance of each point in tennis games, sets, and matches has been analyzed extensively, most famously by Carl Morris in his article "The most important points in tennis", which was published in Optimal Strategies in Sports in 1977.   It is probably impossible to publish analysis of points in tennis without referencing Morris...  here is one of many studies, notable for its visualization of relative point importance within the context of a set:

In 2014, Professors Franc Klassen and Jan Magnus provided ample coverage of the topic in their book "Analyzing Wimbledon".  Most recently Jeff Sackmann wrote a series of blog posts ("How Important is the First Point of Each Game?", "The Pivotal Point of 15-30") drawing on a theoretical model which he has published and utilizing his extensive match database.

My plan is to integrate the insights garnered from such analyses into the TennisVisuals version of Game Tree so that results for each match can be viewed in the context of benchmark figures. I'd like to auto-generate summary reports to go along with Game Tree visualizations, similar to what Saunder has done for Nadal-Djokovic 2014 Roland Garros Final.  I also plan to divide each point "flow line" into errors and winners and highlight "clutch" performance.

Shortly after releasing the initial version of Game Tree, Saunder published a follow-up entitled "Where are you most likely to win a point on Nadal's serve?" In this article he introduced a "Proportional Symbol Game Tree" which shows the percent chance at every possible "state" of the score that an opponent had of winning the point.  It's enticing to think about using a similar Game Tree to visualize a player's service game performance in one match relative to their average over the course of the past year...  perhaps overlapped Proportional Symbols of reduced opacity...

Makeover

Today TAVA moved to a new domain: TennisVisuals.com

There are now proper Instructions for using the latest version of the interface; Examples, which had previously only been noted on Twitter, now have their own page.

Most of the changes I've been working on haven't yet surfaced, but the project has graduated from a hobby hosted on my brother's minimally configured server, which I was capable of crashing, to a true application having a real, and expandable, home in the cloud.

I will be posting new visual elements to the Examples page before they are integrated into the application.  The Examples page will also be a place to gather unique visualizations made by using the yet-to-be-published API, which will make a growing database of ATP and WTA matches generally available.

Personal matches charted using ProTracker Tennis will not be made public.  If you use ProTracker Tennis, you can mail your matches to tennis.aip 'at' gmail.com.  You will receive a link by email to the TAVA visualization of your match.