Argax Project: Thesis: Draft Node

Node Status: COMPLETE

Discussion

The results of this study imply the following conclusions.

IF is a daunting computer game genre. Faced with an interactive fiction game, about one third of participants failed to complete the first game, regardless of the UI used. Participant comments from those who did complete both games still indicated that IF is not everyone's cup of tea.

The traditional IF command line interface has a high input error rate. Across all textual inputs of all participants, approximately 25% of them failed to invoke an author-supported action. Approximately 20% of inputs actually produced an error message. Given that some frustrated users encountered even higher individual error rates, the mean user error is 36%. Every user of the command line interface produced at least one error or ineffective command.

However, participants' survey responses showed that this does not put everyone off of this user interface. Many still preferred the command line interface versus its Skald alternative.

Skald was easier to use than the traditional IF user interface. Skald eliminated the high number of inputs that did not map to author-supported actions. Object affordances, action affordances, and command construction were all rated as being significantly easier when using Skald. 70% of participants also rated Skald as slightly or definitely easier to use than the traditional command UI.

Skald encouraged players to explore the game world to a greater degree. This manifested in a few different ways. First, players used more game world objects. When using the Skald UI, a significantly wider range of objects were used at least once in an action by players. This was not only a one-time use, either; many of those objects were also used significantly more often. Most of the objects that saw additional use were background and scenery objects.

Second, a wider range of unique verbs was used--although this was a significant difference only when playing Captain Fate. Players are still selective, though, as no player ever used every verb, even with Skald.

Most significantly, the Examine verb got triple the use in both games when using the Skald UI. As a game author, this is rewarding: people are actually reading all the descriptions I authored! This closer examination of the world in particular--and the wider use of verbs and objects in general--suggests that authors could potentially make clues to a game's puzzles more subtle when using the Skald UI.

Skald's effectiveness did not shorten participants' game sessions. Skald maps every user input to a valid, author-supported command. However, this did not significantly affect the number of inputs entered during a game, the time spent playing, or the speed of input entry. In short, players are still spending the same amount of time and making the same number of inputs with Skald, but, instead of generating errors 20% of the time, they are exploring game world objects to a greater degree.

Skald does not eliminate all affordance or Gulf of Execution problems. Even when using Skald, unclear affordances were mentioned by a number of participants as the least enjoyable part of their experience. However, this complaint was more often vaguely phrased as "figuring out what to do" was difficult, which could be interpreted as a problem of not having enough options afforded, or having too many options afforded, or having an unclear story objective to provide the player with clear goals and intentions.

Interestingly, in the spirit of IF as a riddle and a challenge, "figuring out what to do" was also given as one of the most enjoyable aspects by some other participants.

More than just material affordances go into determining an experience of world-level agency. The poetics theory of interactive drama states that a user will experience agency provided they have sufficient material affordances and the story-level constraints needed to inform their choices. Although Skald significantly improved participants' ability to recognize both interactive objects, possible actions, and construct commands, it did not significantly impact their feelings of world-level agency. Given that world-level agency did not correlated either positively or negatively with story-level agency, it suggests that something more is going on here than this model suggests.

One likely possibility is that, if a perceived affordance is a relationship between the system and the player, then some feature of the player is also an essential part of this equation. This might be their confidence level, attention to detail, investment in the story, etc. The correlation found between the degree of previous CLI experience and the degree of reported world-level agency lends some credence to this. It seems that future work is required to clarify the full interaction between game world affordances, story level constraints, and the player's intrinsic traits.

The game itself makes a big difference. This is fairly obvious, but the game's content will generally have a greater effect on the player's overall experience than the UI. At the story level, a game is defined by the richness of the narration, complexity of the puzzles, structure of the story, and the depth of the non-player characters. Depending on the author, a game may be longer or shorter, containing more or fewer locations and objects, with those objects supporting more or fewer interactions. While the user interface may make this content more or less accessible to a player, it cannot compensate for defects in this content itself.

In terms of the two specific games used here, there were statistically significant differences in the time required to play the game, user-reported object affordances, action affordances, and the clarity of story-level objectives, regardless of the UI used. There were also some UI-specific effects. For example, Skald lead to more unique verbs being used for Captain Fate, but not for The Queen's Heart.

The effect of UI on story-level concerns is subtle. The UI used did not affect most higher-level concerns, including users' reported sense of story-level agency, experience of a story-like structure, challenges or puzzles that required some thought to overcome, or the presence of significant story-affecting user choices. While some trace of the UI seems evident in overall enjoyment, this effect was not statistically significant.

The UI does still have some subtle story-level influence, though. The use of Skald resulted in fewer participants quitting the game, and it increased the distribution among possible story endings.

I had some concern that Skald would change how users play IF by making puzzles too easy. Results here suggest that it actually makes different story paths more obvious, allowing more players to avoid the more unpleasant possible endings.

Despite its advantages when it comes to ease of use, Skald is not a clear winner with all players. Although more players favored Skald, 30% still stated that they would strongly prefer the command line interface for future games. On other comparison questions--such as which UI they found faster, more enjoyable, and more engaging--opinion was similarly distributed to the extremes. Few participants were middle-of-the-road here. This aligns with previous research (Mehta 2010, 2010) that also found that, despite the recognized limitations of free-text input, usually around half of participants still prefer it for the sense of open-ended agency that it offers.

Prior CLI experience seems to be at least one factor this in preference. Familiarity with command line environments corresponded with the reported ability to more easily construct valid commands, exert world-level agency, experience engagement, enjoy the game overall, and prefer a command line interface for future games. Interestingly, prior experience with IF did not have a significant impact on which UI a player preferred, by any of the various measures used here (easy of use, speed, engaging, etc.).

==>