Repairing Misperceptions of Words Early in a Sentence is More Effortful Than Repairing Later Words, Especially for Listeners With Cochlear Implants

Repetition accuracy was scored in real time by the experimenter and participant responses were manually entered into a data-tracking spreadsheet after the experiment visit for further analysis. For trials where a target word was replaced with noise, any response that was not semantically coherent with the stimulus was counted as an error, as well as any errors elsewhere in the sentence. If the participant's guess at the word replaced by noise was not the “intact” version of the word but still made sense (e.g., “Please clean the floor with this broom”, instead of “Please sweep the floor with this broom”), it was counted as correct. In the case of intact sentences, the target word was defined as the word that would have been masked by noise in the alternate version of the stimulus, to facilitate fair comparison across stimulus types. We also tracked whether participant responses were linguistically coherent, and the presence of multiple errors within trials. An example of an incoherent response would be “The plant hit the soccer ball with the door” (see Winn & Teece, 2021 for further discussion of incoherent responses). The goal of evaluating repetition accuracy in this way was to track whether participant responses had any errors, rather than only focusing on the number of errors within the response. This approach was taken specifically because the words in high-context sentences are not independent; multiple errors within a sentence would not be a conclusive sign that multiple words were misperceived. For example, misperception of a word might result from the listener trying to create coherence with an earlier word that was misperceived, and participants tend to produce these secondary errors when trying to resolve linguistic ambiguity. Winn and Teece (2021) and Gianakas et al. (2022) provided evidence of this effect in both forward and backward directions within the sentence and suggested that a secondary error tends to reduce effort because it promotes coherence.

To evaluate the differences in overall intelligibility between listener groups, errors were estimated on a per-trial level using a binomial (i.e., logistic) mixed-effects model that included fixed effects and interactions between condition (sentence type) and hearing, as well as random intercepts and correlated random effects of condition per listener. Estimated marginal means were calculated from this model to statistically compare the difference in intelligibility scores across hearing groups and express them in plain terms. Intelligibility scores between YTH and OTH listeners were extremely similar (96.4% and 96.6%, respectively) with no statistical differences observed between the groups and thus were treated as a singular group for the intelligibility analysis. The following model formula was used in the prevailing model:where “AnyError” refers to making a mistake on any word in the sentence and getting the sentence incorrect, condition is the different sentence types (early-masked, late-masked, and fully intact) and hearing refers to TH or CI listeners.

Although tracking any error in the participants’ verbal responses already reveals the impact of stimulus type and listener group, there is additional information to be gained from analyzing the different types of errors that were made by CI listeners and how those errors may be related to mental repair. Sentence repetition scores were analyzed in more depth for the CI group using a series of GLMMs that estimated various outcome measures, including (1) the presence of any error within the response and (2) errors on words other than the target word. These models were restricted only to CI listeners because TH listeners did not make many errors, resulting in implausibly high or low beta estimates due to model estimates including values at or close to 0. The model for estimating the presence of an error on words other than the target had the same structure as the model for any errors. Other types of errors, such as target word errors and incoherent responses, were also counted; however, these errors were not frequent enough to be statistically evaluated. The model formula was declared as follows:For the two models described above, when a specific comparison was not available in the original model because both sides of the comparison were deviations from the default (and therefore not directly compared to each other), comparisons were obtained by rotating the same model with the default reassigned, rather than running a post hoc model limited to the specific comparison of interest. For example, to make the comparison of early-masked versus late-masked sentences, a different model was used with the late-masked sentences as the default condition to allow for this direct comparison.

Pupil data were processed as described by Winn et al. (2018) and Winn and Teece (2022). Blinks were detected as a decrease in pupil size to 0 pixels, with the stretch of time corresponding to the blink expanding backward by 80 ms and forward by 120 ms to account for the partial occlusion of the pupil by the eyelids during blinks. The signal was low-pass filtered at 5 Hz using a fourth-order Butterworth filter and then down-sampled to 25 Hz. The baseline pupil size was calculated as the mean pupil size in the time spanning 500 ms before stimulus onset to 500 ms after sentence onset. Each pupil size data point in the trial was expressed as the proportional difference from the trial-level baseline.

Trials were discarded if 30% or more data points were missing between the start of the baseline to 3 s past the onset of the stimulus. CI listeners on average had fewer trials discarded due to missing data (12.7%) and less variation among individuals (s.d. of 1.9%) compared to TH listeners (average of 19.6% trials discarded with s.d. of 1.5%). Other outliers and contaminations were automatically detected through an algorithm that accumulated multiple “flags,” many relating to high-intensity low-frequency fluctuations (hippus) activity during baseline. Flags included baseline slopes that deviated by more than 2 s.d. from other baselines in the block, significant slope of change in pupil size during the baseline, or a reduction in proportional dilation by at least 1.5% immediately after the stimulus onset (indicating strong constriction when dilation was expected). Proportional dilations of 40% or more were flagged, given the expected maximum around 30% (greater proportional dilations usually indicate a contamination during baseline). Three or more flags resulted in a trial being dropped. If a participant had fewer than 12 trials remaining in any condition following outlier detection, that participant's entire data set was dropped. One CI listener and two TH listeners were excluded from analysis for this reason, leaving 61 total listeners to be included for analysis.

Our goal is to quantify differences in the timing and duration of listening effort when listeners have to mentally repair a missing word. To achieve this goal, filtered data that were summarized for each individual in each stimulus condition were estimated using generalized additive mixed-effects models (GAMMs; van Rij et al., 2019). One of the distinct advantages of using GAMMs is the ability to identify stretches of time where there is a meaningful difference between curves, and this can be done during the entire time course of the pupil response during listening and linguistic processing. GAMMs model the data using a combination of Gaussian basis functions that are summed in weighted combination to match the shape of nonlinear data (e.g., the pupil response) allowing for statistical analysis of the entire pupil response function without the need for different analysis windows. The number of basis functions to calculate each smooth function can be specified for each predictor variable and each interaction term, as well as the specified random effects. Another advantage of GAMMs is accounting for the autocorrelation of time-series data (Baayen et al., 2016) or the tendency for the data point at time t to be similar to its preceding data point at time t − 1, which is problematic because it increases the probability of Type I errors. GAMMs have previously been used to model pupillometry data in studies of listening effort (Boswijk et al., 2020; Porretta & Tucker, 2019; Winn, 2024). The details of the GAMMs model presented here are below, and we refer to van Rij et al. (2019) for a more thorough overview of using GAMMs to analyze pupillometry data.

All the models and statistical analyses were executed in R (R Core Team, 2021) and R Studio (RStudio Team, 2020). GAMMs were implemented using the R package “mgcv” version 1.8-42 (Wood, 2023; Wood, 2017), and the R package “itsadug” version 2.4.1 (van Rij et al., 2022) was used for interpretation, validation, and visualization of the statistical analyses. For each model, an initial model was used to calculate the autocorrelation lag value (rho) that would be used in the final model. Model terms included hearing group (CI, OTH, and YTH) and stimulus type (early masked word, late masked word, and fully intact) as fixed effects with different smooth functions fitted over time for each interaction of group and stimulus type. There were random effects of time as a smooth factor for each listener for each stimulus type. Multiple models were computed with different default groups and default stimuli to evaluate across all of the comparisons of interest. An example of a final model terms is shown below, with YTH as the default hearing group and fully intact sentences as the default stimulus-related effects.

bam(pupil ∼

   # parametrics

   is_CI + is_early + is_late + is_early_CI + is_late_CI +

   is_OTH + is_early_OTH + is_late_OTH +

   # basic smooth for time

   s(time, k = 20, bs = “cr”) +

   # difference curve for hearing group

   s(time, by = is_CI, k = 20, bs = “cr”) +

   s(time, by = is_OTH, k = 20, bs = “cr”) +

   # interactions of condition x hearing

   s(time, by = is_early, k = 20, bs = “cr”) +

   s(time, by = is_late, k = 20, bs = “cr”) +

   s(time, by = is_early_CI, k = 20, bs = “cr”) +

   s(time, by = is_late_CI, k = 20, bs = “cr”) +

   s(time, by = is_early_OTH, k = 20, bs = “cr”) +

   s(time, by = is_early_OTH, k = 20, bs = “cr”) +

   # random time smooth per listener

   s(time, Listener, bs = ‘fs’, m = 1, k = 5) +

   # random time smooth per listener interacting with condition

   s(time, Listener, by = is_early, bs = ‘fs’, m = 1, k = 5),

   s(time, Listener, by = is_late, bs = ‘fs’, m = 1, k = 5),

   # inputs for computational efficiency

   method = “fREML”, discrete = TRUE, family = “scat”, nthreads = cores_to_use,

   # account for autocorrelation of each timepoint in the data

   AR.start = start_event, rho = 0.957,

   data = df)

In this model formula, variables denote binary designations. Terms “is_CI” and “is_OTH” categorize the listener into their respective listening groups, “is_early” and “is_late” specify the sentence type, with the other variables representing a combination of these contrasts. For example, “is_early_CI” denotes a trial where an early-masked sentence is presented to a CI listener. This binary notation permits the estimation of the additional effect of CI on the main effect of the early-masked sentences. K represents the number of knots in the combined basis function, with smaller k for random effects. Rho refers to the level of autocorrelation. Family = “scat” refers to the model using the scaled t distribution to determine statistical significance.

Figure 5 shows the percentage of errors by CI listeners for each specific error types, along with a panel showing data for the previous tally of any error. TH listeners did not make enough errors to warrant analysis. The number of true target errors for CI listeners (not correctly repairing the masked word, or in the case of intact sentences, making an error on the word that would have been masked) was small enough that no statistics were conducted. A raw count of target-word errors for CI listeners revealed more errors for sentences with early (n = 45) or late (n = 26) missing words compared to those same words when the sentence was fully intact (n = 10). The second panel shows that the number of times that listeners made an error on the target word was rather similar across the stimulus conditions. That is, even when the sentence was intact, there were some mistakes at a rate roughly comparable to when the words were physically replaced with noise.

The third panel of Figure 5 illustrates how mentally repairing a missing word affected perception elsewhere in the sentence. Compared to when the sentences were fully intact, there were more errors on nontarget words when CI listeners had to repair earlier (β = 1.02, z = 4.27, p < .001) or later (β = 0.69, z = 2.86, p = .004) missing words. Although CI listeners made more errors elsewhere in the sentence when forced to repair an early missing word (n = 112) versus a late missing word (n = 86), this difference did not reach the conventional criterion for statistical significance (β = 0.33, z = 1.715, p = .086).

No statistical comparisons were made for incoherent response, although they occurred more often when repairing early (n = 30) or late (n = 20) missing words compared to when the sentence was intact (n = 7).

Changes in pupil dilation for each group when listening to the different sentence types are shown in Figure 6. The results of the GAMMs analysis are shown as colored bars at the bottom of each panel, which denotes regions of statistical differences between the curves colored by which sentence type resulted in a larger increase in pupil dilation. Model summary tables can be found online as Supplemental Digital Content 1.

The largest effects observed were for stimuli with missing words early in the sentence, which elicited greater pupil dilation across multiple comparisons of interest. First, larger increases in pupil dilation were observed for sentences with early-masked words compared to intact sentences, with each listener group showing a similar duration of increased pupil dilation that is consistent with the timing of the missing word in the sentence (YTH: −1.40 to 4 s relative to sentence offset; OTH: −1.46 to 4 s relative to sentence offset; CI: −1.24 to 4 s relative to sentence offset). Second, sentences with early missing words also elicited greater pupil dilation than sentences with later missing words, but the duration of this difference varied by age and hearing status. Greater pupil dilation for early- versus late-masked words was observed for younger TH listeners only during the listening phase of the trial (between −1.72 and 0.52 s, results reported as time relative to sentence offset), with no differences being observed after the peak in pupil dilation that corresponds to sentence offset. Older TH and CI listeners also showed a similar duration of increased pupil dilation during sentence presentation (OTH: −1.64 to 0.68 s; CI: −1.56–0.68 s). However, unlike the YTH group, the CI and OTH groups showed increased pupil dilation in response to early- versus late-masked stimuli that persisted after sentence offset as well (1.24 for CI, 1.56 for OTH, both extending to the end of the analysis window at 4 s).

Sentences with late-masked words elicited greater pupil dilation compared to fully intact sentences after the end of the sentence presentation (for YNH: between −1.88 and −1.64 and −1.48 and 0.12 s; for OTH: between −0.60 to −0.04 s; and for CI: between −1.32 and −0.20 s). However, the fully intact sentences unexpectedly elicited larger increases in pupil dilation during sentence presentation compared to the late-masked sentences for at least some stretch of time for all groups. An additional analysis of the data verified that this pattern was not dependent on the type of stimulus presented in the previous trial.

Whereas each group demonstrated increased pupil size in response to stimuli that demanded mental repair compared to sentences that were intact, the degree of this increase was different across groups. The difference of pupil dilation between repair conditions and intact conditions was compared across groups using a GAMM that included interaction terms between stimulus type and hearing groups. Figure 7A shows the modeled differences of pupil responses within groups for each stimulus comparison (left three panels; significant stretches already described above), and the comparison of these difference curves between hearing groups (Figure 7B, right panels).

For both YTH and CI groups, there were increases in pupil dilation resulting from early-masked versus intact sentences, for early-masked versus late-masked sentences, and also for late-masked versus intact sentences (left and right column of Figure 7A). All of these increases were significantly larger for the CI group (middle column of Figure 7B). For both OTH and CI groups, there were increases in pupil dilation resulting from early-masked versus intact sentences, for early-masked versus late-masked sentences, and also for late-masked versus intact sentences (middle and right column of Figure 7A). Just as for the comparison of CI to YTH, the increase in pupil dilation for early-masked sentences compared to intact sentences was significantly larger for CI listeners compared to OTH listeners (top right panel Figure 7B). However, the increased dilation observed for early-masked versus late-masked stimuli and the increased dilation for late-masked versus intact stimuli both were not statistically different across the CI and OTH groups (rightmost column of Figure 7B, middle and lower panels). Within the TH group, the increase in pupil dilation resulting from early-masked versus intact stimuli was larger for the older listeners (top left panel in Figure 7B), but no other differences emerged across age groups within the TH sample. Taken together, these results suggest that there is an effect of age and that when accounting for age, there remains a separate effect of using a CI for mentally repairing words early in a sentence.

Incorrect responses tend to result in a larger or more sustained increase in pupil dilation (Winn et al., 2015; Zhang et al., 2021), and the difference in performance scores across stimulus types invites analysis of intelligibility effects on the current data from CI listeners (there were not enough incorrect trials for TH listeners to analyze). Pupil responses for the CI listener group for correct and incorrect trials for each sentence type are shown in Figure 8. The elevation in pupil size was observed when sentences demand repair is sustained for a longer amount of time when the repair was not fully successful (i.e., when there was still a mistake in the response), compared to when the word was correctly inferred. For sentences with early-masked words, incorrect responses led to increased pupil dilation from 0.99 to 4 s relative to sentence offset, which was a longer duration than the corresponding effect for errors in sentences with later-masked words (1.33–4 s relative to sentence offset). This result is consistent with generally larger effects of early mistakes in semantically coherent sentences (Gianakas et al., 2022; Winn & Teece, 2021). For sentences that were presented fully intact, the pattern of sustained pupil dilation for incorrect responses was only briefly different than when the response was correct (2.03–3.13 s relative to sentence offset), although fewer errors were made for those stimuli overall (n = 49) compared to sentences with an early-masked word (n = 150) or a late-masked word (n = 116).

The CI listeners showed prolonged elevated pupil dilation following early-masked trials and also showed a higher rate of mistakes on these trials. Therefore, there was a possibility that the effect of hearing was simply an expression of what happens when any listener makes a mistake, regardless of hearing status. We repeated the analysis using only trials with correct responses, confirming that the effect of hearing (prolonged elevated pupil dilation in CI listeners compared to YTH listeners following trials with early-masked words) persisted. Supplemental Digital Content 1 contains the results of this follow-up analysis.