Neural Activity Underlying the Detection of an Object Movement by an Observer During Forward Self-motion Dynamic Decoding and Temporal Evolution of Directional Cortical Connectivity

Neural activity underlying the detection of an object movement by an observer during forward self-motion: Dynamic decoding and temporal evolution of directional cortical connectivity A B S T R A C T Relatively little is known about how the human brain identifies movement of objects while the observer is also moving in the environment. This is, ecologically, one of the most fundamental motion processing problems, critical for survival. To study this problem, we used a task which involved nine textured spheres moving in depth, eight simulating the observer’s forward motion while the ninth, the target, moved independently with a different speed towards or away from the observer. Capitalizing on the high temporal resolution of magne-toencephalography (MEG) we trained a Support Vector Classifier (SVC) using the sensor-level data to identify correct and incorrect responses. Using the same MEG data, we addressed the dynamics of cortical processes involved in the detection of the independently moving object and investigated whether we could obtain con-firmatory evidence for the brain activity patterns used by the classifier. Our findings indicate that response correctness could be reliably predicted by the SVC, with the highest accuracy during the blank period after motion and preceding the response. The spatial distribution of the areas critical for the correct prediction was similar but not exclusive to areas underlying the evoked activity. Importantly, SVC identified frontal areas otherwise not detected with evoked activity that seem to be important for the successful performance in the task. Dynamic connectivity further supported the involvement of frontal and occipital-temporal areas during the task periods. This is the first study to dynamically map cortical areas using a fully data-driven approach in order to investigate the neural mechanisms involved in the detection of moving objects during observer’s self-motion. 1. Introduction Rapid and accurate visual detection of moving objects is critical for safe and successful interaction with the environment. When the observer is stationary this is straightforward, because only the movement of objects is encoded in the motion on the retina. However, when the observer is also moving, motion is added to the already existing retinal motion and therefore the detection of moving objects in the scene becomes more difficult. Determining how the brain can solve this problem, rapidly and accurately, is of tall importance for understanding how we can safely interact with our environment and successfully accomplish a large number of everyday activities. Those include walking through a dynamic scene, crossing the street in traffic, driving among other moving vehicles such as cars or bicycles, skiing or when playing many other sports and when we want to intercept or avoid objects moving towards us while we are also moving. In these situations, a straightforward way to detect the motion specific to the object movement is to subtract the pattern of retinal motion due to the observer’s self-motion from the complex retinal motion. For example, the observer’s forward motion produces a radial, global pattern of retinal motion referred to as optic flow (Gibson, 1950, 1954; Warren and Hannon, 1988). Visual flow resulting from self-motion can be estimated by using both visual, retinal, and non-visual, extra-retinal cues. Several studies of human psychophysics and of physiology in non-human primates (Gogel, 1990; von Holst and Mittelstaedt, 1950; Wallach et al., 1974), including recent studies by Wexler’s (e.g. Wexler et al., 2001) and DeAngelis and Angelaki’s groups (e.g. MacNeilage et al., 2012) have explored how non-visual, extra-retinal cues about self-movement such as efference copies of motor command, vestibular, or proprioception can be distinguished from the retinal motion which resulted from objects moving in the environment. While these studies demonstrate that the extraretinal information provides helpful cues to an observer in selfmotion to detect moving objects, there is also evidence that this information is not precise enough for the accurate estimation of scenerelative movement of objects (e.g. Tcheang et al., 2005; Wallach et al., 1974; Wexler et al., 2003). Furthermore, the non-visual cues are not always available during self-motion, such as for a passenger in a car that moves at constant speed, or are unreliable, such as the motion experienced by an airline pilot (Gibb et al., 2010). A solution to these types of problems was proposed by Rushton and collaborators (Rushton et al., 2007; Rushton and Warren, 2005) who set out to investigate whether visual only, retinal information, specifically optic flow, can dissociate retinal information resulting from object motion from retinal information due to observer’s self-motion. They proposed the Flow Parsing (FP) hypothesis, which suggests that the radial optic flow field provides a reliable visual-only cue to an observer in forward self-motion which can compensate for the associated retinal motion. After parsing out the optic flow from the retinal image, what is left is the motion of the objects so that they can be detected by an observer in self-motion (Rushton et al., 2007; Rushton and Warren, 2005; Warren and Rushton, 2009a, 2009b, 2007). Although psychophysical aspects of flow parsing were extensively studied, the neurophysiological underpinnings of flow parsing have not yet been fully elucidated. fMRI studies (e.g. Arnoldussen et al., 2013; Cardin et al., 2012; Galletti and Fattori, 2003; Pitzalis et al., 2019, 2015, 2013, 2010) suggested the importance of several cortical areas (e.g. V3A, lateral occipital region (LOR), human motion complex (hMT+), V7, intraparietal sulcus (IPS)) for the extraction of object motion from the observer’s self-motion information. In an fMRI study Calabro and Vaina (2012) computed partial correlations to investigate the cortical networks that interact in solving this type of tasks. They performed cluster analysis which revealed four distinct cortical networks: the first, consisted of the bilateral early visual cortex (EVC, involving V1 and V2), the second network involved the LOR, V3A, V7, kinetic-occipital region (KO/V3B) and hMT+, the third network contained areas responding to visual motion in the parietal lobes (intraparietal sulcus (IPS), ventral intraparietal area (VIP) and dorsal intraparietal sulcus medial (DIPSM)) and the right hemisphere precuneus, while the fourth included several higher level areas around the central sulcus (CS), namely, precentral sulcus (preCS), postcentral sulcus (postCS), subcentral sulcus, ventral CS and frontal eye field (FEF). These cortical areas were suggested to represent the neural substrate supporting the use of scene context in detection of a moving object. Although these fMRI studies make an important contribution towards pinpointing the anatomical and physiological substrate of the detection of moving objects by an observer in forward self-motion, because they measure hemodynamic changes in activated brain regions, they cannot capture the temporal neural dynamics of the computations involved. Therefore, here we used magnetoencephalography (MEG) capitalizing on its high temporal resolution to investigate the neural substrates and their dynamics, necessary for detecting a moving object while parcelling out the observer’s self-motion. We used the same task as in (Calabro and Vaina, 2012, 2011), which involved nine moving textured spheres distributed at different depth levels in the 3D space in front of the observer, eight simulating the observer’s forward motion while the ninth, the target, had an additional independent motion component moving with a different speed towards or away from the observer. Unlike most studies of decision-making processes which investigate the principled combination of speed and accuracy, our experimental paradigm is a Fixed Stimulus Duration (FSD) task where the presentation of the stimulus motion is followed by a delay/working memory period before observers can enter their response. No feedback was provided on the correctness of observer’ decision. Here we use a fully data-driven machine-learning method to predict the response correctness at each time point during a trial. We contrasted the activity patterns revealed by the machine-learning approach with the evoked brain activity elicited by subjects performing the task. Using the salient evoked activity, a set of regions of interest (ROI) was selected and dynamic connectivity was computed to address the question of what is the neural dynamics underlying the detection of an independently moving object when the observer is moving forward. We are using a multistage approach to address the time-varying cortical processing of this task. 2. Methods 2.1. Participants MEG data were collected from 9 healthy volunteers. Due to poor data quality or low performance accuracy, 3 participants were excluded from the analysis. The final sample consisted of 6 participants (2 females, mean age 21.5 ± 1.96 years, age range 19–26 years). All participants were right-handed as confirmed by the Edinburgh Inventory of handedness (Oldfield, 1971), had normal or corrected to normal vision, and had no history of neurological or psychiatric disorders. Stimuli were viewed binocularly. Written informed consent was obtained from each individual prior to the experiment. The study received the approval from the Boston University and the Massachusetts General Hospital Institutional Review Boards and conformed to the principles of the Declaration of Helsinki. 2.2. Experimental design The psychophysical specifics of the stimulus and the experimental paradigm used in functional imaging are described in detail in (Calabro and Vaina, 2012, 2011). The temporal succession of the stimuli was as follows (Fig. 1A). First, in the fade-in phase, 9 textured spheres gradually appeared on the screen over a 2000 ms time period. Once the spheres reached the maximum contrast (at 2000 ms) they remained static for 1000 ms (the static phase), after which the motion phase started and continued for 1000 ms. During the motion phase, 8 of the spheres moved to simulate forward motion of the observer (at 3 cm/ sec) while the ninth sphere, the target, had an independent motion forward or backward and different speeds (2,4,6,8 cm/sec) added to the speed of the observer. Note, all visual properties of the target sphere were similar to other non-target spheres. At the end of the motion, the screen was cleared for 250 ms before all 9 spheres were displayed static at their final locations but projected into a single depth plane so that all of them had a constant size. Four of them, the target and three other randomly selected spheres were shown as gray disks labeled with numerals from 1 to 4. The participant had to respond with a button press which of the labeled spheres was the target (four alternatives-forced choice task, 4AFCT). The response period lasted for 1450 ms. The intertrial interval was 300 ms. At all times, a red dot fixation mark was present at the center of the display and participants were instructed to 2 Fig. 1. A - Moving object detection during simulated observer’s forward self-motion paradigm illustration - intertrial period with a red dot fixation mark displayed at the center of the screen (300 ms), followed by a fade in period (2000 ms) when 9 spheres gradually appeared on the screen. Once the spheres reached the maximum contrast (at 2000 ms) they remained static for 1000 ms. Then the motion period (1000 ms) began and all spheres, but one (the target) moved inducing in the observer a perception of forward self-motion. The target sphere moved with different speeds and direction than the observer. In a 4 alternatives-forced choice (4AFC) task participants were asked to identify the target during the response period (1450 ms) that started after 250 ms of the blank screen displayed following the motion period. B - Multivariate pattern analysis. Sensor readings at each time point were used to train a support vector machine classifier to discriminate trials with correct and incorrect responses. To discriminate between classes of features SVC constructs a hyperplane and uses data points close to the hyperplane as support vectors to increase the margin between classes of points for feature classification (right scatterplot). To facilitate illustration, we show only two MEG channels with positive values. fixate their gaze on it throughout the length of the trial. In total, 160 trials were presented to each participant. Prior to the MEG study subjects underwent half an hour of training with the same psychophysical task that was used in the MEG study to achieve above chance level performance (> 25% correct). Echo) structural images were acquired on a 3 T scanner (Siemens-Trio, Erlagen, Germany) using an 8-channel phase array head coil (distance factor: 50%; slices per slab: 128; FOV: 256; FOV phase: 100; slice thickness: 1.33 mm, TR: 2530 ms, TE: 3.39 ms). 2.5. MEG data preprocessing 2.3. MEG data acquisition For analysis, the data were bandpass filtered between 1 and 100 Hz offline; a 60 Hz notch filter was applied to suppress line-frequency noise. The MEG recordings were divided into epochs from the onset of the motion period until the end of the response period. Thus, the time point t = 0ms equals the motion onset time, t = 1000 − 1250ms the blank screen period, and t = 1250 − 2700ms the response period. Bad channels and trials with peak-to-peak amplitudes exceeding 2 pT/cm for the gradiometers or 6 pT for the magnetometers, or 150 μV in EOG were excluded from the analysis. In total, the number of rejected trials was under 15%, leaving between 137–160 trials per subject. The signal-space projection (SSP) method (Uusitalo and Ilmoniemi, 1997) was employed to remove heartbeat and eye-movement related artifacts from the data. To this end, SSP operators were computed using a principal component analysis of the data containing ocular and cardiac artifacts. The MEG data were acquired at the Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, with a 306channel Neuromag Vectorview whole-head system (Elekta Neuromag, Finland) comprising 102 triplets of two orthogonally oriented planar gradiometers and one magnetometer. The MEG data were recorded in a magnetically shielded room (Cohen et al., 2002) with the lights dimmed. The participants were seated with the screen centered at a distance of 80 cm. The stimuli were projected with an LP350 DLP projector (InFocus, Wilsonville, OR) at a resolution of 1024 × 768 pixels with a refresh rate of 75 Hz. The HPI (Head-Position Indicator) coil locations and the participant’s head shape were digitized using a Fastrak digitizer (Polhemus Inc., Colchester, VT) integrated with the VectorView system. At the beginning of each run the position and orientation of the head with respect to the sensor array was determined by localizing the HPI coils on the basis of the magnetic fields they generate (Uutela et al., 2001). At least 80 points were sampled on the scalp to align the coordinate systems employed in MEG and the anatomical MRI in postprocessing. The vertical and horizontal electrooculogram (EOG) signals were also acquired to monitor eye-movements and blinks. The data were bandpass filtered between 0.5 and 200 Hz prior to sampling at 600 Hz. 2.6. MEG data analysis 2.6.1. Multivariate pattern analysis (MVPA) Methods based on Multivariate Pattern Analysis (MVPA) have been successfully used to decode neural activity (Cichy et al., 2015; Haxby et al., 2001; King and Dehaene, 2014; Mohsenzadeh et al., 2019). We used MVPA to decode the MEG signal patterns corresponding to representations necessary for detecting and selecting the object moving independently from the observer’s self-motion. Specifically, at each time point, we trained a linear Support Vector 2.4. MRI data acquisition T1-weighted MPRAGE (Magnetization Prepared Rapid Gradient 3 Fig. 2. Correct detection of a moving object by an observer in forward self-motion can be accurately predicted based on brain neuromagnetic activity in the interval of 400 - 2200 ms after the initiation of stimulus motion. The top of the figure is a schematic illustration of the stimulus and experimental paradigm. Colored rectangles indicate the data segments included into the analysis. A - the graph illustrates the accuracy of SVC in classifying correct and incorrect responses for each timepoint (grey shadow areas indicate time periods when the accuracy is below chance). Chance level is marked by the horizontal dotted red line. B - SVC pattern mapped on the brain surface at 600 ms, 1000 ms, 1400 ms and 1800 ms. They illustrate the areas that were most critical for the correct classification of responses. Classifier (SVC) using the MEG signals (Fig. 1B) (Cortes and Vapnik, 1995). An SVC can be described as a hyperplane that separates the condition classes as best as possible. SVCs learn a linear binary decision rule, h (x ) = sign{wT x + b} , where the weight vector w and threshold b together define a hyperplane L : wT x + b = 0 . The function h (x ) thus indicates the location of a given point x with respect to L and divides the data into two classes. The SVC algorithm was trained using the M = 204 dimensional planar gradiometer data X= [x1, …, xN ] ∈ M × N where N is the number of trials. We used the Python package scikit-learn (Pedregosa et al., 2011) with a linear kernel and cost equal to one (C = QUOTE 1) to find the hyperplane (w and b ) that best separates the correct and incorrect trials as illustrated in Fig. 1B. The weight vector w was first transformed into a pattern in the data space to identify which brain regions it corresponds to (Gramfort et al., 2014; Haufe et al., 2014). This pattern was then mapped to the cortical surface using the linear inverse operator presented in Section 2.6.2 to yield the cortical patterns shown in Fig. 2B. To train and test the SVC, a five-folded cross validation was used. The average prediction accuracy was computed across participants for each time point to characterize the classification performance from motion onset to the end of the trial. To assess the accuracy of the classifier relative to chance level, the chance level was established as the highest proportion of correct responses across participants (75%). 2.6.2. Source estimation The sources of MEG signals were estimated in an individual cortical surface mesh with 10,242 candidate source locations in each hemisphere, resulting in an approximate spacing of 3 mm between source points. The forward solution was calculated using a single compartment boundary-element model (Hämäläinen and Sarvas, 1989). The cortical surfaces for source placement and the inner skull surface for forward modeling were obtained with Freesurfer (Fischl et al., 1999). The Dynamic statistical parametric mapping (dSPM) approach was employed for source estimation (Dale et al., 2000). The source orientations were fixed to be normal to the cortical surface. The noise-covariance matrix was estimated from the t = −500 − 0ms interval prior to the beginning of stimulus motion. The linear dSPM inverse operator was used to map the averaged evoked activity as a function of time to the cortical mantle. The same procedure was employed to visualize the pattern of cortical activity corresponding to the weights of the SVC. The sourcelevel data was morphed into a common space to average across subjects (Dale et al., 1999; Fischl et al., 1999). The evoked source activity was averaged over time to investigate the brain activity patterns elicited by the task of detecting a moving sphere (the target) by an observer in simulated forward motion. First, the absolute values of the dSPM maps were averaged across the motion period (0−1000 ms), blank period (1000−1250 ms) and response 4 period (1250−2700 ms). Second, to compare the SVC results with the evoked activity, we averaged the source activity within 50 ms windows centered at 600 ms and 1000 ms, all of which were within the motion period. The dSPM source current estimates represent a z -ratio normalized by the noise covariance, and we used a threshold of 95% of overall brain activity within the selected period to display the areas with the highest activity. Since several brain areas with high SVC pattern values were not present in the evoked thresholded activity, we decided to calculate the average activity over the entire trial (0–2700 ms) and also lowered the threshold to 40% to find all the possible ROIs potentially relevant to the task. The ROIs were delineated (Supplementary Fig. 1) and their time series were used for calculating dynamic connectivity with the aim of validating SVC results. The cortical areas were defined based on their anatomical position and relative positions with other areas. surface with the ROIs marked in green. Sinks and sources were measured and plotted within each of the time windows. Significance of connectivity was assessed using a similar procedure as above, except individual connectivity was used as the test statistic instead of in-degree and out-degree. The 95th percentile level of connectivity scores was selected as the threshold significance level, as above so that any score above this threshold was significant to a p < 0.05 level. 2.7. Data availability The authors will make the data available upon reasonable request. 3. Results 3.1. Behavior 2.6.3. Dynamic Granger-Geweke causality Dynamic Granger-Geweke Causality (DGGC) in the source space was used to determine time-varying directional functional connectivity between regions of interest (ROIs) (Lin et al., 2009; Vaina et al., 2010). DGGC is a version of Granger-Geweke Causality (Geweke, 1982) computed with a sliding time window. In each time window the signals were fitted with an autoregressive (AR) model. To compute the Granger score, an AR model was fit to the signal from ROI A, and then separately fit to the signal of ROI A including past values of ROI B. The Granger score is defined as the negative log ratio of the power of the residual error of the fit of the AR model of ROI A including ROI B over the fit with ROI A alone. Thus, if the past values of ROI B help predict values of ROI A, then the power of the residual error of the joint model decreases, thus increasing the Granger score. For Gaussian signals the Granger Score is a measure that reflects information flow from region X to region Y (Barnett et al., 2009). Here, we used the Granger score as a surrogate for measuring qualitatively the “flow of information” between ROIs. The details of the computation of the frequency-domain Granger Scores can be found in Lin et al. (2009). ROIs were selected based on the brain evoked activity, as explained in Section 2.6.2 and are shown in Supplementary Fig. 1. The scores were summed across the gamma band at 1 Hz steps from 30 Hz to 80 Hz. We chose to measure the connectivity in the gamma frequency band since the gamma oscillations have been reported to play an important role in sensory signals processing, attention (Fries et al., 2008; Gray and Singer, 1989; Magazzini and Singh, 2018) and working memory (Başar et al., 2001; Honkanen et al., 2015; Jensen et al., 2007; Kaiser and Lutzenberger, 2005; Miller et al., 2018; Ward, 2003), which are critical for the task described here. To characterize the magnitude of “flow of information” into and out of an ROI, we measured the in-degree and out-degree of DGC scores at that ROI. The in-degree of an ROI is given by the sum of DGC scores over all the connections to that ROI for each time window. Similarly, the out-degree is the sum of all the connections going out of an ROI for each time window. To smooth the Granger scores, a 200 ms uniform sliding window was applied to the in- and out-degrees. We defined “hubs” as regions central to the functional network of Granger causal connections in each time window, that have significantly high in-degree as “sinks” and hubs that have significantly high out-degree as “sources”. In order to identify the hubs, we performed permutation testing. A total of 100,000 random in-degree and out-degree were generated by permuting the Granger scores between areas and across time. We calculated the 95th percentile level to define the threshold significance level, thus any Granger score above this threshold would be significant to a p < 0.05 level. We note that the random in-degree and out-degree have the same distribution because the incoming connection into one ROI is the outgoing connection from another ROI, therefore computing only one common threshold is sufficient. In Fig. 4 only significant mean in-degree and outdegree hubs across time-windows were visualized on the cortical The average response accuracy across participants while performing the task in MEG was 60% (range across participants: 50–75%), whereas the percent of correct responses in a four alternative forced choice task (4AFC) due to chance is equal 25%. The average time of response was 910 ± 190 ms from the onset of the response period (at 1250 ms). 3.2. SVC classification of correct and incorrect trials SVC was trained to discriminate between trials with correct and incorrect responses for each participant by using the sensor signals at a given time point. Fig. 2A illustrates the classification accuracy starting from the motion onset, during the 250 ms of the blank period and until the end of the response period. At each timepoint after the start of motion, the SVC accuracy indicates how accurately the classifier could predict whether a participant would give a correct or incorrect response. Time periods where classification accuracy was below the chance level are marked with a grey shadow area. Fig. 2B shows the source distributions corresponding to the SVC patterns used to classify the correct and incorrect responses. At first, SVC accuracy briefly surpassed the threshold level at 150 ms and went back below chance. SVC reliably classified the trials with correct responses after 400 ms from the motion onset, reaching the highest accuracy during the blank period, between 1000−1200 ms. The SVC pattern had distinct spatial patterns across time. To illustrate the spatial pattern of brain activity important for the SVC successful performance, we selected four time points during the period where SVC accuracy was above chance level (Fig. 2B). At 600 ms SVC accuracy gets consistently above chance level and the highest values in the SVC pattern were located in the left superior temporal sulcus (STS), the right superior parietal lobule (SPL) and the posterior central sulcus (PostCS). At 1000 ms, which marks the end of the stimulus motion, cortical areas with high values in the SVC pattern expanded extensively across the cortex. In the left hemisphere those areas included insula, dorsolateral prefrontal cortex (DLPFC), inferior frontal gyrus (IFG) and medial orbitofrontal cortex (mOFC), and in the right hemisphere, hMT+, anterior part of the middle temporal gyrus (MTG), STS and parahippocampal gyrus (PHG). At the same time, bilateral activation in the central-CS and postCS, isthmus of the cingulate gyrus involving retrosplenial cortex (RSC), early visual cortex (EVC) including V1 and V2 were also contributing to the high SVC accuracy. At 1400 ms, participants had been for 150 ms in the response period during which they had to choose which of the four labelled disks corresponded to the target. At this point, frontal areas (FEF, IFG) in the right hemisphere were strongly involved, while in the left hemisphere the highest values in the SVC pattern were distributed across the temporal and parietaloccipital cortex (PO), namely, V3a and V7, STS and left posterior lateral fissure (PLF). The activity in EVC at 1400 ms was no longer relevant for the correct classification. At 1800 ms, when on average participants entered their response, the highest values in the SVC pattern were 5 mostly seen in the mOFC, superior frontal gyrus (SFG), supplementary motor area (SMA), precuneus and supramarginal gyrus mostly in the right hemisphere. However, our results do not exclude the possibility that some brain areas were activated during both conditions (correct and incorrect response). The low SVC accuracy at the beginning of the stimulus motion suggests that the activation pattern was similar between these two conditions. However, the SVC findings indicate that during periods of time when the SVC accuracy surpasses the chance level (as shown in Fig. 2A) there were some cortical areas (illustrated in Fig. 2B) that expressed different activity between correct and incorrect trials. PostCS. The areas V3a/V7 in the left hemisphere also remained active. During the 250 ms blank period, between the end of the motion period and beginning of the response period, the pattern of activity was lateralized to the right hemisphere including areas active during the stimulus motion: hMT+, STS, inferior temporal sulcus (ITS), EVC, parietal-occipital sulcus (POS) and precuneus. During the response time period (1250−2700 ms) when participants’ task was to decide which of the numbered grey disks was the target, the activations were present mostly in the frontal cortex. There was also activation in motor related areas such as left CS, SMA as well as in left middle cingulate cortex (MCC), right anterior cingulate cortex (ACC) and medial orbito-frontal cortex (mOFC). To localize areas activated under threshold, activation across the entire trial (0−2700 ms) was averaged and a liberal threshold of 40% was used. The areas whose activation was above threshold were labeled (Supplementary Fig. 1) and used later in the DGGC analysis. These areas included EVC (V1 and V2) V3a, V7, LOR, hMT+, STS, IPS, PostCS, left CS, precuneus, middle cingulate cortex (MCC), FEF and DLPFC. Subcortical areas or areas located in the temporal-parietal sulci folding were excluded from the analysis due to their unreliability in MEG measurements. 3.3. Evoked brain activity during the task of a moving object detection by an observer in (simulated) forward self-motion We investigated brain activation patterns during different stimulus periods (motion, blank and response) of this FSD type task to relate them to the areas critical to the classification of the correct/incorrect response of the participants. We used a threshold of 95% of maximum z-score activation across the whole brain to illustrate the areas most salient for task performance. During the motion period we observed strong bilateral activation in EVC (comprising areas V1, V2), V3a/v7, right hMT + and STS as well as in the right PLF. In addition, we investigated cortical activation during the motion period for the same timepoints (600 ms – Fig. 3D and 1000 ms – Fig.3E) as used in the SVC analysis. At 600 ms, the activation was observed in EVC, LOR, hMT+, STS bilaterally, and in the left V3a/ V7. However, activation in the right hemisphere was much stronger for hMT + and STS. In addition, higher order areas in the right hemisphere were also involved, including IPS, SPL, precuneus and posterior lateral fissure (PLF). At 1000 ms, at the end of the motion period, the activation became lateralized to the right hemisphere featuring hMT + and STS together with weaker focal activations in EVC, precuneus and 3.4. Connectivity Gamma-band (30−80 Hz) dynamic Granger causality was averaged across two motion time periods (0−600 ms and 600−1000 ms), the blank and the response periods (Fig. 4). We visualized the connectivity in each ROI as a series of proportional area plots showing areas with significant in-degree (sources, red circles) and out-degree (sinks, blue circles) connectivity. We also report significant connectivity from sources and sinks to other ROIs, as illustrated in Supplementary Fig. 3.1. and 3.2. During the 0−600 ms period of the motion period (Fig.4A), EVC Fig. 3. Brain activation during moving object detection by a forward moving observer: A - brain activation pattern averaged across object motion period (0-1000 ms); B - brain activation averaged across blank screen period (1000 ms-1250 ms); C - Areas that were the most active during the response period (averaged across 12502700 ms); D - the most active brain areas in the middle of the motion period (600 ms); E - brain activation at the end of the motion period (1000 ms). V1, V2 – EVC (early visual cortex), LOR – lateral occipital cortex, hMT+ – human motion complex, ITS – inferior temporal sulcus, STS – superior temporal sulcus, POS – parietaloccipital sulcus, CS – central sulcus, SMA – supplementary motor area, MCC – middle cingulate cortex, ACC – anterior cingulate cortex, mOFC – medial orbito-frontal cortex, IPS – inferior parietal sulcus, SPL – superior parietal lobule, PLF - posterior lateral fissure. 6 Fig. 4. Incoming and outgoing connectivity between cortical ROIs during the task. The area of each circle represents the in-degree and out-degree GC score associated the corresponding ROI that it is overlaying. Red circles represent “sources,” which are ROIs with high outgoing connectivity and blue circles represent “sinks,” which are ROIs with high incoming connectivity. We display GC scores represented by circles only for those areas in which the GC scores were above significance threshold computed by a series of permutations (GC threshold = 85.7). (V1 and V2) in both hemispheres was the major source of connectivity and also, to a lesser extent, the areas hMT+, V7 and STS in the right hemisphere. Outgoing connectivity from these sources was mostly directed to ROIs which were significant sinks of connectivity (left V3a, bilateral LOR, left hMT+, and right FEF). Right hemisphere EVC was also sending information bilaterally to the parietal and frontoparietal ROIs (PostCS, CS, MCC, FEF, DLPFC) (Supplementary Fig. 3.1). The motion continued up to 1000 ms, and the source patterns from 600 to 1000 ms were similar to those described in the previous motion period (Fig. 4B). Bilateral EVC (V1 and V2) and hMT+, V7 and STS in right hemisphere continued to feed information to significant sinks of connectivity (bilateral LOR and V3a, right FEF). However, the number of significant sinks in the left hemisphere extended anteriorly to V7, FEF and DLPFC. Most of the information flow into these areas was from the biggest sources of connectivity (bilateral EVC and right hMT+). During the blank screen period (1000−1250 ms), the outgoing connectivity in EVC decreased compared to the earlier time points (Fig. 4C). This would be expected as no visual information, other than fixation mark, was available during this time interval and probably working memory was the main cognitive process. In both hemispheres EVC was sending information to significant sinks of connectivity (LOR and V3A in the left hemisphere) and also provided significant output to higher order parietal and frontal ROIs in both hemispheres (IPS, precuneus, PostCS, CS and FEF) (Supplementary Fig. 3.2.) Right PostCS was also a significant source broadcasting connections to visual ROIs (right LOR and V3a, left hMT+) and to attention-related ROIs (left precuneus and DLPFC, bilateral FEF). During the response period, EVC and right hMT + once again appeared to be the largest sources of connectivity communicating to significant sinks (bilateral LOR, left V3a and right FEF) (Fig. 4D). This may be explained by the fact that the participants were presented with visual information (Magazzini and Singh, 2018). Bilateral EVC also had significant connection with left temporal (STS), parietal (IPS and PostCS) and frontal (CS, DLPFC, FEF) ROIs (Supplementary Fig. 3.1.). 4. Discussion In the present study we investigated the brain dynamics underlying a psychophysical task that involves the detection of a moving object by an observer in forward (simulated) self-motion. We employed three different approaches to investigate the neurophysiological mechanisms implicated in this task. Classical evoked activations maps allowed illustration of the neural substrates underlying this task and partially revealed the activation patterns reported in our previous fMRI study (Calabro and Vaina, 2012) and others (e.g. Arnoldussen et al., 2013; Cardin et al., 2012; Galletti and Fattori, 2003; Pitzalis et al., 2019, 2015, 2013, 2010). We also investigated directional connectivity among the ROIs with the highest evoked activation during different stimulus time periods and the results provided further evidence of the communication dynamics involved in the task. Using an unbiased datadriven approach we investigated on the millisecond level the brain’s spatial patterns critical for the successful task performance. The findings of this analysis provide novel insights into which areas are relevant to the correct detection of a moving object by an observer in forward (simulated) self-motion. Taken together, our findings provide, in a time-resolved manner, the cortical ROIs involved in the response correctness of the task and how the cortical ROIs mapped with SVC are similar to the ROIs captured by the evoked activity and connectivity information flow. The SVC, evoked activity, and DGGC results provide important insights into the cortical dynamics underlying this task. Specifically, we found different cortical activity and connectivity patterns within two segments of the motion period. In order to investigate the behavioral correlates of these differences, in a post-hoc control test outside the MEG, subjects naïve to the task participated in the same psychophysical experiment, with the exception that in each test version the motion period was set at different time intervals, from 100 ms to 1000 ms. The results demonstrated that the participants’ performance was above chance for detecting the target independent of the speed or direction of 7 its motion (and hence independent of the task difficulty) when the stimulus motion period was 600 ms or more (see Supplementary Fig. 2). The control post-hoc behavioral data showed that as the motion period got closer to the end (600−1000 ms) the correct identification of the target sphere was increased. Consistent with these results, the SVC classification raised steadily above chance after 400 ms during the stimulus motion period. At 600 ms SVC relied mostly on the left hemisphere STS, and at 1000 ms (end of the motion period), it included several fronto-parietal regions in the left hemisphere, while in the right hemisphere it included occipital-temporal regions, and bilaterally the postcentral sulcus. A hallmark of visual-cognitive brain computations is the ability to act at different time scales. In the next two sections we discuss how this leads to incremental acquisition of task relevant information over time, resulting in an increased precision of the stimulus representation that our brain is able to generate. outcome of a trial can be predicted from the activation pattern of these few cortical ROIs (left dorsal STS, right SPL and PostCS) within 600 ms into the motion stimulus, which is congruent with the results of our post-hoc analysis of the behavioral task performance. Consistent with the psychophysical mechanism proposed by Rushton and collaborators (Rushton et al., 2007; Rushton and Warren, 2005), we suggest that flow parsing is a candidate mechanism for implementing our task. The significant overlap between the areas found to be active in our study during the first 600 ms of the motion period and the areas reported to be engaged in flow parsing, as discussed in the introduction, support this hypothesis (Calabro and Vaina, 2012; Galletti and Fattori, 2003; Pitzalis et al., 2019). However, it is important to ask the question of whether the optic flow field alone, can fully account for observers’ performance, or whether 3D scene-context information is used for identifying the target. Royden and collaborators’ psychophysical studies (Royden and Connors, 2010; Royden and Moore, 2012) showed that observers are sensitive to target deviations (speed and direction) from the optic flow. Calabro and Vaina (2012) investigated possible strategies that observers may use to solve the task described here without incorporating the 3D scene context. For example, the choice of the target may have been based entirely on retinal speed of the spheres. However, in our previous psychophysical study we showed that when the distribution of retinal speed was altered, observers’ performance did not change (Calabro et al., 2011), thus performance was not determined by the absolute retinal speed. Alternatively, observers could have used relative motion among the motion vectors (speed and/or direction) and choose for the target the sphere with maximum magnitude or moving inward, while the other eight spheres were moving outward (simulating the observer’s forward motion). Calabro and Vaina (2012) developed a model in which the response to the task, detection of the independently moving sphere, was selected by using only relative motion among the nine spheres that constituted the stimulus. They performed a large number of simulations to determine whether the subjects’ performance could be explained solely by a relative motion strategy (for comparisons between subjects’ performance and the model see Fig. 6 in Calabro and Vaina (2012)). Their model showed that for approaching (positive) velocities the relative speed strategy could account for subjects’ performance, however, for the receding target sphere (negative velocities) the model did not provide above chance correct response because in these situations the speed of the target nearly always was within the range of the speeds of the spheres whose motion was due to the observer’s motion. The same situation occurs if motion in depth cues are considered, instead of direction. Thus, Calabro and Vaina (2012) concluded that the poor performance on detecting the receding sphere condition suggests that subjects’ performance does not rely on strategies based on relative speed and/ or direction among the moving spheres. Consistent with our previous results, the data presented here suggests that to solve this task subjects must use more than just relative motion among the spheres. Our results are consistent with mechanism implementing the flow-parsing hypothesis (Rushton and Warren, 2005; Warren and Rushton, 2009a, 2007), by which the target must be related to the entire scene such that its world-centric object motion can be determined. 4.1. Cortical activity in the first 600 ms of the stimulus motion supports the use of the flow parsing mechanism for solving the task At 600 ms into the stimulus motion period, the application of SVC revealed that the main ROIs involved in distinguishing correct and incorrect responses were the left dorsal STS, right SPL and PostCS. Imaging studies have shown that in humans, SPL is involved in goal directed spatial attention orientation (Shomstein et al., 2010) and in attention modulation during visual motion (Büchel et al., 1998). The dorsal STS (especially STP) was reported to be involved in optic flow processing in the macaque (Anderson and Siegel, 1999; Raffi and Siegel, 2004), and, in humans, in biological, object motion and optic flow conditions (Aspell et al., 2005; Grossman et al., 2000; Grossman and Blake, 2002, 2001; Howard and Howard, 1994; Vaina et al., 1998; Vaina and Soloviev, 2004). The activation in these and other areas were also observed in evoked brain activity reported in our study. The visual motion-sensitive areas, especially the right hMT+, and IPS were also active during this time period, however, they did not play a role in the SVC prediction of correct response. Activations in these areas are consistent with the known brain’s neuronal sensitivity to optic flow (e.g. Cardin et al., 2012; Duffy and Wurtz, 1991; Greenlee, 2000; Holliday and Meese, 2008; Morrone et al., 2000; Pitzalis et al., 2013, 2010; Vaina et al., 1998; Wurtz, 1998) which, according to the Flow Parsing hypothesis, are responsible for identification and parsing out the retinal global motion resulting from the observer’s forward selfmotion, attributing the remaining retinal motion to scene relative object motion (Warren and Rushton, 2007). To define the directional functional connectivity among the activated regions of interest in this task we conducted a follow up analysis of the cortical computations involved in determining time-varying directional functional connectivity between the ROIs active during the different time windows. We found that DGGC connectivity in the first motion period (0−600 ms) matched the evoked response, with early visual cortex (V1, V2) and motion-responsive areas (hMT+, STS) acting as sources in the network. Specifically, there was a right-hemisphere bias in the hubness of hMT + and STS as sources of connectivity. This is in line with the literature suggesting right-hemisphere dominance of spatial cognition (Hugdahl, 2000; Oleksiak et al., 2011; Smith et al., 1996; Vogel et al., 2003; Walter et al., 2003). Together, these results present the picture that most of the computations during 0−600 ms time period are involved in the processing of the visual motion stimulus. The role of early visual regions is to propagate sensory information over dorsal and ventral visual streams as was suggested in another MEG study focused on the dynamics visual processing (Nunes et al., 2019). Such processing pattern potentially underpins primary evaluation of the stimulus that occurs at the beginning of all trials disregarding the outcome. That would explain why during this time period the SVC pattern is very sparse and limited to few higher-order cortical areas. Nevertheless, our SVC results show that the 4.2. Establishment of target representation during the 600−1000 ms stimulus motion interval At 1000 ms SVC accuracy increased drastically. Compared to the previous time period (0−600 ms), now there was a large number of ROIs with high values in the SVC pattern (IFG, insula, DLPFC, mOFC in the left hemisphere, bilateral EVC, CS, PostCS, right hMT+, PHG, aMTG). In contrast the evoked activity pattern diminished by the end of the motion period. However, several ROIs active at 600 ms (hMT+, STS, precuneus, PostCS in the right hemisphere, v3a/v7 and POS in the left hemisphere) remained active until the end of the motion period. It 8 is likely that evoked activity becomes more focal as the representation of the stimulus is finalized after being computed in the previous motion interval. Increase in the values in the SVC pattern in the left prefrontal areas suggests that spatial attention and memory are involved in maintaining the target location and in inhibiting the observers’ tendency to enter their response prior to the response period. This is in agreement with the DGGC results during 600−1000 ms motion period which indicate that left prefrontal regions are significant sinks of connectivity receiving most of the information from EVC and right hMT+. Taken together, our findings suggest a close interplay between the cortical visual areas involved in computing optic flow and in detecting the target, and higher order areas involved in spatial attention and memory for location. This interaction is critical for the correct response, as shown by SVC accuracy curve (Fig. 2A) and congruent with previous reports of the tight cooperation between spatial attention and working memory and neural substrates representing these cognitive domains in building, retrieving and updating mental representations (Lepsien and Nobre, 2006; Nobre et al., 2004). 4.4. Visual, motion and frontoparietal ROIs work in unison to generate the response The average brain activation pattern during response period involved by and large several frontal regions: right mOFC, ACC, left CS, SMA, and MCC. Most of these areas are also critical for the accurate classification of correct and incorrect trials response as shown by SVC (right mOFC, SMA and MCC at 1800 ms after motion onset). The mOFC and IFG are important for inhibition of motor response (Berlin and Bohlin, 2002; Swick et al., 2008; Thompson-Schill et al., 2002). There is evidence from animal studies showing an extensive involvement of frontal areas in conditions when delayed response is required and their critical impact on task performance (Goard et al., 2016; Kojima and Goldman-Rakic, 1982; Miller et al., 1996). It was proposed that the role of prefrontal cortex is to accommodate flexibility in the response options before making the final decision (Stokes et al., 2013). Based on our findings, we also suggest that from the end of the stimulus motion period to the time when response is entered, it is necessary for the frontal regions mentioned above to work in concert to retain the target representation while evaluating the answer options to decide on the final response. 4.3. Active maintenance of target representation in space during the blank period Given that our experiment design is of FSD type, the blank period is particularly interesting since during this time, the response choices are not yet available although participants have detected the target. Thus, observers have to retain a representation of the target location within the other spheres and inhibit the tendency of entering the response. This may also occur during the 1000 ms of the stimulus motion, that is if subjects have formulated a hypothesis regarding which of the moving spheres corresponds to the target location, they still must await the response period (after 1250 ms) and then in a 4AFC enter their decision regarding the target identity. Remarkably, the accuracy of the SVC during the blank period (1000−1250 ms) is highest, suggesting that a stimulus representation characterized by the specific brain state on which the SVC relies has been constructed and is available to the perceiver. Despite no incoming visual information, the evoked activity during the blank period shows activations in EVC (V1, V2), the right hMT+, POS, pSTS and precuneus, which are the areas reported to be sensitive to optic flow (Michels et al., 2009; Raffi et al., 2002; Wada et al., 2016), suggesting that the participants maintain the spatial position of the target in working memory for later use to enter their response (during the response time period). The connectivity computed during the blank time-period (Supplementary Fig. 3.2.) shows that the PostCS presented outward flow of information towards several areas, including both hemisphere V3a, LOR, hMT+, precuneus, FEF, and left DLPFC. Aside from the visual and motion processing areas that are mostly involved in computing the stimulus representation, there is also activation in the precuneus, which is involved in visuospatial and working memory tasks, mental imagery, and spatially-guided behavior (Cavanna and Trimble, 2006; Fletcher et al., 1995), and the frontoparietal areas, FEF and DLPFC, known to play an important role in spatial attention (Corbetta and Shulman, 2002; Kelly et al., 2008; Liu et al., 2003; Schall et al., 2004; Serences et al., 2004) and working memory (Curtis and D’Esposito, 2003; Fuster, 2000; Goldman-Rakic, 1991). Given the connections from postCS to the areas listed above (both hemisphere V3a, LOR, hMT+, precuneus, FEF, and left DLPFC), we suggest that postCS is central to maintaining the target’s spatial location between the end of the stimulus motion and the beginning of the response period, whereas, the precuneus and DLPFC are involved in the storage of the representation of the motion stimulus computed by motion-responsive areas, such as hMT and V3a. 5. Conclusion We used a three-prong analysis to elucidate time-varying cortical processing during a task of parsing out an object motion while the observer is in simulated forward self-motion. To our knowledge, this is the first MEG study to dynamically contrast the brain activity necessary for correctly accomplishing the task by employing a data-driven machine learning approach and by analyzing the activity evoked by the task performance. Consistent with behavioral measures, the classification accuracy of the linear Support Vector Classifier (SVC) employed raised steadily above chance after 400 ms during the stimulus motion. The SVC first relied mostly on the left hemisphere STS, and later into the stimulus involved several areas in both hemispheres including fronto-parietal regions on the left and occipital-temporal regions on the right, as well as the postcentral sulcus bilaterally. The evoked activity was localized in occipital and parietal areas specialized for motion processing or spatial attention, whereas, the areas responsible for the correctness of response also involved regions in the frontal lobes. Based on the pattern of information flow revealed by the connectivity analysis (DGGC), the interplay between these areas is necessary for computing the representation of the stimulus and address task requirements. This study demonstrates that the correct response can be predicted with the highest precision only when the representation of the target location is computed and actively maintained in working memory until the response period. 9