The variable responses of sensory neurons tend to be weakly correlated (spike-count correlation, r_sc). This is widely thought to reflect noise in shared afferents, in which case r_sc can limit the reliability of sensory coding. However, it could also be due to feedback from higher-order brain regions. Currently, the relative contributions of these sources are unknown. We addressed this by recording from populations of V1 neurons in macaques performing different discrimination tasks involving the same visual input. We found that the structure of r_sc (the way r_sc varied with neuronal stimulus preference) changed systematically with task instruction. Therefore, even at the earliest stage in the cortical visual hierarchy, r_sc structure during task performance primarily reflects feedback dynamics. Consequently, previous proposals for how r_sc constrains sensory processing need not apply. Furthermore, we show that correlations between the activity of single neurons and choice depend on feedback engaged by the task.