4 52 4. End-to-end optimization of prosthetic vision maximizes information transfer and has no knowledge about practical requirements, such as sparsity. Therefore, the possible phosphene encoding strategies that may be found by the model are not limited to ecologically useful solutions. This is exemplified by the ‘inverted’ phosphene encoding strategy with a large average electrode activity of 95.65%. Here, finding an undesirable phosphene encoding strategy can be considered an expected consequence of an imprecise learning objective. To guide the model towards useful latent SPV representations, in experiments 2 and 3 we implement additional regularizing constraints (sparsity), and explore different optimization tasks, as discussed below. An important requirement for automated optimization with deep learning, is that all components of the artificial neural network make use of differentiable operations. In this paper we contribute a basic implementation of a fully differentiable phosphene simulation module, including a straight-through estimator for quantized phosphene activation. Further studies could adapt or extend this implementation to test automated optimization for different phosphene simulations, for instance, varying the number of electrodes, and the positions of the phosphenes. 4.4.2. Tailored optimization to sparsity constraints Implementing additional constraints in the optimization procedure may provide a general solution to account for practical, medical or biophysical limitations of the prosthetic device. For instance, the inflammatory response of brain tissue is a major concern that limits the long-term viability of cortical electrode implants (Fernández et al., 2020; Polikov et al., 2005) and these adverse effects may partly be avoided by limiting the chronic electrical stimulation itself (Lewis et al., 2015; McCreery et al., 1988). The results of experiment 2 demonstrate that implementation of a such a sparsity constraint may help to regularize the electrode activity. Comparing the results with experiment 1, we can observe that even with a low value for the sparsity parameter κ, the new objective function causes the model to find a more ecologically useful encoding strategy. Notably, a larger sparsity weight κresults in fewer active electrodes, but also in impaired reconstruction performance. Choosing a balanced value forκ, depending on the needs of the patient, can be seen as a part of a tailored optimization approach of image preprocessing in prosthetic vision. Importantly, the proposed method enables the implementation of virtually any type of additional constraint that can be incorporated in the optimization procedure. Other examples of biophysical limitations for prosthetic vision, besides sparse electrode activation, could include minimal distance for simultaneously activated electrodes, maximal spread of electrode use or minimal temporal separation. Future research focusing on such biophysical limitations could extend the proposed method to include such or other constraints in the optimization procedure. 4.4.3. Task-specific optimization for naturalistic settings Due to the relative complexity, and the presence of non-relevant information, the encoding of naturalistic scenes into phosphenes remains a challenge and it requires taskdependent processing. This challenge is explicitly addressed by the proposed end-to-end approach.
RkJQdWJsaXNoZXIy MTk4NDMw