Initially, I was tempted to say that ANATHEM is the anti-MARTIAN, but this is too dualist, setting up a narrow-minded binary opposition (considering that ANATHEM itself contains sequences of orbital dynamics, as in THE MARTIAN). So I think that we could talk rather in terms of a continuum, taking science fiction as the literature of “cognitive estrangement” (Darko Suvin) or as “degrees of subjunctivity” (Samuel Delany).
From this perspective, THE MARTIAN puts the emphasis on the cognition rather than on the estrangement, and is undoubtedly hard sf. ANATHEM puts the emphasis on the estrangement, and is better described as science fantasy. However, science itself comports its own factor of estrangement, especially in the mathematics and physics of the last one hundred years.
The science mobilised by THE MARTIAN does not go beyond notions of common sense and of junior high school maths and science, with no philosophical dimension. The science of ANATHEM is the more estranging science of quantum theory and the multiple worlds interpretation, and it appeals to the philosophical lineage of Plato-Leibniz-Kant-Husserl combined with more modern philosophy of science and maths.
ANATHEM is the more self-referential novel, and explicitly takes into account degrees of estrangment in its idea of multiple hylaean theoric worlds and multiple cosmoi, so we have (please forgive the jargon) both lateral and vertical forms of estrangement.