Maybe I've been looking in the wrong places, but it seems that there is not much discussion out there on how we should be looking towards the aviation industry for insights into how our relationship with automation should proceed.
After all, pilots have been dealing with "autopilot" systems for decades now!
Salient lessons I think:
* There is a general skepticism of "what the machine says" that is drilled into them from early on - Yes, many do depend on navigation systems much of the time, as they are generally reliable. But, where possible, they are reminded to look out the window / use multiple sources and confirm whether that matches what the machines say.
On that note, a very important part of their job is to constantly monitor and track the system information, cross-checking that against their expectations. Trust but verify.
* Between that and the need for redundancy in case of failure (e.g. backup radios, backup map + light sources, backup instruments), you'll also soon find that many pilots are actually wary about night flying, where the risks are greatly heightened, as darkness makes it harder to spot deadly situations. In other words, you should be careful about putting yourself in risky situations to begin with! Manage the risky situations by avoiding in the first place
* The need for regular "stick and rudder time" - i.e. there is widespread understanding that they do need to manually exercise / practice their skills *without using the automation*, so that they remain capable of stepping in when the automation goes berserk / breaks, which *will* happen, often at the worst possible times.
* It is also heartening that while this automation has existed for decades, there has been sufficient strong advocacy for maintaining two-pilot operations to maintain human-in-the-loop operations.
