A Formal Approach to the Problem of Logical Non-Omniscience

Scott Garrabrant
(Machine Intelligence Research Institute, Berkeley, CA)
Tsvi Benson-Tilsen
(Machine Intelligence Research Institute, Berkeley, CA)
Andrew Critch
(Machine Intelligence Research Institute, Berkeley, CA)
Nate Soares
(Machine Intelligence Research Institute, Berkeley, CA)
Jessica Taylor
(Machine Intelligence Research Institute, Berkeley, CA)

We present the logical induction criterion for computable algorithms that assign probabilities to every logical statement in a given formal language, and refine those probabilities over time. The criterion is motivated by a series of stock trading analogies. Roughly speaking, each logical sentence phi is associated with a stock that is worth $1 per share if phi is true and nothing otherwise, and we interpret the belief-state of a logically uncertain reasoner as a set of market prices, where pt_N(phi)=50% means that on day N, shares of phi may be bought or sold from the reasoner for 50%. A market is then called a logical inductor if (very roughly) there is no polynomial-time computable trading strategy with finite risk tolerance that earns unbounded profits in that market over time. We then describe how this single criterion implies a number of desirable properties of bounded reasoners; for example, logical inductors outpace their underlying deductive process, perform universal empirical induction given enough time to think, and place strong trust in their own reasoning process.

In Jérôme Lang: Proceedings Sixteenth Conference on Theoretical Aspects of Rationality and Knowledge (TARK 2017), Liverpool, UK, 24-26 July 2017, Electronic Proceedings in Theoretical Computer Science 251, pp. 221–235.
Published: 25th July 2017.

ArXived at: https://dx.doi.org/10.4204/EPTCS.251.16 bibtex PDF
References in reconstructed bibtex, XML and HTML format (approximated).
Comments and questions to: eptcs@eptcs.org
For website issues: webmaster@eptcs.org