About a Proof Pearl: A Purported Solution to
a PoplMark Challenge Problem that Is Not One
Abstract
The PoplMark Challenge comprises a set of problems intended to measure the strength of reasoning systems in the realm of mechanizing programming language meta-theory at the time the challenge was enunciated. Included in the collection is the exercise of demonstrating transitivity of subtyping for a specific algorithmic formulation of subtyping for an extension of System F. The challenge represented by this problem derives from the fact that, for the given formulation, subtyping must be proved simultaneously with another property called narrowing. In a paper published as a proof pearl, Brigitte Pientka claimed to have presented a solution to the problem in which “the full power of parametric and higher-order judgments” is exploited to “get the narrowing lemma for free.” We show this claim to be inaccurate. In particular, we show that the simplification is in substantial part the result of changing the formulation of the subtyping relation in a way that modifies the challenge rather than the outcome of the manner in which the argument is mechanized.
The PoplMark Challenge [1] identified a collection of benchmarks for measuring the status of reasoning frameworks at the time that it was presented from the perspective of mechanizing the meta-theory of programming languages. The problems in this collection are oriented around the typed -calculus with second-order polymorphism known as System F extended with subtyping and records. The formalization task that is of specific interest here is what is referred to in the mentioned paper as the transitivity of algorithmic subtyping. This problem, which was identified as Challenge 1A, was considered worthy of inclusion in the benchmarks because the proof of transitivity requires an intricate inductive argument in which an auxiliary lemma called the narrowing lemma must be established simultaneously. In [3], Brigitte Pientka claimed to have presented a solution in the Twelf system that “plays to the strengths of the logical framework LF” and that as a benefit gets the “tedious narrowing lemma, which must normally be proved separately, for free.” Unfortunately, what is really presented does not constitute a solution to Challenge 1A. Specifically, the “simplification” in the proof results not from the choice of encoding but from changing the formalization of subtyping to a form that comes close to including narrowing in a typing rule.
We provide substance to the above observations in the rest of this note. We first recall the original formalization task and then explain our comments about the “solution” presented in [3].
The formalization task
Challenge 1A relates solely to the type language that includes subtyping but does not encompass record types. The two syntactic entities that are relevant to describing the subtyping relation in this context are types and type environments. Expressions in these categories are given by the following grammar rules:
The token represents variables in these rules. A type of the form represents a binding of over the type . Type environments bind variables that occur in types within their scope and they identify subtyping constraints for the variables so bound. Variables bound by an environment are assumed to be named uniquely and the subtyping rules implicitly permit the renaming of bound variables in types to ensure that this property holds. The presentation in [1] uses the same symbol for both <: and . We avoid this overloading here to highlight issues that arise from the confusion of the two in [3].
The subtyping relation that is of interest is written as , to be read as “ is a subtype of relative to the type environment .” This relation is determined by the rules in Figure 1. The SA-Top rule has the requirement that must be well-scoped with respect to , i.e. all its free variables must be bound in . Similarly, the SA-Refl-TVar rule has the proviso that the variable must be bound in .111It can be shown that these assumptions suffice to ensure that is derivable only for types and that are well-scoped. It may be desirable to impose well-scoping requirements on as well, in which case the order in which bindings appear in it will be important. We discuss this observation further later in the note. This collection of rules has an algorithmic flavor in that it determines a process for checking a purported subtyping relation that is syntax-directed. The form of the SA-Trans-TVar rule is crucial to this property. Note that because of the form of this rule it is not automatically the case that is derivable if appears in . However, this turns out to be the case if is well-scoped with respect to because then is derivable.
Challenge 1A consists of providing a mechanized proof of the fact that the subtyping relation defined by the rules in Figure 1 is transitive. Specifically, a proof is to be provided for the following proposition within a chosen system for formal reasoning:
Prop 1.
If and are derivable then so is .
The difficulty in showing this property, and the reason why it is interesting as a challenge problem, is that it must be proved simultaneously with the following property that is called narrowing:
Prop 2.
If and are derivable then is also derivable.
In the standard informal proof, the properties are established by an induction on the structure of , with the first property being shown first and then used in showing the second. The proof of the first property requires an additional induction on the derivation of and that of the second property similarly requires an additional induction on the derivation of .
While the description of the challenge allows for small changes to the subtyping rules, such as those needed to accommodate a higher-order abstract syntax style of formalization, it places a requirement that these such changes should result in an “obviously equivalent” system. As an example, mention is made in [1] of a declarative presentation of subtyping that explicitly include rules for reflexivity and transitivity. Such a presentation obviates a proof of transitivity of subtyping and thus constitutes another “nice example how the original problem specification influences the proofs about it” in the sense hinted at at the end of Section 2 in [3]. However, it also illustrates fairly starkly how changing the “problem specification” in such ways is not acceptable because it undermines the intended challenge.
The issues with the purported higher-order solution
The claimed solution to the challenge in [3] uses an encoding of the subtyping rules in LF that is based on a higher-order abstract syntax style. However, it is unnecessary to get into the details of this encoding to illuminate the difficulty with the claims therein. For simplicity of presentation, we will adhere to a presentation style that is close to that of the original description of the challenge. We note that this style is also used at the outset in [3].
The presentation of subtyping in [3] begins by confusing subtyping in bindings in type environments with subtyping judgements. In the description of the challenge in this note, we have highlighted the distinction between the two by deliberately using the symbol for one and <: for the other. In keeping with the presentation in [1], the symbol <: is used in [3] for both.222The symbol that is actually used in [3] is but we will assume it to be <: here to ensure consistency with the earlier discussion. However, unlike in [1] where a distinction is maintained despite the overloading of notation, the two notions are conflated in [3] by the inclusion of the following rule:
A further distinction between the subtyping rules in the original presentation and the one in [3] is that the latter builds in transitivity with respect to variable subtyping directly by replacing the SA-Trans-TVar rule with the following:
The changes to the formulation of subtyping that are described above are substantial and do not satisfy the criterion of obvious equivalence required by the challenge. In particular, they considerably simplify the informal proof of transitivity of subtyping. Unlike with the original presentation of subtyping, the narrowing lemma can be proved independently of the transitivity property. Further, Prop 1, the transitivity property, can be proved by an induction only on ; the additional induction on the derivation of is not needed.333The development in [3] continues to use an induction on both but the second induction can be avoided.
In the above discussion, we have assumed that the narrowing lemma still needs to be proved. The claim in [3] is that this lemma is obtained for free. A closer look at the development shows that this “simplification” is actually the result of assuming the validity of a further rule related to subtyping that has the following form:
Observe that the sensibility of this rule depends on the conflation of subtyping in bindings in type environments with the subtyping judgement. While the narrowing lemma is an easy consequence of this rule in a situation where the SA-Tr-TVar rule is also present, the assumption of such a rule represents a significant deviation from the formulation of subtyping in Challenge 1A. We note that this rule can be shown to be admissible in the typing calculus of interest, but doing this requires an argument that is as complicated as the direct proof of narrowing. This observation raises in a different way the question of what it means to get something for free.
A comment on ordering of bindings in type environments
Typing calculi often attribute a significance to the order of bindings in contexts or type environments. In the discussions here, this sensitivity is manifest specifically in the statement of Prop 2. While order is important for well-scoping of type environments, the proofs of transitivity of subtyping and narrowing are not dependent on order. Moreover, ignoring order has the effect of simplifying the formalization of the arguments. It is possible to realize such an effect in a system like Abella [2], which treats contexts by default as multisets, by not imposing a well-scoping constraint initially on type environments. Once the desired properties have been proved when type environments have this less constrained form, it is easy to show that they must also hold when type environments are required to be well-scoped.
Acknowledgements
The observations in this note were made in the course of conducting research supported by the National Science Foundation under Grant No. CCF-1617771. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
- [1] B. E. Aydemir, A. Bohannon, M. Fairbairn, J. N. Foster, B. C. Pierce, P. Sewell, D. Vytiniotis, G. Washburn, S. Weirich, and S. Zdancewic. Mechanized metatheory for the masses: The PoplMark challenge. In Theorem Proving in Higher Order Logics: 18th International Conference, number 3603 in LNCS, pages 50–65. Springer, 2005.
- [2] D. Baelde, K. Chaudhuri, A. Gacek, D. Miller, G. Nadathur, A. Tiu, and Y. Wang. Abella: A system for reasoning about relational specifications. Journal of Formalized Reasoning, 7(2), 2014.
- [3] B. Pientka. Proof pearl: The power of higher-order encodings in the logical framework LF. In K. Schneider and J. Brandt, editors, Proceedings of the 20th International Conference on Theorem Proving in Higher Order Logics (TPHOLs), volume 4732 of LNCS, pages 246–261. Springer, 2007.