Verification methodology

OGR Slip2D is verified by comparing what it computes with something it did not produce: a factor of safety published in a paper, a survey or a reference manual, a closed-form solution, or an identity that has to hold exactly. This page describes the method. The cases themselves are set out in Section 4.2 and Section 4.3.

4.1.1The principle: an external reference#

How far a factor of safety can be trusted depends on the evidence behind the program that computed it. The usual evidence, that the program gives today the number it gave yesterday, is weak: a test that compares the output with a recording of the program's own earlier output protects against change, not against error, and if the recording contains a fault the test enforces it. OGR Slip2D is therefore verified against references that it did not produce, and a comparison with its own output is not recorded as verification.

The rule is the first condition that the contribution rules of the project set for a new method, solver or formula: it needs a reference value (a published case, a closed-form solution or an analytic property) and not a snapshot of what the code currently prints (CONTRIBUTING.md). The Verification page summarises the outcome for a first reading; this chapter gives it case by case.

4.1.2Four kinds of reference#

Table 4.1.1 lists the kinds of reference that count, with an example of each and the section where it is used.

4.1.3Checking the search and checking the method#

A published factor of safety can answer two different questions, and a case is built according to the one its source allows.

• A search case. The source gives the minimum factor of safety of a slope. The case runs the search that its model declares and compares the critical factor of safety of each declared method with the published one, so the search and the method are checked together: a search that misses the critical circle returns a higher factor of safety and the case fails. Cases 001 to 006 are of this kind.
• A stated-surface case. The source states the slip surface and gives the factor of safety on it. The case evaluates each method on that surface, with no search, so the number measures the method alone. Case 007 is of this kind.

Only the second kind can tell one method from another. On a critical circle that each method finds for itself, the difference between two methods mixes the method with the search, and a method that returned the answer of a simpler one could pass unnoticed (Section 4.1.5).

4.1.4How the tolerance of a case is chosen#

The tolerance of a case is the relative difference it accepts between the factor of safety computed by OGR Slip2D and the reference value:

where

$F$

factor of safety computed by OGR Slip2D

$F_{\mathrm{ref}}$

reference value of the case (fos in its esperado.json)

$\varepsilon$

tolerance of the case (tolerancia_relativa), as a fraction: 0.02 is 2 %

The tolerance lives in the case and not in the test, because it depends on the quality of the source: a value read from a figure does not deserve the precision of a tabulated one. It is chosen by asking three questions of the source.

1. How far do the published numbers disagree with one another? A survey publishes a referee value and the mean of the programs that took part, and they differ. A tolerance narrower than that spread would make the case fail because of the quality of its reference.
2. What does the source leave unstated? A parameter that the problem does not give has to be deduced or assumed, and the effect of that choice on the factor of safety is part of the tolerance.
3. Which published value has statistical or independent backing? The mean of many programs is preferred to the result of one program, so that the output of no particular program is written into a case.

The test of the case files fails a case that declares no tolerance, or one above 10 %, because beyond that a validation stops discriminating anything. Table 4.1.2 gives the tolerance of each case with the reason recorded in its case file.

Where a source publishes several numbers, the case states which one it uses and why, and for a method that the source gives no value for the case declares none. The ACADS survey, for instance, gives for problems 1(a) and 1(c) a mean Bishop value and a mean of all the programs together, but none for the individual rigorous methods; Spencer and GLE / Morgenstern–Price are declared only in case 007, where the problem states the circle and the survey gives a referee value. A test of the suite guards the rule for cases 001 to 004 and requires that at least one case separates a rigorous method from Bishop's against a published number.

4.1.5When a case fails#

A failing case is not edited to make it pass. If a reference value is not reproduced, the fault is in the code or in the expected value, and a validation case exists to tell which. Neither the expected value nor the tolerance is adjusted to the output: the tolerance belongs to the source, and the expected value is a fact about a publication.

One defect shows why the kind of case matters. Until version 0.1.106 the Spencer method (Spencer, 1967) and the GLE / Morgenstern–Price method (Morgenstern & Price, 1965) returned, on a circular surface, the factor of safety of Bishop's simplified method (Bishop, 1955) to six figures. No case of the folder could see it: each compared the minimum found by a search with a published minimum, and Bishop's value is a plausible answer. The defect, which was in fact three defects, lasted some eighty versions. Case 007 was written to make it visible. The circle is stated by the problem, and the two published references, a Bishop mean of 2.204 and a referee value of 2.29, differ by 3.9 %, so a method of complete equilibrium that returns Bishop's number fails. The Spencer method of version 0.1.105 gave 2.2088 on that circle, more than 3 % below the referee value and outside the 2 % tolerance of the case. The measurements are in the audit kept in the repository.

4.1.6The validation folder#

The cases live in the validacion/ folder of the repository, one subfolder per case, numbered in the order in which they were added:

validacion/
├── README.md
└── casos/
    ├── 000-plantilla/         template; the runner skips it because it has no model
    ├── 001-acads-1a/
    │   ├── caso.md            what the case is and where its expected value comes from
    │   ├── modelo.ogr         the model, ready to open in the program
    │   └── esperado.json      expected values, tolerance and how to run the case
    └── ...

The .ogr file is the model of the case, as plain JSON (Section 5.4). Table 4.1.3 lists what esperado.json holds and what the runner does with it.

The model, the expected value and its source are versioned; project files of other programs, screenshots of other programs and third-party reports are not. The distinction is one of copyright: the .ogr file is the project's own, and the expected number is a fact that can be cited.

4.1.7Running the cases#

One test, tests/test_validation_cases.py, walks every case folder that has a model and an esperado.json, so adding a case needs no code. The project uses its own test runner and pytest is not required; the test modules cannot be imported outside the runner. From a clone of the repository (Section 1.1.5):

python tests/_runner.py validation_cases     # the seven cases and the checks of their files
python tests/_runner.py published_cases      # the checks that accompany them

A selection by file name prints a FILTERED RUN warning around its totals and is not evidence for a release; the whole suite, which is what a release needs, is run with no arguments. Besides reproducing the expected values, the test checks that every folder has its three files, that esperado.json is valid, that every case cites a source and declares a sane tolerance, that the method identifiers and the search types exist, and that at least one case actually runs, so that an empty folder cannot pass unnoticed.

To add a case: build the model in the program and save it as modelo.ogr; write caso.md with the geometry, the materials and, above all, where the expected number comes from, because without a source an expected value is only an opinion; fill in esperado.json; and run the test. New cases are welcome: a slope whose factor of safety is known from a publication or from a hand calculation, with its source (open a validation case).

4.1.8Rules that complement the cases#

A reference value cannot see every kind of fault, and the contribution rules of the project add three more habits to the cases.

• No setting may do nothing. Every new option comes with a test that checks that it changes the result, because a control with no effect leads the user to believe that the analysis honours it.
• Anomalies are reported before they are corrected. A behaviour that looks wrong is described, with the evidence, before the code is changed.
• Tests do not leak state. The language, the units and the caches are restored, because a leak makes unrelated tests fail only in a full run.

4.1.9What the verification does not show#

4.1.10References#

• Bishop, A.W. (1955). The use of the slip circle in the stability analysis of slopes. Géotechnique, 5(1), 7–17.
• Morgenstern, N.R. & Price, V.E. (1965). The analysis of the stability of general slip surfaces. Géotechnique, 15(1), 79–93.
• Spencer, E. (1967). A method of analysis of the stability of embankments assuming parallel inter-slice forces. Géotechnique, 17(1), 11–26.