Solid Test Suite Update December 2019

Goal

Overarching goal

Produce test suites that can be used to verify the compliance of any server implementation to the Solid specifications

What it is not

  • Finished
  • Unit or integration tests for internal use
  • Tests to verify that apps work correctly, or has certain values
  • A reimplementation of existing test suites
  • A complete verification of certain workflows, like Data Portability

Project up to now

  • I have tightened the framework code
  • Incorporated the feedback from Oxford meeting
  • Made many experiments to see how
    • the RDF can be made most human-readable
    • have a high degree of test reuse
  • Created an EARL formatter for test results
  • Started writing tests to verify Solid as of now
  • I've been working mostly with Sarven on the specification

The test suite

Test suite components

Test::FITesque::RDF

aka "The Framework"

The release page

Anatomy of tests, updated


:test_bob_r_nr a test:AutomatedTest ;
    test:purpose "Check that Bob can only read Non-RDF resource when he is authorized read only."@en ;
    test:test_script <http://example.org/httplist#http_req_res_list> ;
    test:params [
        <http://example.org/httplist/param#bearer> <https://idp.test.solidproject.org/tokens/BOB_ID_GOOD> ;
        test:steps (
            [
                test:request :test_bob_nr_read_req ;
                test:response_assertion :ok_res ;
            ]
            [
                test:request :test_bob_nr_head_req ;
                test:response_assertion :ok_res ;
	    ]
	    [ ... ]	
        )
    ] .

TAP::Formatter::EARL

  • TAP = Test Anything Protocol
  • EARL = Evaluation and Report Language
  • Can be used on any TAP output
  • All testers should output EARL for reports
  • Current formatter creates only basic output
  • I wasn't impressed with available parser classes

At crossroads

  1. Complete focus on specification writing
  2. Write tests for current state of Solid
  3. Pursue Beautiful Linked Data

State of the Art Tests

To put firmly down how Solid works now.

Invalid authentication tokens

Test status codes returned by using all HTTP methods with varying permissions for

  • RDF sources
  • Non-RDF sources
  • Containers

Without using LDP

Findings

  • NSS5 has a problematic relationship towards invalid tokens
  • Solid does not require client to supply LDP interaction models in NSS5
  • With the hierarchy decision, LDP client-supplied interaction models are technically redundant
  • Suggests that the important part of LDP are the LDP-server-managed triples

Test reuse within the framework

  • Dimensions can be added: E.g. LDP interaction model
  • Imaginebly many dimensions. Likely to see a combinatorial explosion.
  • Simpler: Merging files to add relevant tests or generate RDF using SPARQL CONSTRUCT

Lament of the lonesome developer

Largely been a one-man show

Difficult to incorporate feedback when it occurs on a quarterly basis

Now, lets write specs