Solid Test Suite Update September 2019

Goal

Overarching goal

Produce test suites that can be used to verify the compliance of any server implementation to the Solid specifications

Goal up to now

Bring the framework to a state where it can comfortably support spec writing

What it is not

Finished
Unit or integration tests for internal use
Tests to verify that apps work correctly, or has certain values
A reimplementation of existing test suites
A complete verification of certain workflows, like Data Portability

Specification Writing Integration

Current spec is too loose to derive clear conformance critieria
- Good tests can't be written
Spec writing will gain from having test suite developed in parallell
- So far, the suite consists of smaller examples
- E.g. authentication-oidc-tests from NSS5

Authentication - the easy way out

Struggled with worries around interactivity

Clearly separate system under test from system doing testing

Identity Provider isn't system under test

So it could provide a bearer token to anyone that asks

Relative URIs are convenient

Tests must be reusable without change across different systems under tests
No hardcoded https://server:3000/foo/bar in the RDF
All URIs for resources on the system under test are relative
But this mechanism can only be used for one host
Lead to a public, fake IDP living on https://idp.test.solidproject.org/

Anatomy of tests


:test_list a test:FixtureTable ;
    rdfs:comment "These tests have been strongly influenced by NSS' authentication-oidc-test.js"@en ;
    test:fixtures (
        :setup
        :get_unauth
        :get_auth_alice
        :get_auth_bob
        :teardown
) .


:get_auth_alice a test:AutomatedTest ;
    test:purpose "Authenticated as Alice GET operations on resources."@en ;
    test:test_script <http://example.org/httplist#http_req_res_list> ;
    test:params [
        <http://example.org/httplist/param#bearer> <https://idp.test.solidproject.org/tokens/ALICE_ID_GOOD> ;
        test:requests (
            :get_private_for_alice_req
            :get_own_origin_private_for_alice_req
            :get_alice_trust_origin_private_for_alice_req
            :get_other_origin_private_for_alice_req
        ) ;
        test:responses (
            :get_ok_res
            :get_ok_res
            :get_ok_res
            :origin_unauthz_res
        )
	] .

Structure differs from previous all-hands:


<http://example.org/httplist#http_req_res_list>
    a nfo:SoftwareItem ;
    deps:test-requirement "Web::Solid::Test::HTTPLists"^^deps:CpanId ;
    nfo:definesFunction "http_req_res_list" .

Issue open to make this URI dereferenceable


:get_alice_trust_origin_private_for_alice_req a http:RequestMessage ;
    http:method "GET" ;
    httph:origin <https://goodapp.example> ; # Given in Alice's profile as trusted
    http:requestURI </test-auth/private-for-alice.txt> .


:get_ok_res a http:ResponseMessage ;
    http:status 200 .

A lot of open space around this

Something entirely different


:check_acl_location a test:AutomatedTest ;
    test:purpose "Determine location and write ACL document"@en ;
    test:test_script <http://example.org/httplist#http_req_res_list_location> ;
    test:params [
        <http://example.org/httplist/param#bearer> <https://idp.test.solidproject.org/tokens/ALICE_ID_GOOD> ;
        test:requests ( :check_acl_location_req :put_new_acl_req ) ;
        test:responses ( :check_acl_location_res :put_new_acl_res )
    ] .


:check_acl_location_res a http:ResponseMessage ;
    httph:link '<(.*?)>;\\s+rel="acl"'^^dqm:regex ;
    http:status 200 .