Solid Test Suite

Goal

Produce test suites that can be used to verify the compliance of any server implementation to the Solid specifications

What it is not

• Unit or integration tests for internal use
• Tests to verify that apps work correctly, or has certain values
• A reimplementation of existing test suites

Existing test suites

• LDP Test suite
• LDN Test suite
• SPARQL Test suite
• ...

Reuse: Package all in a Docker container, and integrate results with Earl

Communication

• Weekly calls
• solid/test-suite
• Code on github

Tests described with RDF

Goal

To describe test cases in RDF, parameterized so that very little code is needed to implement the actual test

Framework developed

• Test::FITesque::RDF, built on a framework by Scott McWirther et al to formulate fixture tables in arbitrary data structures.
• Currently, the framework can take key-value pairs or lists of HTTP requests and responses.
• Framework is 180 SLOC of Perl. Test script 44-100 SLOC.
• Thus, actual test scripts can test a number of requests, and test against expected responses. Or some other use of them.

But stop!

• Choice of Perl was not sufficiently rooted in the organization
• Implementation language was intended as a mere implementation detail
• Got me scrambling to see how I could implement it
• ...but we eventually decided to do the RDF first, and then implement
• ...but that too gave little progress
• Test suite call agreed that given the little code needed, I should continue my framework

RDF example

	      
<#test-list> a test:FixtureTable ;
    test:fixtures <#public-writeread-unauthn-alt> .

<#public-writeread-unauthn-alt> a test:Test ;
    dc:description "A simple test for PUT then GET"@en ;
    test:handler "Web::Solid::Test::HTTPLists"^^deps:CpanId ;
    dc:identifier "http_req_res_list_unauthenticated" ;
    test:params [
        test:requests ( <#public-writeread-unauthn-alt-put-req> <#public-writeread-unauthn-alt-get-req> ) ;
        test:responses ( <#public-writeread-unauthn-alt-put-res> <#public-writeread-unauthn-alt-get-res> ) 
] .

<#public-writeread-unauthn-alt-put-req> a http:RequestMessage ;
            http:method "PUT" ;
            httph:content_type "text/turtle" ;
            http:content "</public/verypublic/foobar.ttl#dahut> a <http://example.org/Cryptid> ." ;
            http:requestURI </public/verypublic/foobar.ttl> .
          
<#public-writeread-unauthn-alt-put-res> a http:ResponseMessage ;
    http:status 201 .

<#public-writeread-unauthn-alt-get-req> a http:RequestMessage ;
    http:method "GET" ;
    http:requestURI </public/verypublic/foobar.ttl> .
            
<#public-writeread-unauthn-alt-get-res> a http:ResponseMessage ;
httph:content_type "text/turtle" .

Other suites

LDN test suite can have test descriptions parsed from spec, tracks links, provenance and EARL results

Complementary to our code, consider for integration.

Integration with specification

• The test suite can be on the level of NSS5, but not reach the goal unless test writing goes hand-in-hand with spec writing
• How about RDFa in the spec for the tests?

Status

• Existing tests suites runs in a Docker container
• Outputs some numbers:

  Server              	LDP Basic           	Websockets-pub-sub  	RDF-fixtures
  gold                	1/90                	0/1                 	3/7
  inrupt-pod-server   	56/90               	1/1                 	3/7
  node-solid-server   	15/90               	0/1                 	2/7
  trellis             	53/90               	0/1                 	4/7	

• RDF based test suite has a just few tests, but a working framework that should cover many cases

Risks

• Specification is not sufficiently clear on conformance
• My JavaScript-fu
• That it might be hard to capture the full complexity of the spec without making the RDF complex

Upcoming work

TODO: Action Plan

• Continue writing RDF tests, modify test scripts and framework as needed (probably not a lot)
• Consensus in test-suite group that it is a pluss to not write test suite and implementation in the same language
• But if needed, there's not a lot of code
• Write EARL integration (possibly integrate into Sarven's work)
• Once specification work gets underway, work closely with that