Solid Test Suite


Produce test suites that can be used to verify the compliance of any server implementation to the Solid specifications

What it is not

  • Unit or integration tests for internal use
  • Tests to verify that apps work correctly, or has certain values
  • A reimplementation of existing test suites

Existing test suites

  • LDP Test suite
  • LDN Test suite
  • SPARQL Test suite
  • ...

Reuse: Package all in a Docker container, and integrate results with Earl


  • Weekly calls
  • solid/test-suite
  • Code on github

Tests described with RDF

To describe test cases in RDF, parameterized so that very little code is needed to implement the actual test

Framework developed

  • Test::FITesque::RDF, built on a framework by Scott McWirther et al to formulate fixture tables in arbitrary data structures.
  • Currently, the framework can take key-value pairs or lists of HTTP requests and responses.
  • Framework is 180 SLOC of Perl. Test script 44-100 SLOC.
  • Thus, actual test scripts can test a number of requests, and test against expected responses. Or some other use of them.

But stop!

  • Choice of Perl was not sufficiently rooted in the organization
  • Implementation language was intended as a mere implementation detail
  • Got me scrambling to see how I could implement it
  • ...but we eventually decided to do the RDF first, and then implement
  • ...but that too gave little progress
  • Test suite call agreed that given the little code needed, I should continue my framework

RDF example

<#test-list> a test:FixtureTable ;
    test:fixtures <#public-writeread-unauthn-alt> .

<#public-writeread-unauthn-alt> a test:Test ;
    dc:description "A simple test for PUT then GET"@en ;
    test:handler "Web::Solid::Test::HTTPLists"^^deps:CpanId ;
    dc:identifier "http_req_res_list_unauthenticated" ;
    test:params [
        test:requests ( <#public-writeread-unauthn-alt-put-req> <#public-writeread-unauthn-alt-get-req> ) ;
        test:responses ( <#public-writeread-unauthn-alt-put-res> <#public-writeread-unauthn-alt-get-res> ) 
] .

<#public-writeread-unauthn-alt-put-req> a http:RequestMessage ;
            http:method "PUT" ;
            httph:content_type "text/turtle" ;
            http:content "</public/verypublic/foobar.ttl#dahut> a <> ." ;
            http:requestURI </public/verypublic/foobar.ttl> .
<#public-writeread-unauthn-alt-put-res> a http:ResponseMessage ;
    http:status 201 .

<#public-writeread-unauthn-alt-get-req> a http:RequestMessage ;
    http:method "GET" ;
    http:requestURI </public/verypublic/foobar.ttl> .
<#public-writeread-unauthn-alt-get-res> a http:ResponseMessage ;
httph:content_type "text/turtle" .

Other suites

LDN test suite can have test descriptions parsed from spec, tracks links, provenance and EARL results

Complementary to our code, consider for integration.

Integration with specification

  • The test suite can be on the level of NSS5, but not reach the goal unless test writing goes hand-in-hand with spec writing
  • How about RDFa in the spec for the tests?


  • Existing tests suites runs in a Docker container
  • Outputs some numbers:
    Server              	LDP Basic           	Websockets-pub-sub  	RDF-fixtures
    gold                	1/90                	0/1                 	3/7
    inrupt-pod-server   	56/90               	1/1                 	3/7
    node-solid-server   	15/90               	0/1                 	2/7
    trellis             	53/90               	0/1                 	4/7	
  • RDF based test suite has a just few tests, but a working framework that should cover many cases


  • Specification is not sufficiently clear on conformance
  • My JavaScript-fu
  • That it might be hard to capture the full complexity of the spec without making the RDF complex

Upcoming work

TODO: Action Plan

  • Continue writing RDF tests, modify test scripts and framework as needed (probably not a lot)
  • Consensus in test-suite group that it is a pluss to not write test suite and implementation in the same language
  • But if needed, there's not a lot of code
  • Write EARL integration (possibly integrate into Sarven's work)
  • Once specification work gets underway, work closely with that