Testing 5d-6d dualities with fractional D-branes

6d SCFTs compactified on a circle can often be studied from nonperturbative 5d super-Yang-Mills theories, using instanton solitons. However, the 5d Yang-Mills theories with 6d UV fixed points frequently have too many hypermultiplet matters, which makes it difficult to use the ADHM techniques for instantons. With the examples of 6d N=(1,0) SCFTs with Sp(N) gauge symmetry and 2N+8 fundamental hypermultiplets, we show that one can still make rigorous studies of these 5d-6d relations in the 'fractional D-brane sectors'. We test the recently proposed 5d duals given by Sp(N+1) gauge theories, and compare their instanton partition functions with the elliptic genera of 6d self-dual strings.