Parametrized XCTest in Swift
The problem
I'm coding a lib for video resizing. Crop, size to fit/fill etc. I need an extensive test coverage to verify calculations.
Goals
I wanted to generate a huge set of input parameters with corresponding output to assert with. Then pass them to my XCTest as an input and admire all green checkmarks in Xcode tests report for all items of the parameters set separately.
But Xcode didn't allow me to do it straightforwardly. Luckily I've found a couple of workarounds.
Workaround #1 - You haven't seen it
The first idea is to simply run a forEach loop for all the parameters and perform assertion inside.
typealias TestCaseParameter = (input: Int, output: Int)
let parameters = [TestCaseParameter] = [(1, 2), (3, 4), (5, 6), (1, 1)]
parameters.forEach {
XCTAssertEqual($0.input, $0.output)
}
Awful solution. It doesn't let us know which of the parameters has passed successfully and which of them failed. The XCTest will be represented as a single test in the report.
Workaround #2 - Dynamically add tests to XCTestSuite
At first sight it seemed like the intended way of doing that sort of things. Buuuut still not exactly.
typealias ParametrizedTestExampleParameter = (input:(a: Float, b: Float), output: Float)
class ParametrizedTestExample: XCTestCase {
private var parameter: ParametrizedTestExampleParameter? = nil
override open class var defaultTestSuite: XCTestSuite {
let testSuite = XCTestSuite(name: NSStringFromClass(self))
addTestsToInvocationsWithParameters((input: (12, 3), output: 9), toTestSuite: testSuite)
addTestsToInvocationsWithParameters((input: (12, 2), output: 10), toTestSuite: testSuite)
addTestsToInvocationsWithParameters((input: (12, 4), output: 8), toTestSuite: testSuite)
addTestsToInvocationsWithParameters((input: (12, 5), output: 7), toTestSuite: testSuite)
return testSuite
}
private class func addTestsToInvocationsWithParameters(_ parameter: ParametrizedTestExampleParameter, toTestSuite testSuite: XCTestSuite) {
testInvocations.forEach { invocation in
let testCase = ParametrizedTestExample(invocation: invocation)
testCase.parameter = parameter
testSuite.addTest(testCase)
}
}
func substract(a: Float, b: Float) -> Float {
return a - b
}
func testSubstract() {
guard let parameter = parameter else {
XCTFail("Test has no input parameter")
return
}
XCTAssertEqual(parameter.output, substract(a: parameter.input.a, b: parameter.input.b))
}
}
All parametrized tests are represented as a single test in report, but with several test scheme actions. Much better, but still no way to know which of the items in the list corresponds to which parameter.
Another disadvantage is that failure in any of the parameters fails the whole test:
Please, ping me if anyone found a way to turn this approach into something useful.
Workaround #3 - Dynamically add test methods via Obj-C Runtime
God bless ManWithBear. He shared an example of adding test methods dynamically via Obj-C Runtime.
Personally, I hate the idea of adding methods to any class during runtime. It's so weird and unpredictable way to bring chaos to the code.
But this time it turned out to be the best (or the only?) solution I could find.
All you need to do is to add .h file with base ParamtrizedTestCase interface:
#ifndef ParametrizedTestCase_h
#define ParametrizedTestCase_h
#endif /* ParametrizedTestCase_h */
#import <XCTest/XCTest.h>
/// SEL is just pointer on C struct so we cannot put it inside of NSArray.
/// Instead we use this class as wrapper.
@interface _QuickSelectorWrapper : NSObject
- (instancetype)initWithSelector:(SEL)selector;
@end
@interface ParametrizedTestCase : XCTestCase
/// List of test methods to call. By default return nothing
+ (NSArray<_QuickSelectorWrapper *> *)_qck_testMethodSelectors;
@end
.m file with implementation:
#import <Foundation/Foundation.h>
#include "ParametrizedTestCase.h"
@interface _QuickSelectorWrapper ()
@property(nonatomic, assign) SEL selector;
@end
@implementation _QuickSelectorWrapper
- (instancetype)initWithSelector:(SEL)selector {
self = [super init];
_selector = selector;
return self;
}
@end
@implementation ParametrizedTestCase
+ (NSArray<NSInvocation *> *)testInvocations {
// here we take list of test selectors from subclass
NSArray<_QuickSelectorWrapper *> *wrappers = [self _qck_testMethodSelectors];
NSMutableArray<NSInvocation *> *invocations = [NSMutableArray arrayWithCapacity:wrappers.count];
// And wrap them in NSInvocation as XCTest api require
for (_QuickSelectorWrapper *wrapper in wrappers) {
SEL selector = wrapper.selector;
NSMethodSignature *signature = [self instanceMethodSignatureForSelector:selector];
NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
invocation.selector = selector;
[invocations addObject:invocation];
}
/// If you want to mix parametrized test with normal `test_something` then you need to call super and append his invocations as well.
/// Otherwise `test`-prefixed methods will be ignored
return invocations;
}
+ (NSArray<_QuickSelectorWrapper *> *)_qck_testMethodSelectors {
return @[];
}
@end
And do not forget about bridging header with just a single line
#include "ParametrizedTestCase.h"
How does it work
Base XCTestCase has an array of NSInvocations that contains invocations for each of test methods. So we override it in our base ParametrizedTestCase implementation.
We can think of NSInvocation as a wrapper over a method that we want to test.
It wraps method Signature and Selector. Where Signature contains method name and its parameters list while Selector is a string representation of method name, used in Obj-C Runtime.
Usage
class RuntimeTestsExample: ParametrizedTestCase {
/// This is our parametrized method. For this example it just print out parameter value
func p(_ s: String) {
print("Magic: \(s)")
}
override class func _qck_testMethodSelectors() -> [_QuickSelectorWrapper] {
/// For this example we create 3 runtime tests "test_a", "test_b" and "test_c" with corresponding parameter
return ["a", "b", "c"].map { parameter in
/// first we wrap our test method in block that takes TestCase instance
let block: @convention(block) (RuntimeTestsExample) -> Void = { $0.p(parameter) }
/// with help of ObjC runtime we add new test method to class
let implementation = imp_implementationWithBlock(block)
let selectorName = "test_\(parameter)"
let selector = NSSelectorFromString(selectorName)
class_addMethod(self, selector, implementation, "v@:")
/// and return wrapped selector on new created method
return _QuickSelectorWrapper(selector: selector)
}
}
}
_qck_testMethodSelectors func does the following thing:
Takes method to test with array of parameters.
- Every pair of (method, parameters) is wrapped into clojure
- Creates Selector for every clojure, with the name starting with "test_"
- Wraps each selector into
_QuickSelectorWrapper - Returns array of selector wrappers.
_QuickSelectorWrapper is just anNSObject that wrapps Selector. It's a workaround to use NSArray of selectors: we can't add Selector to NSArray directly as it's just a simple C String.
We override _qck_testMethodSelectors method in our TestCase implementation, while base ParametrizedTestCase uses it to get an array of selector wrappers and create array of NSInvocations for them.
Uuuh, finally explained.
All the difficulties above are caused by the fact we can't use NSInvocation in Swift directly. So we have to bridge and do it in Obj-C.
Not Great, Not Terrible
Ugly but working way of solving the problem and achieving the main goal.
Thanks to method Selector naming, each pair of (method, parameter) is represented as a separate test method, allowing to check, which parameters caused test failure and what's the success rate of the test on the whole set of parameters:
I'm sure, you've already found some behaviour or data driven test framework, that allows to do the same thing. If so, ping pls or give a link right in comments. Thanks in advance!
Comments