We use Liquibase in our project as a DB change management tool. We use it to create our DB schema with the basic configuration our application needs to run.
This is, however, not enough for development or (unit) testing. Why? Because for each test case, we need to have data in the database in a particular state. E.g. I need to test that the system rejects the action of a user that does have the required quota for the action he/she wants to do. So I create a user with a 0 quota and then try to perform the action and see whether the system allows it or rejects it. To not waste time setting our test data repeatedly, we use special Liquibase scripts that set up what we need in our test environment (such as a user with a 0 quota), so that we do not have to do this manually.
For the Payment System project, we used Liquibase to run plain SQL scripts to insert the data that we needed. It worked well enough, but this approach has some disadvantages.
Mainly, the person writing the scripts has to have knowledge of our database and the relations between various tables, so there is a steep learning curve. Another issue is that all of the scripts have to be updated when a larger DB schema change takes place.
Therefore, during our implementation of the Quota System, I took inspiration from the work of my colleague, who used a kind of high-level DSL as a convenient way to setup the Quota system and I turned it into a production-ready feature on top of Liquibase. This solved the problem of manual execution (the scripts always run during the application startup and are guaranteed to run exactly once).
For the DSL, I chose Groovy, since we already use it for our tests, and there is no interoperability issue with Java based Liquibase.
Liquibase has many extension points and implementing a custom changelog parser seemed the way to go.
The default parser for XML parses the input file, generates internal Liquibase objects representing particular changes, and then transforms them to SQL, which is executed on the DB.
I created similar a parser, which was registered to accept Groovy scripts.
The parser executes the script file, which creates internal Liquibase objects, but, of course, one DSL keyword can create more insert statements. What is more, when a DB schema changes, the only place that needs to change is the DSL implementation, not all of the already created scripts.
Example Groovy script:
importFile('changelog.xml')
bw = '3' //using guid of identifier which is already present in database
color = identifier name:'color'
print = identifier name:'PRINT', guid:'100' //creating identifier with specific guid
q1 = quota guid:'q1', name:'quota1', limit:50, identifiers:[bw, print], period:weekly('MONDAY')
q2 = quota guid:'q2', name:'quota2', limit:150, identifiers:[color, print], period:monthly(1)
for (i in 1..1000)
quotaSubject guid: 'guid' + i, name:'John' + i, quotas:[q1,q2]
This example shows that the Groovy script can reference another Liquibase script file – e.g., the default changelog file, which creates the basic DB structure and initial data.
It also shows the programmatic creation of quotaSubjects (accounts in the system), where we can use normal Groovy for a loop to simply create many accounts for load testing.
QuotaSubjects have assigned quotas, which are identified by quota identifiers. These can be either created automatically, or we can reference already existing ones.
The keywords identifier, quota, quotaSubject, weekly, and monthly are just normal Groovy functions, that take Map as an argument, which allows us to pass them named parameters.
Before execution, the script is concatenated with the main DSL script, where the keywords are defined.
Part of the main DSL script that processes identifiers:
QuotaIdentifier identifier(Map args) {
assert args.name, 'QuotaIdentifier name is not specified, available params are: ' + args
String guid = args.guid ? args.guid : nextGuid()
addInsertChange('QUOTA_IDENTIFIER', columns('GUID', guid) << column('NAME', args.name) << column('STATUS', 'ENABLED'))
new QuotaIdentifier(guid)
}
private def addInsertChange(String tableName, List<ColumnConfig> columns) {
InsertDataChange change = new InsertDataChange()
change.setTableName(tableName)
change.setColumns(columns)
groovyDSL_liquibaseChanges << change
}
The calls produce Liquibase objects, which are appended to variables accessible within the Groovy script. The content of the variables constitutes the final Liquibase changelog, which is created after the processing of the script is done.
This way, the test data file is simple to read, write, and maintain. A small change in the changelog parser also allowed us to embed the test data scripts in our Spock test specifications so that we can see the test execution logic and test data next to each other.
@Transactional @ContextConfiguration(loader = YSoftAnnotationConfigContextLoader.class)
class QuotaSubjectAdministrationServiceTestSpec extends UberSpecification {
// ~ test configuration overrides ==================================================================
@Configuration @ImportResource("/testApplicationContext.xml")
static class OverridingContext {
@Bean
String testDataScript() {
"""groovy: importFile('changelog.xml')
bw = identifier name:'bw'
color = identifier name:'color'
print = identifier name:'print'
a4 = identifier name:'a4'
p_a4_c = quota guid:'p_a4_c', name:'p_a4_c', limit:10, identifiers:[print, a4, color], period:weekly('MONDAY')
p_a4_bw = quota guid:'p_a4_bw', name:'p_a4_bw', limit:20, identifiers:[print, a4, bw], period:weekly('MONDAY')
p_a4 = quota guid:'p_a4', name:'p_a4', limit:30, identifiers:[print, a4], period:weekly('MONDAY')
p = quota guid:'p', name:'p', limit:40, identifiers:[print], period:weekly('MONDAY')
q1 = quota guid:'q1', name:'q1', limit:40, identifiers:[print], period:weekly('MONDAY')
q2 = quota guid:'q2', name:'q2', limit:40, identifiers:[print], period:weekly('MONDAY')
q_to_delete = quota guid:'q_to_delete', name:'q_to_delete', limit:40, identifiers:[print], period:weekly('MONDAY')
quotaSubject guid: 'user1', name:'user1', quotas:[p_a4_c, p_a4_bw, p_a4]
quotaSubject guid: 'user2', name:'user2', quotas:[p_a4_c, p_a4_bw, p_a4]
quotaSubject guid: 'user3', name:'user3', quotas:[p_a4_c, p_a4_bw, p_a4, p]
quotaSubject guid: 'user4', name:'user4', quotas:[p_a4_c]"""
}
}
// ~ instance fields ===============================================================================
@Autowired private QuotaSubjectAdministrationService quotaSubjectAdministrationService;
@Autowired private QuotaAdministrationService quotaAdministrationService;
// ~ findByGuid ====================================================================================
def "findByGuid should throw IllegalInputException for null guid"() {
when:
quotaSubjectAdministrationService.findByGuid(null)
then:
thrown(IllegalInputException)
}