Outsystems grey box testing

While I was working at one of my customers I learned from colleagues(dev and test) to create a test repository in Outsystems as an artifact to test the system under test. While this artifact was mostly used for blackbox purposes (end to end) I think you can also set it up for gray box testing. I did not like the bdd framework in the forge as I do not see value in bdd testing if it is not a process test, and only a simple unit which developers look at(speaking in the mindset of a tester). Besides that I think it was too maintenance prone and you cannot quickly add more tests. So I thought to show an alternative (which has its cons but something can be interesting for certain clients/companies)

I Just published a linkedin post on how easy it is to set up a “unit” graybox test of an outsystems service action via a rest expose. To provide more proof :) I will show you the handson example I created in 1 hour from scratch, adding this in a cicd pipeline in gitlab. (and 1.5 hour of creating this post, so 2.5 hours)

Why can this be handy over setting up a bdd test in the forge!

First: In my thought collobaration: Most testers have knowledge about postman, and embedding your unit/module under test can provide a other mindset then a developer usually has (negative testing etc).

Second: Second thing I think of is easy scaleability in cicd once you have this simple proces up and running , think of testing a whole business flow instead of a module, which can have huge impact if it is not working properly with every change you make.

Third: you really can add and change tests super quickly if you are proficient with postman

Cons: you have to make a testartifact espace and you can only use it for public exposed actions, so business processes. For private actions you still should use the bdd framework unit test from the forge.

Ok let me explain handson what I did in order to test the public calculation server action

Step 1: Create the server action. The server action does nothing more then adding integers in which an expected output is provided. So you can add to inputs e.g. adding 1 + 1 equals 2.

Step 2: I created a rest api in which i call the server action (unit under test I would call it). This rest api I expose to the outside world as a test artifact.

After deploying you will see an exposed rest api

Step 3: Once that was created some postman tests in which I set up the entry and exit criteria for the unit under test, in this case the 1 + 1 test. This is pretty simple and usually as a tester I would advise to test the flow which has most risk/damage if we deploy this to production, coverage is not the main decision criteria for me but that is another topic :-)

The Test of this server action example (calculator) is checking if the calculation goes ok, and or if it is not allowed to put a string, But of course you can think of all kinds of other unit test I did not think of(including performance — the rest api wrapper time). All tests in the collection can be called and will be run in sequence. you can organize it in such a way that one collection is a server action e.g. or a subset of server actions, whatever suits your need. In postman you can also document easily what the goal is of a certain system under test(in this case a unit test

Step 4: Embedding the postscript in cicd gitlab (newman). This is slightly more tricky as you have to know outsystems as well as javascript (nodejs) and gitlab. But once you know the basics it is really nothing more than copy paste and you can focus on the real deal, and that is the unit greybox test.

First of all create a repository in gitlab and clone this to your favorite operating system and use your favorite code editor (mine visual studio code). After that install newman (npm install -g newman). Once that is installed install html extra with npm to create some fancy reporting.

Then create a javascript file in which you put the following code (unittest.js)

Also good to mention if you want to run collections in parallel you can easily use the power of javascript as the the execution of 1 to n collections is not written in a async/await mode. (note a postman collection runs in sequence), and this make it very handy if you have 100+ collections to test and want fast results

At last you need to define the gitlab-ci.yml file

Step 5: Running the build and get nice reporting as an Artifact in gitlab. Once checking in gitlab the pipeline will run and jeejjjj…. it builded perfectly after one go!

on the right hand side you can browse to the job Artifacts

in the Artifacts section click on the link name and the browser will open this html file which looks like below, Titles etc you can change in the setting to your liking, it has dark light mode. Also you can downdrill to the real issue in the unit test, to validate if the test is incorrect or the software

Hope this helps as an alternative, in some customer situations you can use the forge bdd/unit test frameworks, but just if you have some skillset also in javascript it can be very interesting to create a lot of tests easily to maintain, happy testing! Gitlab repo can be found here

Related posts:

This time feels like the end of something at least for a few months. Maybe forever though. I am processing this in a much different way, one where I’m not necessarily sad but relieved. Maybe I’ve had…

Mercedes grew up privileged, as you would expect given her parentage. Her father was Harold Duke, San Andreas Congressman with his eye on the senate, and her mother was Margaret Sullivan, heiress and…

If you check around the web looking for the best affiliate niches in 2020, you get a lot of the same old crap. It’s people who did 15 minutes of research and came up with the same topics last year…