At Y Soft, we use robots to test our solutions for verification and validation aspects, we are interested if the system works according to required specifications and what are the qualities of the system. To save time and money, it is possible to use a single robot to test multiple devices simultaneously. How is this done? It is very simple so let’s look at it.

When performing actions to operate given a device, the robot knows where the device is located due to a calibration. The calibration file contains transformation matrix that can transform location on the device to robot’s coordinate system. The file also contains information about the device that is compatible with the calibration. How the calibration is computed is covered in this article. There is also calibration of the camera that contains information about the region of interest, meaning where exactly is the device screen located in the view of a camera. All of the calibration files are stored on the hard drive.

Example of calibration files for two Terminal Professionals:

{  
   "ScreenXAngle":0.35877067027057225,
   "DeviceId":18,
   "DeviceModelId":18,
   "DeviceName":"Terminal_Professional 4, 10.0.5.182",
   "MatrixArray":[[0.728756457140,0.651809992529,-0.159500429741,75.4354297376],[-0.683749126568,0.734998176419,-0.10936964140,71.1249458777],[0.0422532822652,0.187897834122,0.981120733880,-34.923427696],[0.0,0.0,0.0,1.0]] ,
   "ScreenSize":{  
      "Width":153.0,
      "Height":91.0
   }
}
{  
   "ScreenXAngle":0.25580856461537194,
   "DeviceId":27,
   "DeviceModelId":18,
   "DeviceName":"Terminal_Professional 4, 10.0.5.112",
   "MatrixArray":[ [0.713158843830,-0.686471581194,0.220191724515,-176.983055],[0.699596463347,0.6783511825194,-0.15148794414,-71.7788394],[-0.05297850752,0.2635031531279,0.963621817536,-29.83848504],[0.0,0.0,0.0,1.0] ],
   "ScreenSize":{  
      "Width":153.0,
      "Height":91.0
   }
}


For the robot to operate on multiple devices, all of the devices must be within the robot’s operational range, which is quite limited, so this feature is currently is only used for smaller devices, like mobile phones and Terminal Professional. It is theoretically possible to use a single robot on more devices, but for practical purposes, there are usually only 2 devices. Also, all devices must be at relatively the same height, which limits testing on multifunctional devices that have various height and terminal placement. Space is also limited by the camera’s range, so multiple cameras might be required, but this is not a problem as camera calibration also contains the unique identifier of a camera. Therefore a robot can operate on multiple devices using multiple cameras or just a single camera if devices are very close to each other.

Before testing begins, a robot needs to have all calibrations of devices available on the hard drive and all action elements (buttons) need to be within its operational range. Test configuration contains variables such as DEVICE_ID and DEVICE2_ID which need to contain correct device IDs as stored the robot’s database. Which tests will run on the devices and the duration of the tests also need to be specified. Tests used for these devices are usually measurement and endurance tests, which run in iterations. There are multiple variants for these tests, for example, let’s say we wish to run tests for 24 hours on two devices and each device should have an equal fraction of this time. This means that the test will run for 12 hours on one device and 12 hours on the other, which is called consecutive testing. Another variant is simultaneous testing, which means that the robot will alternate between the devices after each iteration for a total time of 24 hours. The robot loads the calibration for another device after each iteration and continues with the test on that device.  This is sometimes very useful should one device become unresponsive, the test can continue on the second device for the remaining time. Results of each iteration of the test for each device are stored in a database along with other information about the test and can be viewed later.

Testing multiple devices with a single robot also makes it possible to test and compare different versions of an application or operating system (in this case on Terminal Professional) without ending the test, reinstalling of the device and running the test again. This saves a lot of time and makes the comparison more accurate.

This is an introduction to UAM project – management server for Universal appliances (new TPv4, eDee and SafeQube) and the Vert.x framework on which it is developed.

There was a need for easy installation and upgrade of OS and applications running on an UA so we needed a system to manage UAs with as little user interaction as possible.
The main requirements are robustness (we will deal with unrealiable networks and possible power outages), small executables (there is limited storage on the SafeQube) and little disk access (we don’t want to wear out our SafeQube flashdisks).

UAThe system has 2 parts:

  • UA management server – server application which will be deployed next to CML and will contain APIs for system administration
  • UA management server proxy – application cluster which will be deployed on SafeQubes and will cache application binaries and configuration

Both applications are developed using Vert.x framework. UAM server uses also database as configuration storage and proxy applications use Hazelcast for clustering.

Vert.x is an application framework for programming servers (especially web servers). It is built on top of Netty so no application container like Tomcat or Jetty is necessary.

I will show several examples of working with Vert.x so you can try out writing your own applications.

Starting a web server is simple:

vertx.createHttpServer()
    .requestHandler(request -> {
        System.out.println("Received request from " + request.remoteAddress().host());
    })
    .listen(80);

Using this approach is fast, but you have to handle which business logic to call for which endpoint and it can get quite messy.
Fortunately, we can do better using Vert.x web module:

Router router = Router.router(vertx);
router.route("/hello").handler(context -> {
    String name = context.request().getParam("name");
    context.response().end("Hello " + name);
});

router.get("/ping").handler(context -> {
    context.response().end("Pong!");
});

vertx.createHttpServer()
     .requestHandler(router::accept)
     .listen(80);

This way, we can easily separate business logic for our endpoints.

Do you need request logging or security? You can add more routes for same endpoints. Routes are then processed in the order of addition to the router and can call next route in the chain or just end the response.

Router router = Router.router(vertx);
router.route().handler(CookieHandler.create());
router.get("/secured-resource").handler(context -> {
    if ("password".equals(context.getCookie("securityToken"))) {
        context.next();
    } else {
        context.response().setStatusCode(401).end("sorry, you are not authorized to see the secured stuff");
    }
});

router.get("/secured-resource").handler(context -> {
    context.response().end("top secret stuff");
});

Do you want more than just APIs? There are template engines for Handlebars, MVEL, Thymeleaf or Jade if you want to use their functionality.

router.get().handler(TemplateHandler.create(JadeTemplateEngine.create()));

Things get more interesting when you configure Vert.x instances to form a cluster. Currently, the only production ready cluster manager is built on top of Hazelcast, so we will use it.

VertxOptions options = new VertxOptions();
options.setClustered(true);
options.setClusterHost("192.168.1.2");

Vertx.clusteredVertx(options, result -> {
    if (result.succeeded()) {
        Vertx vertx = result.result();
        //initialize what we need
    } else {
        LOGGER.error("Failed to start vertx cluster", result.cause());
    }
});

As you can see, starting Vert.x cluster isn’t that complicated. Just be careful to set correct cluster host address or you may find out that the cluster doesn’t behave how you would expect.
And what functionality is available for clustered applications? There is clustered event bus, where you can use point-to-point or publish-subscribe messaging. Just register your consumers and send them messages to logical addresses without ever caring on which node the consumer resides.

vertx.eventBus().consumer("/echo", message -> {
    message.reply(message.body());
});

vertx.eventBus().send("/echo", "ping");

I hope this short introduction to Vert.x will make you curious to look at its documentation and start writing your own applications on top of it.

One of the systems our team develops is UI for end-users, where users can view and manage their print related data.

The system is designed as a simple web application, where we make AJAX calls to Spring controllers which delegate the calls to two other systems, no database is present.

One of the requirements on the system was to support about 1000 concurrent users. Since Tomcat has by default 200 threads and the calls to the other systems may take long (fortunately it’s not the case at the moment), we have decided to make use of Servlet 3.0 async. This way, each AJAX call from the browser uses up Tomcat thread only for preparation of a call to other system. The calls are handled by our asynchronous library for communication with SafeQ and asynchronous http client for communication with Payment System which both have own threadpools and fill in the response when they get a reply.

Since we depend so much on other systems’ performance, we wanted to monitor the execution time of the requests for better tracking and debugging of production problems.

There are several endpoints for each system and more can come later, so in order to avoid duplication, we have decided to leverage Spring aspect programming support. We have created an aspect for each business service (one for SafeQ, one for Payment System) and it was time to implement the measurement business logic.

In synchronous scenario, things are pretty simple. Just intercept the call, note start and end time and if it took too long, just log it in a logfile.

@Around("remoteServiceMethod()")
public Object restTimingAspect(ProceedingJoinPoint joinPoint) throws Throwable {
    long start = System.currentTimeMillis();

    Object result = joinPoint.proceed();

    long end = System.currentTimeMillis();

    long executionInMillis = end - start;
    if (executionInMillis > remoteServiceCallDurationThresholdInMillis) {
        LOGGER.warn("Execution of {} took {} ms.", joinPoint, executionInMillis);
    }

    return result;
}

This won’t work in our asynchronous case. The call to joinPoint.proceed(), which will make call to other system returns immediately without waiting for a reply. Reply is processed in a callback provided to one of async communication libraries. So we have to do a bit more.

We know the signature of our business methods. One of the arguments is always callback, which will process the reply.

public void getEntitlement(ListenableFutureCallback callback, String userGuid, Long costCenterId)

If we want to add our monitoring logic in a transparent way, we have to create special callback implementation, which will wrap the original callback and track the total time execution.

class TimingListenableFutureCallback implements ListenableFutureCallback {

    private ListenableFutureCallback delegate;
    private StopWatch timer = new StopWatch();
    private String joinPoint;

    public TimingListenableFutureCallback(ListenableFutureCallback delegate, String joinPoint) {
        this.delegate = delegate;
        this.joinPoint = joinPoint;
        timer.start();
    }
        
    @Override
    public void onSuccess(Object result) {
        logExecution(timer, joinPoint);
        delegate.onSuccess(result);
    }

    @Override
    public void onFailure(Throwable ex) {
        logExecution(timer, joinPoint);
        delegate.onFailure(ex);
    }
}

And then we have to call the target business method with properly wrapped callback argument.

@Around("remoteServiceMethod() && args(callback,..)")
public Object restTimingAspect(ProceedingJoinPoint joinPoint, ListenableFutureCallback callback) throws Throwable {
    String joinPointName = computeJoinPointName(joinPoint);
    
    Object[] wrappedArgs = Arrays.stream(joinPoint.getArgs()).map(arg -> {
        return arg instanceof ListenableFutureCallback ? wrapCallback((ListenableFutureCallback) arg, joinPointName) : arg;
    }).toArray();
    
    LOGGER.trace("Calling remote service operation {}.", joinPointName);
    
    return joinPoint.proceed(wrappedArgs);
}

There is similar implementation for the other async messaging library.

I hope this solution will help you solve similar problems in your applications in an elegant manner 🙂

Wishing you good luck and great success in year 2016.

Here is small puzzle game for you:

PF 2016 Puzzle Game

http://ysofters.com/pf2016

Source code is available at GitHub.

You can play the game on tablet, phone or computer.

Based on: Enigma game, Kiwi.JS, Font Awesome.

Limitation for users with Windows 10 with touch displays: Mouse does not work, you can use only touch. Bug was already reported to authors of Kiwi.JS framework.

You can enjoy also PF games from previous years: PF 2015, PF 2014, PF 2013,PF 2012,PF 2011, PF 2010

For those, who has attended Robot Framework workshop on Test Crunch conference you can find more details about environment setup, source codes and books below.

Installing Robot Framework on your computer

  1. Download and install Python 2.7 (32/64 bit based on your OS) from https://www.python.org/downloads/
  2. Add Python location into Path environment variable (e.g. c:\Python27\;c:\Python27\Scripts\)
  3. Download and install wxPython 2.8 with unicode support (32/64 bit based on Python version) from http://sourceforge.net/projects/wxpython/files/wxPython/2.8.12.1/
  4. Install pip (package manager) by  opening command line and running command python get-pip.py
  5. Open command line and run following commands to install Robot framework and additional libraries
    pip install robotframework
    pip install robotframework-ride
    pip install robotframework-selenium2library
  6. Start RIDE by running ride.py in command line

Workshop source code

Books and other study materials

Recording from workshop will be available soon.

Stay tuned and enjoy Robot Framework.

In QA we use robotic arm to autonomously operate a multifunctional device (MFD) according to a given test that is repetitive, time consuming or not performable by a human.

How does the robot know where is the screen located? How do the 2D screen coordinates transform to the robot’s system?

calib1 clanekRobot moves the end effector (stylus) in the 3D Cartesian coordinate system with the origin [0,0,0] at the center of the first servomotor. Position of the stylus is then transformed to angles of all servomotors by inverse kinematics. It is also possible to calculate position of stylus given the angles of the servomotors (forward kinematics). However the screen of the MFD is a 2D plane in the 3D space with unknown origin, dimensions and rotations. Robot needs to know where the screen is located relatively to its origin in order to correctly tap any button on the screen.

Dimensions of the screen need to be measured by hand in millimeters. We use 2D coordinates with origin at the bottom left corner to define position on the screen. In 3D space position and rotation of any plane is uniquely determined by three non-collinear points. If these three points are known, transformation matrix can be found. This matrix multiplied by position on the plane is equal to corresponding position in 3D space.
Previously we used ‘basic’ calibration where the robot is navigated through the bottom left corner [0, 0] (origin of the screen), top left corner [0, height] and top right corner [width, height]. At each corner, stylus’ position (in 3D space) is saved and transformation matrix is calculated. This method of calibration requires a lot of precision because even a slight deviation from the corner leads to a similar deviation in every robot’s tap so robot might not accurately tap the desired button. There is no feedback from the MFD, but sometimes there is no other way to perform the calibration.

calib2 clanek

With the new semi-automatic calibration we created our custom version of Terminal server (component which handles communication with MFD) that detects any tap on the MFD screen and sends its coordinates (X,Y in pixels) into the Robot application. Screen resolution is also required so Terminal Server sends that on demand. With the knowledge of screen dimensions and screen resolution robot is able to calculate position of a tap (in pixels) to position on the screen (in millimeters) and save the end effector’s position. The semi-automatic calibration procedure is almost the same as the basic one but robot can be navigated to any point within a marked rectangle, not just the specific point at the corner. This nullifies the need for precision. However, in this case a problem occurred in form of inaccurate values of Z axis. For this purpose we have developed an automatic recalibration. This recalibration takes data gained from semi-automatic calibration and automatically repeats the procedure of semi-automatic calibration with the knowledge of existing corners of the screen. It goes through those three corners same as before, however it starts higher above each point and slowly descends to accurately measure the Z coordinate. After recalibration all data from semi-automatic are forgotten and replaced with the values from automatic calibration. This procedure eliminates any error made by an engineer during calibration and makes the robot´s calibration nearly perfect.

As many of you may know, Android supports native printing since Android 4.4. This means, that there is new api handling communication between application from which user prints and application that later sends the job to the printer.

android print

So how it works? First lets have a look at applications from which user prints.

Main responsibility of these applications is to prepare output for print in pdf format. This includes for example paging or updating image to landscape or portrait mode.

Application from which user prints uses then system PrintManager service.

PrintManager printManager = (PrintManager) getSystemService(Context.PRINT_SERVICE);

Document output is prepared with PrintDocumentAdapter which is passed as second parameter of PrintManagers print function.

printManager.print(jobName, new PrintDocumentAdapter(...), printAttributes);

Now we are heading to the second part of job printing, where we have to discover printers and send them our job. This is responsibility of PrintService.

Printer discovery

We can either add printers manually. Set their ip address and port, or we can look for network printers in the local network.

Let’s have a look on how to find printers which support Zeroconf discovery in local network. Implementations of Zeroconf are for example Avahi daemon or Bonjour.

When printer discovery in Android is started, onCreatePrinterDiscoverySession() method in PrintService is called. Here we have to create our PrinterDiscoverySession.

Responsibilities of PrinterDiscoverySession are pretty straightforward.

  • find and add discovered printers
  • remove previously added printers that disappeared
  • update already added printers

In this example we will use NsdManager. NsdManager is native Android solution for finding zeroconf services. On the other hand its functionality is very limited, but for purpose of this demo it’s satisfactory. There exist other and better solutions, for example JmDNS. Current limitation of NsdManager is not being able to load txt records of mDNS messages.

In order to use NsdManager we have to implement two interfaces. DiscoveryLisener (handles discovery callbacks) and ResolveListener (handles resolving callbacks). I will call them OurDiscoveryListener and OurResolveListener.

First in onStartPrinterDiscovery() method we create new instance of DiscoveryListener and start the actual discovery.

discoveryListener = new OurDiscoveryListener(nsdManager);
nsdManager.discoverServices(SERVICE_TYPE, NsdManager.PROTOCOL_DNS_SD, discoveryListener);

This is pretty self-explanatory. We specify discovery service type, which is either “_ipps_.tcp.” or “_ipp_.tcp”, depending on the fact if we want to encrypt ipp messages or don’t.

And when service is found, then OurDiscoveryListener will handle what happens in individual states of discovery.

In the following code we can see that for example when service is found we try to resolve it with NsdManager.

public void onServiceFound(NsdServiceInfo service) {
    nsdManager.resolveService(service, new OurResolveListener());
}

Resolving service means, that we try to get more information about the service. This includes host ip and port. OurPrinterResolveListener then handles states what should happen when resolving succeeds or fails. When resolving succeeds, we process gained data and save it for future use.

Last part of printer discovery is to find more details about selected printer and checking whether is this printer still available. This is handled in onStartPrinterStateTracking() method.

Discovering details about printer can be done for examaple with ipp operation Get-Printer-Attributes and according to received data, set the printer information. Second function is to keep tracking of the printer state.

The following code sample just shows how to set few printer capabilities, which should be set according to the printer attributes. This doesn’t contain  tracking of printer state.

@Override
public void onStartPrinterStateTracking(PrinterId printerId) {
    // check for info we found when printer was discovered
    PrinterInfo printerInfo = findPrinterInfo(printerId);

    if (printerInfo != null) {
        PrinterCapabilitiesInfo capabilities = new PrinterCapabilitiesInfo.Builder(printerId)
                .setMinMargins(new PrintAttributes.Margins(200, 200, 200, 200))
                .addMediaSize(PrintAttributes.MediaSize.ISO_A4, true)
                .addMediaSize(PrintAttributes.MediaSize.ISO_A5, false)
                .addResolution(new PrintAttributes.Resolution("R1", "200x200", 200, 200), false)
                .addResolution(new PrintAttributes.Resolution("R2", "200x300", 200, 300), true)
                .setColorModes(PrintAttributes.COLOR_MODE_COLOR |
                        PrintAttributes.COLOR_MODE_MONOCHROME, PrintAttributes.COLOR_MODE_COLOR)
                .build();

        printerInfo = new PrinterInfo.Builder(printerInfo)
                .setCapabilities(capabilities).build();

        // We add printer info to system service
        List<PrinterInfo> printers = new ArrayList();
        printers.add(printerInfo);
        addPrinters(printers);
    }
}

When different printer is selected, then onStopPrinterStateTracking is called and onStartPrinterStateTracking again.

 

Printing:

Android itself doesn’t contain implementation of any printing protocol. Because of this I created small IPP parser. But that’s topic for another day.

Here I will only show example of handling queued print job.

In the following code we pick one job according to id from saved processed jobs and set his state to printing. Class PrintTask in the following example is just Android AsyncTask which in background creates IPP request and appends job data.

public void handleQueuedPrintJob(PrintJobId printJobId, PrinterId printerId) {
    final PrintJob printJob = mProcessedPrintJobs.get(printJobId);
    if (printJob == null) {
        return;
    }

    if (printJob.isQueued()) {
        printJob.start();
    }

    OurPrinter ourPrinter = ourDiscoverySession.getPrinter(printerId);

    if (ourPrinter != null) {
        AsyncTask <Void, Void, Void> printTask =
                new PrintTask(printJob, ourPrinter);
        printTask.execute();
    }
}

In case that we have decided to use ipps, we also have to set correct certificate. Next step is to create new HttpURLConnection. (or HttpsURLConnection for secure transfer).

The last thing we have to do is to write into the output stream our IPP message, send it and wait for response from server.

Android manifest file

We have to set necessary permissions in the Android manifest file, in order to be able to run the PrintService.
Add android.permission.BIND_PRINT_SERVICE when creating the service. Example:

...
<service android:name=".OurPrintService" 
    android:permission="android.permission.BIND_PRINT_SERVICE">

    <intent-filter>
        <action android:name="android.printservice.PrintService" />
    </intent-filter>
</service>
...

This allows system to bind to the PrintService. Otherwise the service wouldn’t be shown in the system.

Also

<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.START_PRINT_SERVICE_CONFIG_ACTIVITY" />

permissions are needed for printer discovery and to access files in external storage.