At Y Soft, we use robots to test our solutions for verification and validation aspects, we are interested if the system works according to required specifications and what are the qualities of the system. To save time and money, it is possible to use a single robot to test multiple devices simultaneously. How is this done? It is very simple so let’s look at it.

When performing actions to operate given a device, the robot knows where the device is located due to a calibration. The calibration file contains transformation matrix that can transform location on the device to robot’s coordinate system. The file also contains information about the device that is compatible with the calibration. How the calibration is computed is covered in this article. There is also calibration of the camera that contains information about the region of interest, meaning where exactly is the device screen located in the view of a camera. All of the calibration files are stored on the hard drive.

Example of calibration files for two Terminal Professionals:

{  
   "ScreenXAngle":0.35877067027057225,
   "DeviceId":18,
   "DeviceModelId":18,
   "DeviceName":"Terminal_Professional 4, 10.0.5.182",
   "MatrixArray":[[0.728756457140,0.651809992529,-0.159500429741,75.4354297376],[-0.683749126568,0.734998176419,-0.10936964140,71.1249458777],[0.0422532822652,0.187897834122,0.981120733880,-34.923427696],[0.0,0.0,0.0,1.0]] ,
   "ScreenSize":{  
      "Width":153.0,
      "Height":91.0
   }
}
{  
   "ScreenXAngle":0.25580856461537194,
   "DeviceId":27,
   "DeviceModelId":18,
   "DeviceName":"Terminal_Professional 4, 10.0.5.112",
   "MatrixArray":[ [0.713158843830,-0.686471581194,0.220191724515,-176.983055],[0.699596463347,0.6783511825194,-0.15148794414,-71.7788394],[-0.05297850752,0.2635031531279,0.963621817536,-29.83848504],[0.0,0.0,0.0,1.0] ],
   "ScreenSize":{  
      "Width":153.0,
      "Height":91.0
   }
}


For the robot to operate on multiple devices, all of the devices must be within the robot’s operational range, which is quite limited, so this feature is currently is only used for smaller devices, like mobile phones and Terminal Professional. It is theoretically possible to use a single robot on more devices, but for practical purposes, there are usually only 2 devices. Also, all devices must be at relatively the same height, which limits testing on multifunctional devices that have various height and terminal placement. Space is also limited by the camera’s range, so multiple cameras might be required, but this is not a problem as camera calibration also contains the unique identifier of a camera. Therefore a robot can operate on multiple devices using multiple cameras or just a single camera if devices are very close to each other.

Before testing begins, a robot needs to have all calibrations of devices available on the hard drive and all action elements (buttons) need to be within its operational range. Test configuration contains variables such as DEVICE_ID and DEVICE2_ID which need to contain correct device IDs as stored the robot’s database. Which tests will run on the devices and the duration of the tests also need to be specified. Tests used for these devices are usually measurement and endurance tests, which run in iterations. There are multiple variants for these tests, for example, let’s say we wish to run tests for 24 hours on two devices and each device should have an equal fraction of this time. This means that the test will run for 12 hours on one device and 12 hours on the other, which is called consecutive testing. Another variant is simultaneous testing, which means that the robot will alternate between the devices after each iteration for a total time of 24 hours. The robot loads the calibration for another device after each iteration and continues with the test on that device.  This is sometimes very useful should one device become unresponsive, the test can continue on the second device for the remaining time. Results of each iteration of the test for each device are stored in a database along with other information about the test and can be viewed later.

Testing multiple devices with a single robot also makes it possible to test and compare different versions of an application or operating system (in this case on Terminal Professional) without ending the test, reinstalling of the device and running the test again. This saves a lot of time and makes the comparison more accurate.

Members of our Robot team have participated in the worldwide competition of robots on the 11 April to 12 April in 2015, which was held in Vienna, Austria. There were a lot of competition categories including Humanoid sumo, where we participated. Over 600 robots were registered over all categories and 16 robots in the Humanoid sumo.

Robot specifications

Each humanoid robot has to meet certain specifications, e.g. maximum dimensions and weight. The rules also require having a head, two legs, two arms and a name. We named our robot YSoft Ragnarök, which is a great foretold battle from Norse mythology. The limit for the weight is 3000 g, which was quite problematic for us. At the beginning of the competition we had to reduce robot’s weight to 2997 g by removing some insignificant parts. Our strategy is a great stability which many other robots lack, but it comes with demand for heavy parts especially at the bottom of the robot. Heavy body also reduces mobility and speed so we had to develop better solution in order to find the opposing robot reliably, then move directly to it and wreck it. For this purpose we used ultrasonic sensors with maximum range of 2 meters, high precision and low power consumption.

robot clanekArena rules

Tournament started with qualifications and continued with single match elimination. Each match startswith two robots in opposing corners facing each other. The main goal is to push the other robot out of the arena or to knock it down. If any robot is pushed out of the arena, it can be placed within the arena again, however it must be placed face down. If the robot can autonomously stand up, the match continues. Team gains 3 points for pushing the other robot out of the arena. If any robot falls in the arena, the opposing team gains 1 point. Two points are awarded for a robot that knocks the opponent to the ground. Match ends if any robot is knocked out (and cannot stand up), it does not move for a period of time or the time for the match runs out (the maximum time is 3 minutes). The competitor with highest score wins.

Our performance

We have beaten every robot in the qualification group and successfully advanced into the semifinals without suffering a loss. However, this had already happened in the past when we lost the next two matches to finish in the 4th place. This time we tried not to repeat such outcome. The first semifinal match against Mexican robot Speedy Gonzales was quite even as the opponent avoided contact with us so we only managed to knock it down once. The second match versus another Mexican robot Atom was more one-sided because it could not get up after knockout (this match can be viewed here).

Once we got into the finals, we have faced our old rival from Poland, robot DUE. At the end we have beaten them and won the first place (video). Polish robots DUE and UNO took both 2nd and 3rd place.

 

In QA we use robotic arm to autonomously operate a multifunctional device (MFD) according to a given test that is repetitive, time consuming or not performable by a human.

How does the robot know where is the screen located? How do the 2D screen coordinates transform to the robot’s system?

calib1 clanekRobot moves the end effector (stylus) in the 3D Cartesian coordinate system with the origin [0,0,0] at the center of the first servomotor. Position of the stylus is then transformed to angles of all servomotors by inverse kinematics. It is also possible to calculate position of stylus given the angles of the servomotors (forward kinematics). However the screen of the MFD is a 2D plane in the 3D space with unknown origin, dimensions and rotations. Robot needs to know where the screen is located relatively to its origin in order to correctly tap any button on the screen.

Dimensions of the screen need to be measured by hand in millimeters. We use 2D coordinates with origin at the bottom left corner to define position on the screen. In 3D space position and rotation of any plane is uniquely determined by three non-collinear points. If these three points are known, transformation matrix can be found. This matrix multiplied by position on the plane is equal to corresponding position in 3D space.
Previously we used ‘basic’ calibration where the robot is navigated through the bottom left corner [0, 0] (origin of the screen), top left corner [0, height] and top right corner [width, height]. At each corner, stylus’ position (in 3D space) is saved and transformation matrix is calculated. This method of calibration requires a lot of precision because even a slight deviation from the corner leads to a similar deviation in every robot’s tap so robot might not accurately tap the desired button. There is no feedback from the MFD, but sometimes there is no other way to perform the calibration.

calib2 clanek

With the new semi-automatic calibration we created our custom version of Terminal server (component which handles communication with MFD) that detects any tap on the MFD screen and sends its coordinates (X,Y in pixels) into the Robot application. Screen resolution is also required so Terminal Server sends that on demand. With the knowledge of screen dimensions and screen resolution robot is able to calculate position of a tap (in pixels) to position on the screen (in millimeters) and save the end effector’s position. The semi-automatic calibration procedure is almost the same as the basic one but robot can be navigated to any point within a marked rectangle, not just the specific point at the corner. This nullifies the need for precision. However, in this case a problem occurred in form of inaccurate values of Z axis. For this purpose we have developed an automatic recalibration. This recalibration takes data gained from semi-automatic calibration and automatically repeats the procedure of semi-automatic calibration with the knowledge of existing corners of the screen. It goes through those three corners same as before, however it starts higher above each point and slowly descends to accurately measure the Z coordinate. After recalibration all data from semi-automatic are forgotten and replaced with the values from automatic calibration. This procedure eliminates any error made by an engineer during calibration and makes the robot´s calibration nearly perfect.