In latest project we came across the problem of securing communication, where peers don’t share backend language. Our goal was to securely generate shared secret between Java server and .NET client. Each language supports all features needed for the key agreement, but these features aren’t always compatible, due to different encoding or data representation.

This article should be quick and easy solution how to agree on shared secret between such server and client with hand-on examples. Each subchapter contains concrete example on how to send, receive and process data needed for respected operation. Authentication and signature scheme are similar to guides published internet-wide, the main topic- key agreement is presented with our insights and comments on how to make it work.

The reason I have included all the code snippets even if they are “straight from internet” is that I wanted to simply group all these schemes to one place, so that you don’t have to find examples on Oracle, CodeRanch, StrackOverflow, MSDN Microsoft documentation, … Honestly it was a little bit frustrating for me, so there you go 🙂

If you are interested just in key agreement, feel free to skip right to the key agreement example, as I would like to discuss authentication and signature schemes first in order to prevent attacks on anonymous key agreement.

Please beware of incautious use of the published code. Your protocol might require different configuration and by simple copy-pasting you may introduce some security issues or errors to it.

Certificate based authentication

Consider each peer to have certificate signed by some root CA.

In my examples I’ll be working mostly with byte arrays as it is most universal way to represent data.

In most cases whole certificate chain is transferred to other side. Using one certificate (self-signed) is just simplification of process below.

.NET authentication

Java peer can transform certificate into byte array using Certificate.getEncoded() method. Byte array is then sent to counterpart.

.NET peer parses certificate chain data and stores parsed certificates into List<byte[]> to ease processing.

Code below shows the authentication process. Process differs from standard validation in a way, that we are sending only certificate chain. From the chain peers certificate is extracted. From the rest of received certificate chain we create subchain and check if we were able to create complete chain from peer to Certificate Authority.

For simplicity revocation checks and certificate usage field (OID of the field – 2.5.29.15) are being ignored. In real environment you should definitely check this list and field, but in this article it is not really necessary.

/**
* receivedCertificateChain previously transformed byte array containing whole certificate chain. This chain is transformed
*                          into list of separate certificates from chain 
*/
public bool certificateChainValidation(List<byte[]> receivedCertificateChain)
{
  var otherSideCertificate = new X509Certificate2(receivedCertificateChain[0]);
  var chain = new X509Chain { ChainPolicy = { RevocationMode = X509RevocationMode.NoCheck, VerificationFlags = X509VerificationFlags.IgnoreWrongUsage } };
  chain.ChainPolicy.ExtraStore.AddRange(receivedCertificateChain.Skip(1).Select(c => new X509Certificate2(c)).ToArray());

  return chain.Build(otherSideCertificate));
}

Root CA certificate must be trusted by your system to be successfully validated.

Java authentication

.NET peer can transform each certificate into byte array using X509Certificate2.RawData property and send it to counterpart.

First, Java peer needs to define validation parameters and Trust anchor. In this case it’s root CA.

After that, you validate received certificate chain. If method validate(CertPath, PKIXParameters) is processed without any exception, you can consider received certificate chain correctly validated.

/**
* certificatePath received data of certificate chain from counterpart 
*/
public boolean certificateChainValidation(byte[] certificatePath) {
  try {
    //generate Certificate Path from obtained byte array
    CertificateFactory cf = CertificateFactory.getInstance(“X.509”);
    CertPath received = cf.generateCertPath(new ByteArrayInputStream(certificatePath));

    //get all trusted entities from dedicated truststore
    Set<TrustAnchor> trustedCAs = getAnchors();

    //set the list of trusted entities and turn off the revocation check
    //revocation is by default set to true, so if you don't turn it off explicitly, exceptions will bloom
    PKIXParameters params = new PKIXParameters(trustedCAs);
    params.setRevocationEnabled(false);

    CertPathValidator cpv = CertPathValidator.getInstance("PKIX");
    //if no exception is thrown during validation process, validation is successful
    cpv.validate(received, params);
  } catch (CertificateException | NoSuchAlgorithmException | InvalidAlgorithmParameterException | CertPathValidatorException e ) {
    throw new CertificateException("Could not validate certificate", e);
    return false;
  }
  return true;
}

getAnchors() is a method which simply loads all entries from dedicated truststore and returns them as Set<TrustAnchors>.

Signatures

Without correct signature scheme whole authentication scheme would be compromised, as any attacker would be able to replay intercepted certificate chain. Therefore while sending outgoing certificate chain, each peer has to include signature to it. It is sufficient for such signature to be reasonably short (resp. long) nonce. This nonce should be freshly generated by the communication counterpart and can be sent before authentication attempt (or as an authentication challenge). Nonce could be any-random value or  static counter, but must be definitely unique.

During the key agreement phase, signatures are used to sign public keys that are exchanged between peers to ensure the integrity of data.

.NET signature generation

First you need to use certificate with included private key – PKCS#12 certificates.

Flags as PersistKeySet and Exportable have to be included to certificate loading, as without them you are not able to obtain private key for signatures from the keystore. PersistKeySet flag ensures that while importing certificate from .pfx or .p12 file, you import private key with it. Exportable flag means that you can export imported keys (mainly private key).

As mentioned in this comment we need a little workaround to force crypto provider to use SHA256 hashes.

public byte[] signData(byte[] dataToSign)
{
  X509Certificate2 cert = new X509Certificate2(PATH_TO_CERTIFICATE, PASSWORD, X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);
  RSACryptoServiceProvider privateKey = (RSACryptoServiceProvider)cert.PrivateKey;
 
  RSACryptoServiceProvider rsaClear = new RSACryptoServiceProvider();
  // Export RSA parameters from 'privateKey' and import them into 'rsaClear'
  // Workaround to force RSACryptoServiceProvider use SHA256 hash
  rsaClear.ImportParameters(privateKey.ExportParameters(true));
  return rsaClear.SignData(dataToSign, "SHA256");
}

Java signature generation

Straight-forward example presented internet-wide.

/**
* data       data to be signed
* privateKey private key imported from key pair storage
*/
public byte[] signData(byte[] data, PrivateKey privateKey) {
  try {
    Signature signature = Signature.getInstance(signatureType);
    signature.initSign(privateKey);
    signature.update(data);
    return signature.sign();
  } catch (NoSuchAlgorithmException | SignatureException | InvalidKeyException e) {
    throw new SignatureException(“Could not create signature”, e);
    return null;
  }
}

.NET signature verification

First thing to mention is, that you shouldn’t create X509Certificate2 object directly from byte arrays (5th tip in this article). Depending on use of your protocol it might result into stalling and slowing whole protocol, what you definitely don’t want. But again, for simplicity I’m creating object from byte array directly. Mitigation of mentioned problem can be seen in linked article.

/**
* signature    signature of received data
* receivedData data received from counterpart. These data were also signed by counterpart to ensure integrity
*/
public bool verifySignature(byte[] signature, byte[] receivedData)
{
  X509Certificate2 cert = new X509Certificate2(PATH_TO_CERTIFICATE);

  RSACryptoServiceProvider publicKey = (RSACryptoServiceProvider)cert.PublicKey.Key;
  return publicKey.VerifyData(receivedData, "SHA256", signature);
}

Java signature verification

Straight-forward example presented internet-wide.

/**
* receivedData data received from counterpart
* signature    signature of receivedData
* publicKey    public key associated with privateKey object from "Java signature generation"
*/
public boolean verifyData(byte[] receivedData, byte[] signature, PublicKey publicKey) {
  try {
    Signature signature = Signature.getInstance(“SHA256withRSA”);
    signature.initVerify(publicKey);
    signature.update(receivedData);
    return signature.verify(signature);
  } catch (NoSuchAlgorithmException | SignatureException | InvalidKeyException e) {
    throw new SignatureException(“Could not validate signature”, e);
    return false;
  }
}

 

Key pair generation

Both, key pair generation and key agreement use BouncyCastle library (BC) for the crypto operations. During key generation phase, you need to specify key agreement algorithm. In this project we used elliptic curve Diffie-Hellmann (“ECDH”) key agreement. Elliptic curves were used to generate key pair mainly due to their advantages (performance, key size).

.NET key pair generation

The only difference from standard key generation is encoding of generated public key in a way that Java will accept the key and is able to create PublicKey object.

private static AsymmetricCipherKeyPair generateKeypair()
{
  var keyPairGenerator = GeneratorUtilities.GetKeyPairGenerator("ECDH");
  var ellipticCurve = SecNamedCurves.GetByName(ELLIPTIC_CURVE_NAME);
  var parameters = new ECDomainParameters(ellipticCurve.Curve, ellipticCurve.G, ellipticCurve.N, ellipticCurve.H, ellipticCurve.GetSeed());
  keyPairGenerator.Init(new ECKeyGenerationParameters(parameters, new SecureRandom()));
  return keyPairGenerator.GenerateKeyPair();
}

Now you need to send public key to other peer. Unfortunately, it is necessary to format public key in following way, otherwise Java peer will not be able to correctly create PublicKey object and create shared secret.

AsymmetricCipherKeyPair keyPair = generateKeypair();
byte[] publicKeyData = SubjectPublicKeyInfoFactory.CreateSubjectPublicKeyInfo(keyPair.Public).GetDerEncoded();

Java key pair generation

public static KeyPair generateKeyPair() {
  try {
    Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
 
    ECNamedCurveParameterSpec ecNamedCurveParameterSpec = ECNamedCurveTable.getParameterSpec(ELLIPTIC_CURVE_NAME);
    KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("ECDH", PROVIDER);
    keyPairGenerator.initialize(ecNamedCurveParameterSpec);
    return keyPairGenerator.generateKeyPair();
  } catch (NoSuchAlgorithmException | InvalidAlgorithmParameterException | NoSuchProviderException e) {
    throw new EncryptionException("Could not generate key pair", e);
  }
}

 

Key agreement

.NET key agreement

At this step, you need to beware the issue in C# BoucyCastle library, which results into failed key agreement even if you have correct keys (correct means same data, but different indentation => 0x0059534F4654 is different from 0x59534F4654).

Problem is that BC library creates shared secret as BigInteger. BigIntegers trims all leading zeroes and at conversion from it to byte array these zeroes aren’t included into array.

Unfortunately until publication day issue wasn’t still fixed in BC and you need to check key lengths by yourselves.


private static byte[] DeriveKey(AsymmetricCipherKeyPair myKeyPair, byte[] otherPartyPublicKey)
{
  IBasicAgreement keyAgreement = AgreementUtilities.GetBasicAgreement(KEY_AGREEMENT_ALGORITHM);
  keyAgreement.Init(myKeyPair.Private);

  //check otherPartyPublicKey length

  //shared secret generation
  var fullKey = keyAgreement.CalculateAgreement(PublicKeyFactory.CreateKey(otherPartyPublicKey)).ToByteArrayUnsigned();  
  return fullKey;
}

Java key agreement

Straight-forward example presented internet-wide.

public SecretKey secretDerivation(PublicKey receivedPublicKey, PrivateKey myPrivateKey) {
  try {
    KeyAgreement keyAgreement = KeyAgreement.getInstance(KEY_AGREEMENT_ALGORITHM, PROVIDER);
    keyAgreement.init(privateKey);
    keyAgreement.doPhase(publicKey, true);
    return keyAgreement.generateSecret(“AES”);
  } catch (NoSuchAlgorithmException | InvalidKeyException | NoSuchProviderException e) {
    throw new EncryptionException("Could not generate secret", e);
  }
}

 

From these examples You might gain bad feeling as all examples are rather brute and messages aren’t sent/received in no standard way- only as byte arrays. Best way is to exchange protocol messages in more standardized way. By doing so, you start heavily relying on third party library.

More standardized version of protocol should use Cryptographic Message Syntax. We will have a closer look on this option in next blog article.

While choosing the right security layer for messaging via ZeroMQ, there were two main considerations: built-in security CurveZMQ or usage of more conventional TLS backend?

First option is ZeroMQ’s proprietary security protocol based on elliptic curve cryptography and Daniel’s Bernstein NaCl cryptographic library. CurveZMQ utilizes not only Bernstein’s Curve25519 elliptic curve, but also other ciphers, designed by him, as well- f.e. Salsa20 and poly1305. Other NaCl’s ciphers can be found on its web page under Public-key and Secret-key cryptography chapters.

From the beginning this option was really hot candidate to use, as according to papers about performance of Salsa20 and performance of Curve25519, NaCl crypto functions are faster than standard crypto functions. Also the fact that it was already included in ZeroMQ was big plus. Unfortunately NaCl library and its functions were created in years 2008-2010 and aren’t used widely as f.e. AES or SHA are, therefore aren’t matured enough as other libraries, what brings a little bit panic to our mind and ultimately was the cause of not using it.

Second option was the usage of TLS backend- TLSZMQ. This is a demo project of Ian Barber to secure ZeroMQ communication using OpenSSL library. Unfortunately TLS isn’t built into ZeroMQ, so for now this is the only option how to use TLS and ZeroMQ together 🙁

Implementations comparison

To find the best approach we had few requirements to check. Mainly it was the speed of messaging and library and crypto functions maturity, but other requirements such as support and liveliness of implementation, licensing, quality of code also weighted in decision.

From this set of metrics only 2 appeared to be ultimately decisive:

  • Library and crypto functions maturity
  • Speed of messaging

While the maturity of NaCl library was strong argument why not to use CurveZMQ, performance of NaCl mentioned before spoke strongly in favor of it.

To finally resolve which approach to use and determine the total overhead of each implementation I have created performance tests to find out if TLSZMQ could be used in our environment. The test suite for each implementation was designed to be a block of 20 separate tests running for 10 minutes each, to be sure of reliable test outcomes. Even the message sizes were chosen to reflect the size of messages to be really sent in environment.

total persec

At this point, results clearly spoke strongly in favor of CurveZMQ. On the other hand, there was a conflict with maturity of crypto functions used in CurveZMQ and use of single elliptic curve to generate keys. This could cause many problems, if the curve or other functions would be compromised in the future.

At this point I realized that tests were done with OpenSSL’s by default chosen cipher suite RSA-AES256-SHA384 (OpenSSL version 1.0.1p). There is nothing wrong with this cipher suite, but we have to admit that these algorithms were unnecessarily secure (understand with high overhead).

Yes, I know what I just said and that it sounds really bad, but we must also consider the environment of messaging (its encryption) and performance of the system as well. Only after thorough examination we should accommodate ciphers to real requirements and not just use cipher suites with unnecessarily high security (and overhead) to overkill it.

Therefore, tests were rerun with more appropriate (weaker but still strong) cipher suites. The requirement was to get as much close to CurveZMQ performance as possible:

  • elliptic curves used everywhere possible
    • key exchange [ECDH]
    • authentication [ECDSA]

Any EC could be used, not just one.

  • AES doesn’t have to use 256b key as 128b long keys are still considered pretty strong.

From few acceptable cipher suites, this one specific peaked out: ECDSA-AESGCM128-SHA256.

This new finding has shown that new cipher suite increased the messaging performance considerably (sometimes effectively doubling the speed) [see graphs].

From the results one can see, that TLSZMQ isn’t that much inferior to CurveZMQ and due to TLS’s maturity it is a better choice. The final argument is that test results of TLSZMQ were worse, but sufficient in YSoft SafeQ environment, so the choice of implementation was pretty straight-forward.

Truth is, that NaCl and Curve25519 are really fun (and much easier) to play with and I encourage you to at least have a look at them, but in the end TLS backend is more flexible in terms of crypto primitives (ciphers, key generation) and doesn’t bring that much security concerns. And that’s what we are looking for in secured ZeroMQ messaging.

As I have written some time ago, we use OWASP Dependency check in order to scan our product for known vulnerabilities. But when you have reports from many components of the product, you need to manage them somehow. In the previous article, I’ve briefly mentioned our tool that processes the reports. In this article, I’ll present this tool. FWIW, we have open-sourced it. You can use it, you can fork it, you can send us pull requests for it.

Vulnerable libraries view

The vulnerable libraries view summarizes vulnerabilities grouped by libraries. They are sorted by priority, which is computed as number of affected projects multiplied by highest-severity vulnerability. This is probably the most important view. If you choose a vulnerable library, you can check all the details about vulnerabilities, including affected projects. When you decide to update a library to a newer version, you will probably wonder if there is some known vulnerability present in the new library version. The ODC Analyzer allows you to check it after simply entering your desired library version.

One might find the ordering to be controversial, mostly using the highest-rated vulnerability. Maybe the scoring system is not perfect (and no automatic scoring system can be perfect), but I find it reasonable. I assume that highest-scored vulnerability is likely some remote code execution triggerable by remote unauthenticated attacker. In such case, does having multiple such vulnerabilities make the situation worse? Well, it might slightly increase the probability that the vulnerability will be exposed to the attacker, but having two such vulnerabilities can hardly make it twice as bad as having just one. So I wanted one highest-rated vulnerability to be a ceiling of risk introduced by the vulnerable library to the project. Maybe we could improve the algorithm to rate multiple low-severity vulnerabilities higher than the severity of the highest-rated vulnerability. I already have an idea how to do this, but it has to be discussed.

Vulnerabilities view

You can also list all vulnerabilities affecting one of the projects scanned. They are sorted by severity multiplied by number of affected projects. Details available there are mostly the same as in the vulnerable libraries view.

Another interesting feature is vulnerability suppression. Currently, ODC Analyzer can generate a suppression that can be pasted to suppressions.xml, so it is taken into account the next time when running vulnerability scans. We consider making it more smart and moving the suppression to ODC Analyzer itself.

Filters

Unless you have just a single team, most people are probably not interested in all the vulnerabilities of all the projects. They will want to focus on only their projects instead. ODC Analyzer allows focusing on them by filtering only one project or even a subproject. One can also define a team and filter all projects belonging to this team.

Teams are currently defined in configuration file, there is no web interface for it now.

E-mail notifications

One can subscribe to new vulnerabilities of a project by e-mail. This allows one to watch all relevant vulnerabilities without periodic polling of the page.

Export to issue tracking system

We are preparing export to issue tracking system. There is some preliminary implementation, but we might still perform some redesigns. We can export one issue per vulnerability. The rationale behind this grouping is that this allows efficient discussion when multiple projects are affected by the same vulnerability. A side effect of such design is that you need an extra project in your issue tracker and you will might want to create child issues for particular projects.

Currently, only JIRA export is implemented. However, if you want to export it to <insert name of your favourite issue tracker here>, you can simply implement one interface, add a few of code for configuration parsing and send us a pull request 🙂

Some features under consideration

Affected releases

We perform scans on releases. It would be great to add affected releases field to vulnerabilities. We, however, have to be careful to explain its meaning. We are not going to perform these scans on outdated unsupported releases, thus they will not appear there. So, it must be clear that affected releases are not exhaustive in this way.

Discussion on vulnerabilities

We consider adding a discussion to vulnerabilities. One might want to discuss impact on a project. These discussions should be shared across projects, because it allows knowledge sharing across teams. However, this will probably be made in issue tracker systems like JIRA, as we don’t want to duplicate their functionality. We want to integrate them, though.

Better branches support

If a project has two branches, it can be currently added as two separate projects only. It might be useful to take into account that multiple branches of a software belong to the same project. There are some potential advantages:

  • Issues in production might have higher urgency than issues in development.
  • Watching a particular project would imply watching all the branches.

However, it is currently not sure how to implement it and we don’t want to start implementation of this feature without a proper design. For example:

  • Some projects might utilize build branches in Bamboo.
  • Some projects can’t utilize build branches, because there are some significant differences across branches. For example, some projects switch from Maven to Gradle.
  • Is it useful to allow per-branch configuration (e.g., two branches belonging to different teams or watching only one branch)?
  • Should the branches be handled somewhat automatically?
  • How to handle different branching models (e.g. master + feature branches, master + devel + feature branches, …)?

Library tagging

Library tagging is useful for knowing what types of libraries are in the system. It is partially implemented. It works, but it has to be controlled using direct access to the database. There was never a GUI for adding a tag. When you have some existing tags, there is a GUI for adding these tags to a library, but there is no way to add one permission for that.

Homepage

The project was originally designed to be rather a single-page. This did not scale, so we added some additional views. The current homepage is rather a historical left-over and maybe it should be completely redesigned.

List of all libraries

Non-vulnerable libraries are not so interesting, but one might still want to list them for some purposes. While there is a hidden page containing all the libraries (including a hidden CSV output), it is not integrated to the rest of the application. We have needed this in the past, but we are not sure how to meaningfully integrate it with the rest of the system without creating too much of clutter.

Conclusion

We have implemented a tool useful for handling libraries with known vulnerabilities and cooperation across teams. This tool is in active development. If you find it useful, you can try it. If you miss a feature, you can contribute by your code. If you are unsure if your contribution is welcome, you can discuss it first.

While caring about security of our code is arguably important, it is not enough for building a secure product. Vulnerabilities might also arise from a 3rd party component. Handling of those vulnerable libraries is thus another essential aspect of building a secure product.

Database of vulnerabilities

A simple way for managing a large number of 3rd party libraries might be using a vulnerability database. There is well-known National Vulnerability Database that contains information about many vulnerabilities. This is also what OWASP Dependency Check uses. Let me introduce it.

NVD aims to contain all known vulnerabilities of publicly available software. An entry about a vulnerability has some CVE number. (CVE means “Common Vulnerabilities and Exposures”, so CVEs are not limited to vulnerabilities, but it is not so important for now.) You can look at some example of CVE. Various details might be available, but the level of details may depend on multiple factors. For example, a not-yet-publicly-disclosed vulnerability might have a CVE number, but its description will be obviously not very verbose.

CVE numbes are usually assigned to CPEs (“Common Platform Enumeration”). A CPE is an identifier of vulnerable software. For example, the mentioned CVE-2005-1234 contains information that it affects cpe:/a:phpbb_group:phpbb-auction:1.0m and cpe:/a:phpbb_group:phpbb-auction:1.2m.

How OWASP Dependency Check works?

In a nutshell, it scans a project (JARs, POM files, ZIP files, native libraries, .NET assemblies, …) and tries to assign a CPE to each entity. Note that there is much of heuristics involved. Obviously, there is also much what can go wrong.

When CPEs are assigned, it looks for vulnerabilities (CVEs) assigned to those CPEs. A scan is written to a result file. Multiple formats are supported. The two most prominent formats are HTML (for direct reading) and XML (for further processing). We use the XML output in order to process multiple reports of multiple projects and assign them to particular teams. (One team is usually responsible for multiple projects.)

Integration with projects

Maven

OWASP Dependency Check has a mature Maven plugin. It basically works out-of-box and you can adjust probably any parameter supported by OWASP Dependency Check. We didn’t have to modify the project, as it can be run by mvn org.owasp:dependency-check-maven:check and the configuration can be adjusted by passing -Dproperty=value parameters.

Gradle

There is also some plugin for Gradle. Unfortunately, it is not as mature as the Maven plugin. We had to modify it in order to make it working in our environment. I’d like to merge those changes with the original project. Even after the modifications, it is still not ideal. The most prominent issue is that it includes test dependencies. Excluding them in the Groovy plugin is not as simple as with Maven plugin, because Gradle can have many configurations and each of those configurations might have different dependencies. I have no simple clue how to distinguish important configurations from others. However, this was not a really painful issue, as these dependencies are considerably less common and usually don’t have any known vulnerability in NVD, as they usually aren’t touched by untrusted input.

How we scan our projects?

Running those scans manually on multiple sub projects and evaluating all the reports manually would take too much time. So, we have developed a tool for automating some of the manual work.

First, scans are run automatically every day. There is no need to run it manually on every single subproject. Results of these scans are stored in XML format. Some adaptations specific to our environment are applied. For example, scans are able to run without any connection to the Internet. (Maven plugin can be configured without any modification, while Gradle plugin required to be modified for that.) Of course, we have to download all the vulnerability database separately, outside of the scan process.

Second, our tool downloads the results from our Bamboo build server and processes them. There are some sanity checks that should warn us if something breaks. For example, we check freshness of the vulnerability database.

Third, issues related to particular subprojects are automatically assigned to corresponding teams. Issues can be prioritized by severity and number of occurrences.

While ODC is a great tool, there are some issues connected to using it. I’ll discuss some of them there.

Complexity of library identifiers

How many ways of identifying a library do you know? Maybe you will suggest using groupId and artifactId and version (i.e. GAV) by which you can locate the library on a Maven repository. (Or you can pick some similar identifier used on another platform than Java.) This is where we usually start. However, there are multiple other identifiers:

  • GAV identifier: as mentioned above.
  • file hash: ODC uses SHA1 hash for lookups in Maven database. Note that there might be multiple GAV identifiers for one SHA1 hash, so there is 1:1 relation. Moreover, when we consider snapshot, we theoretically get m:n relation.
  • CPE identifier: This one is very important for ODC, as ODC can’t match vulnerabilities without that. Unfortunately, there is no exact algorithm for computation of CPE from GAV or vice versa. Moreover, multiple GAVs might be assigned to one CPE. For example, Apache Tomcat consists of many libraries, but all of them have just one CPE per version. Unfortunately, the ODC heuristic matching algorithm might also assign multiple CPEs to one GAV in some cases.
  • GA identifier: This is just some simplification of GAV identifier, which misses the version number. There is nothing much special about this identifier for ODC, but we have to work with that, too.
  • intuitive sense: For example, when you mention “Hibernate”, you probably mean multiple GAs at the same time.

Note that this complexity is not introduced by ODC. The complexity is introduced by the ecosystem that ODC uses (e.g. CPEs) and by the ecosystem ODC supports (e.g. Maven artifacts).

Bundled libraries

While überJARs might be useful in some cases, they are making inspection of all the transitive dependencies harder. While ODC can detect some bundled libraries (e.g. by package names or by POM manifests if included), this is somehow suboptimal. Moreover, if ODC finds a bundled dependency, it might look like a false positive at first sight.

Libraries without a CPE

Some libraries don’t have a CPE. So, when ODC can’t assign any CPE, it might mean either there is no CPE for the library (and hopefully no publicly known vulnerability) or ODC has failed to assign the CPE. There is no easy way to know that is the case.

False positives

False positives are implied by the heuristics used mainly for detecting CPE identifier. We ca’t get rid of all of them, until a better mechanism (e.g. CPE identifiers in POMs) is developed and widely used. Especially the latter would be rather a long run.

False positives can be suppressed in ODC in two ways. One can suppress assignment of a specific CPE to a specific library (identified by SHA1 hash). If a more fine-grained control is needed, one can suppress assignment of a single vulnerability to a specific library (identified by SHA1 hash). Both of them make sense when handling false positives.

Handling a CPE collision

For example, there is NLog library for logging in .NET. ODC assigns it cpe:/a:nlog:nlog. This might look correct, but this CPE has been used for some Perl project and there is some vulnerability for this CPE that is not related to the logging library.

If one suppressed matching the CPE, one could get some false negatives in future, as vulnerabilities of the NLog library might be published under the same CPE, according to a discussion post by Jeremy Long (the author of the tool).

In this case, we should not suppress those CPEs. We should just suppress CVEs for them.

Obviously mismatched CPEs

CPE might be also mismatched obviously. While the case is similar to the previous one, the CPE name makes it obvious that it does not cover the library. In this case, we can suppress the CPE for the SHA1 hash.

Missing context

Vulnerability description usually contains a score. However, even if the score is 10 out of 10, it does not imply that the whole system is vulnerable. In general, there are multiple other aspects that might change the severity for a particular case:

  • The library might be isolated from untrusted input.
  • The untrusted input might be somehow limited.
  • The library might run in sandboxed environment.
  • Preconditions of the vulnerability are unrealistic for the context.

None of them can be detected by ODC and automatic evaluation of those aspects is very hard, if it is not impossible.

Conclusion

We have integrated OWASP Dependency Check with our environment. We have evaluated the quality of reports. The most fragile part is matching a library to a CPE identifier, which may lead to both false negatives (rarely seen) and false positives (sometimes seen). Despite those issues, we have found ODC to be useful enough and adaptable to our environment.

Most systems today need to handle the user authentication. That means, the password entered during user registration must be stored in the system for later comparison.

It is obvious that the passwords must not be stored in plain-text form. In that case, if an attacker succeeded in getting access to the database, where these passwords are stored (e.g. using SQL Injection), he would obtain the whole list of user names with their corresponding passwords. Then it is very simple for him to impersonate a valid user.

Hashing

However, to check, if the password entered by the user is correct, we do not need the original password. It is enough to have a suitable information, which uniquely identifies it and can be easily computed from each password entering the system.

Such information is the password hash. Hash algorithm is a one-way function, generating a fixed-length string from the inputs (in this case from the given password) with no possibility to derive these inputs back from the computed string. Another property of a cryptographic hash function is that change of one input bit leads to change of many bits in the resulting hash. When the hash function is collision-free, we can assume that the identical hashes imply the identical inputs, from which these hashes are computed.

So instead of the password itself, only its hash will be stored in the system. Every time a user tries to login to the system, hash of the password entered is computed and compared to the stored one.

Slow hashing

However, cryptographic hash functions such as MD5 or SHA are not appropriate. The purpose of these functions is calculation of digest of large amount of data to ensure its integrity. This digest needs to be computed in as short time as possible, and thus these hash functions are designed to be fast. This property is, however, not desirable for password hashing.

As an example take the MD5 function. One 2.13GHz core is able to compute cca 6 million MD5 hashes per second using Cain & Abel tool. Trying every single possible 8 character long lowercase alphanumeric password then takes approximately 130 hours. And that is only one core. Modern computers use more of them, for example with six such cores a password can be cracked in less than a day. Furthermore, we can definitely assume that an attacker has much better equipment.

In order to prevent an attacker from trying millions of hashes per second, we need to use a slow cryptographic hash function for password hashing. Several hash functions were specifically designed for this purpose. These functions include: PBKDF2, bcrypt, scrypt.

Work factor parameters

These hash functions are not only slow, they also come up with work factor parameters defining how expensive the hash computation will be. Although the scrypt function is the youngest one (designed in 2009), it has an advantage over the older ones – it not only defines the CPU cost, but also the memory requirements. That is why scrypt is recommended function for password storage and this article talks mainly about it.

Scrypt uses following work factor parameters:

  • N – number of iterations, related to both memory and CPU cost
  • r – size of the RAM block needed, related to memory cost
  • p – parallelization, defines maximum number of threads, related to CPU cost

These parameters allow to set the memory needed and time it takes to compute one hash. The approximate memory usage for a single hash generation can be computed from the parameters using the following formula:

memory  =  N  ·  2  ·  r  ·  64

The time, on the other hand, is platform-dependent. The graph below shows dependency of time needed for single hash computation on the work factor parameters N and r. The parallelization parameter is set to 1 in all cases. The values in the graph were measured using CryptSharp, the C# implementation of scrypt function, on Windows Server 2012 with four 2.2GHz cores.

csharp_Server12_scrypt

It is needed to specify the computation time as a compromise between the usability and security provided. For example, if we have a system with only one login at a time and high security is needed, we set the parameters to make computation take cca one second. However, in case of many parallel logins this time needs to be set to only few milliseconds.

We can take the above example of password hash cracking. Using scrypt function (CryptSharp implementation) with parameters N=210, r=4 and p=1, hashing of one password takes approximately 10ms, i.e. this 2.2GHz core is able to compute 100 hashes per second. Then computation of all possible 8 character long lowercase alphanumeric passwords takes 895 years.

Attacker goals

Imagine an attacker, who obtained the list of user names and corresponding password hashes. There are now three goals he can have:

  • Crack a password of one specific user (e.g. admin)
  • Crack a password of any user
  • Crack passwords of a longer list of users

Attacks

In the first option the attacker has a password hash and wants to find the corresponding password it was computed from. He can use brute force or dictionary attack, i.e. try many possible inputs to the hashing function and compare the results with the obtained password hash.
An effective method for trying so many hashes is usage of lookup tables. The general idea is to pre-compute hashes of possible passwords and store them in a lookup table data structure (or Rainbow tables for lower memory requirements). Comparison of these pre-computed values with given hash is much faster than hash computation.

The second option is simpler. The only thing needed is to compute hashes of possible inputs and compare each result with all password hashes in the obtained list. Sooner or later the attacker will hit some match.

For cracking a longer list of hashes the attacker does not need to crack one password at a time, he will instead compare each computed hash with all hashes from the list. This way cracking of a hashes list takes approximately the same time as cracking only one specific password.

Salt

The above attacks work because each password is hashed the same way, the same password always results in the same hash. The simplest way of preventing against this is salting. That means, a random string (salt) is generated for each password and used together with it to create a hash.
It is needed to ensure uniqueness of the salts, thus they really need to be randomly generated. Any random number generator can be used, however, cryptographically secure RNGs, such as RNGCryptoServiceProvider in C# or SecureRandom in Java, are recommended.

The salt is a non-secret value, it needs to be stored together with the password hash to ensure its availability to the hash function. Thus, if someone gets access to the hashes, he automatically gets also all the salts. However, the salt power is not in its secrecy, but in randomness.

With different salt, same passwords result in different hashes. Pre-computed hash attack is infeasible due to a large additional memory requirements – an attacker needs to store pre-computed hashes for each possible salt.

Cracking password of any user is reduced to cracking password of a specific one, since the salt for each user password is different.

Also cracking of a larger list of hashes is more complicated with different salt for each password, the attacker has no other choice than cracking one password at a time.

Pepper

In order to increase security even more, we can use another randomly generated string – pepper. In comparison to the salt, pepper needs to be kept secret as it is used as an HMAC key. HMAC is a one-way algorithm based on hash function generating fixed-length string from the input message and a secret key, which in our case is generated pepper.

Since pepper is a secret key, it needs to be generated using a cryptographically secure random number generator, such as RNGCryptoServiceProvider.

public static byte[] GeneratePepper()
{
    byte[] pepper = new byte[32];
    RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
    rng.GetBytes(pepper);
    return pepper;
}

When generated, the pepper must be stored separately in a configuration file with restricted access.

Although an attacker had enough resources to be able to crack the hash function, he would still need this secret value for obtaining the user password. And with pepper randomly generated for each system instance, if one instance is compromised, other remain secure.

Overall scheme

The overall hashing of the password with both salt and pepper looks as follows:

scrypt ( Base64 ( HMAC ( ‘SHA256’, password, pepper ) ), salt, workFactors )

And the C# implementation of this scheme using CryptSharp library:

public byte[] HashPassword(String password, byte[] pepper, byte[] salt)
{
    if (salt == null)
    {
        Console.WriteLine("Password hash not created - salt is null.");
        return null;
    }

    String encodedHmac = HmacBase64(password, pepper);
    return CryptSharp.Utility.SCrypt.ComputeDerivedKey(Encoding.UTF8.
         GetBytes(encodedHmac), salt, n, r, p, null, HASH_LENGTH);
}

private static string HmacBase64(string password, byte[] pepper)
{
    if (pepper == null)
    {
        Console.WriteLine("Password hash not created - pepper is null.");
        return null;
    }
    HMACSHA256 hmac = new HMACSHA256(pepper);
    hmac.Initialize();
    byte[] buffer = Encoding.UTF8.GetBytes(password);
    byte[] rawHmac = hmac.ComputeHash(buffer);
    return System.Convert.ToBase64String(rawHmac);
}

Conclusion

User passwords must never be stored as plain text, always compute its hash using a slow cryptographic hash function. To each password generate random salt and use this value together with the password for hash computation. For higher level of security generate random secret pepper for each system instance.

Of course, security of the user password depends on the password itself. An attacker could still try frequently used passwords such as “123456”, however, with secure storage we can protect him from trying too many of them and from obtaining the strong ones.

After finishing hard-coded passwords detector, I have focused on improving the detection of the most serious security bugs, which could be found by static taint analysis. SQL injection, OS command injection and Cross-site scripting (XSS) are placed as top first, second and fourth in CWE Top 25 most dangerous software errors (while well-known buffer overflow, not applicable to Java, is placed third). Path Traversal, Unvalidated Redirect, XPath injection or LDAP injection are also related types of weaknesses – unvalidated user input can exploit syntax of an interpreter and cause a vulnerability. Injections in general are the risk number one in OWASP Top 10 too, so a reliable open-source static analyser for those kinds of weaknesses could really make the world a more secure place 🙂

FindBugs already has detectors for some kinds of injections, but many bugs is missed due to insufficient flow analysis, unknown taint sources and sinks and also targeting zero false positives (even though there are some). In contrast, the aim of bug detectors in FindSecurityBugs is to be helpful during security code review and not to miss any vulnerability – there was some effort to reduce false positives, but before my contribution almost all taint sinks were reported in practice. Unfortunately, searching a real problem among many false warnings is quite tedious. The aim of the new detection mechanism is to report more high-confidence bugs (with minimum of false positives) than FindBugs detectors plus report lower-confidence bugs with decreased priority not missing any real bugs while having false positives rate much lower than FindSecurityBugs had originally.

For a reliable detection, we need a good data-flow analysis. I have already mentioned OpcodeStackDetector class in previous articles, but there is a more advanced and general mechanism in FindBugs. We can create and register classes performing a custom data-flow analysis and request those results later in detectors. Methods are symbolically executed after building control flow graph made of blocks of instructions connected by different types of edges (such as goto, ifcmp or exception handling), which are attempted to be pruned for impossible flow. We have to create a class to represent facts at different code locations – we want to remember some information (called a fact) for every reachable instruction, which can later help us to decide, whether a particular bug should be reported at that location. We need to model effects of instructions and edges on facts, specify the way of merging facts from different flow branches and make everything to work together. Fortunately, there are existing classes designed for extension to make this process easier. In particular, FrameDataflowAnalysis models values in the operand stack and local variables, so we can concentrate on the sub-facts about these values. The actual fact is then a frame of these sub-facts. This class models effects of instructions by pushing the default sub-fact on the modelled stack and popping the right amount of stack values. It also automatically moves sub-facts between the stack and the part of the frame with local variables.

Lets have a look, which classes had to be implemented for taint analysis. If we want to run custom data-flow analysis, a special class implementing IAnalysisEngineRegistrar must be created and referenced from findbugs.xml.

<!-- Registers engine for taint analysis dataflow -->
<EngineRegistrar
    class="com.h3xstream.findsecbugs.taintanalysis.EngineRegistrar"/>

This simple class (called EngineRegistrar) makes a new instance of TaintDataflowEngine and registers it with global analysis cache.

public class EngineRegistrar implements IAnalysisEngineRegistrar {

    @Override
    public void registerAnalysisEngines(IAnalysisCache cache) {
        new TaintDataflowEngine().registerWith(cache);
    }
}

Thanks to this, in the right time, method analyze of TaintDataflowEngine (implementing ImethodAnalysisEngine) is called for each method of analyzed code. This method requests objects needed for analysis, instantiates two custom classes (mentioned in next two sentences) and executes the analysis.

public class TaintDataflowEngine
    implements IMethodAnalysisEngine<TaintDataflow> {

    @Override
    public TaintDataflow analyze(IAnalysisCache cache)
            throws CheckedAnalysisException {
        CFG cfg = cache.getMethodAnalysis(CFG.class, descriptor);
        DepthFirstSearch dfs = cache
            .getMethodAnalysis(DepthFirstSearch.class, descriptor);
        MethodGen methodGen = cache
            .getMethodAnalysis(MethodGen.class, descriptor);
        TaintAnalysis analysis = new TaintAnalysis(
            methodGen, dfs, descriptor);
        TaintDataflow flow = new TaintDataflow(cfg, analysis);
        flow.execute();
        return flow;
    }

    @Override
    public void registerWith(IAnalysisCache iac) {
        iac.registerMethodAnalysisEngine(TaintDataflow.class, this);
    }
}

TaintDataflow (extending Dataflow) is really simple and used to store results of performed analysis (used later by detectors).

public class TaintDataflow
        extends Dataflow<TaintFrame, TaintAnalysis> {

    public TaintDataflow(CFG cfg, TaintAnalysis analysis) {
        super(cfg, analysis);
    }
}

TaintAnalysis (extending FrameDataflowAnalysis) implements data-flow operations on TaintFrame but it mostly delegates them to other classes.

public class TaintAnalysis
        extends FrameDataflowAnalysis<Taint, TaintFrame> {

    private final MethodGen methodGen;
    private final TaintFrameModelingVisitor visitor;

    public TaintAnalysis(MethodGen methodGen, DepthFirstSearch dfs,
            MethodDescriptor descriptor) {
        super(dfs);
        this.methodGen = methodGen;
        this.visitor = new TaintFrameModelingVisitor(
            methodGen.getConstantPool(), descriptor);
    }

    @Override
    protected void mergeValues(TaintFrame frame, TaintFrame result,
            int i) throws DataflowAnalysisException {
        result.setValue(i, Taint.merge(
            result.getValue(i), frame.getValue(i)));
    }

    @Override
    public void transferInstruction(InstructionHandle handle,
            BasicBlock block, TaintFrame fact)
            throws DataflowAnalysisException {
        visitor.setFrameAndLocation(
            fact, new Location(handle, block));
        visitor.analyzeInstruction(handle.getInstruction());
    }

    // some other methods
}

TaintFrame is just a concrete class for abstract Frame<Taint>.

public class TaintFrame extends Frame<Taint> {

    public TaintFrame(int numLocals) {
        super(numLocals);
    }
}

Effects of instructions are modelled by TaintFrameModelingVisitor (extending AbstractFrameModelingVisitor) so we can code with the visitor pattern again.

public class TaintFrameModelingVisitor
    extends AbstractFrameModelingVisitor<Taint, TaintFrame> {

    private final MethodDescriptor methodDescriptor;

    public TaintFrameModelingVisitor(ConstantPoolGen cpg,
            MethodDescriptor method) {
        super(cpg);
        this.methodDescriptor = method;
    }

    @Override
    public Taint getDefaultValue() {
        return new Taint(Taint.State.UNKNOWN);
    }

    @Override
    public void visitACONST_NULL(ACONST_NULL obj) {
        getFrame().pushValue(new Taint(Taint.State.NULL));
    }

    // many more methods
}

The taint fact – information about a value in the frame (stack item or local variable) is stored in a class called just Taint.

The most important piece of information in Taint is the taint state represented by an enum with values TAINTED, UNKNOWN, SAFE and NULL. TAINTED is pushed for invoke instruction with a method call configured to be tainted (e.g. getParameter from HttpServletRequest or readLine from BufferedReader), SAFE is stored for ldc (load constant) instruction, NULL for aconst_null and UNKNOWN is a default value (this description is a bit simplified). Merging of taint states is defined such that if we could compare them as TAINTED > UNKNOWN > SAFE > NULL, then merge of states is the greatest value (e.g. TAINTED + SAFE = TAINTED). Not only this merging is done where there are more input edges to a code block of control flow graph, but I have also implemented a mechanism of taint transferring methods. For example, consider calling toLowerCase method on a String before passing it to a taint sink – instead of pushing a default value (UNKNOWN), we can copy the state of the parameter not to forget the information. Merging is also done in more complicated examples such as for append method of StringBuilder – the taint state of the argument is merged with the taint state of StringBuilder instance and returned to be pushed on the modelled stack.

There were two problems with taint state transfer which had to be solved. First, taint state must be transferred directly to mutable classes too, not only to their return values (plus the method can be void). Not only we set the taint state for an object when it is being seen for the first time in the analysed method and then the state is copied, but we also change it according to instance methods calls. For example, StringBuilder is safe, when a new instance is created with non-parametric constructor, but it can taint itself by calling its append method. If only methods with safe parameters are called, the taint state of StringBuilder object remains safe too.  For this reason, effect of load instructions is modified to mark index of loaded local variable to Taint instance of corresponding stack item. Then we can transfer taint state to a local variable with index stored in Taint for specified methods in mutable classes. Second, taint transferring constructors (methods <init> in bytecode) must be handled specifically, because of the way of creating new objects in Java. Instruction new is followed by dup and invokespecial, which consumes duplicated value and initializes the object remaining at the top of the stack. Since the new object is not stored in any variable, we must transfer the taint value from merged parameters to the stack top separately.

Bugs related to taint analysis are identified by TaintDetector (implementing Detector). For better performance, before methods of some class are analyzed, constant pool (part of the class file format with all needed constants) is searched and the analysis continues only if there are references for some taint sinks. Then TaintDataflow instance is loaded for each method and locations of its control flow graph are iterated until taint sink method is found. This means, we find all invoke instructions used in a currently analysed method and check, whether the called methods are related to the searched weaknesses. Facts (instances of Taint class) from TaintDataFlow are extracted for each sink parameter of a sink method. Bug is reported with high confidence (and priority), if the taint state is TAINTED, with medium confidence for UNKNOWN taint state and with low confidence for SAFE and NULL (just for the case of a bad analysis, these warnings are not normally shown anywhere). Taint class also contains references for taint source locations, so these are shown in bug reports to make review easier – you should see a path between taint sources and the taint sink. TaintDetector itself is abstract, so it must be extended to detect concrete weakness types (like command injection) and InjectionSource interface implemented to specify taint sinks (the name of the interface is a bit misleading) and items in a constant pool to specify candidate classes.

public class CommandInjectionDetector extends TaintDetector {

    public CommandInjectionDetector(BugReporter bugReporter) {
        super(bugReporter);
    }

    @Override
    public InjectionSource[] getInjectionSource() {
        return new InjectionSource[] {new CommandInjectionSource()};
    }
}

CommandInjectionSource overwrites method getInjectableParameters, which returns an instance of InjectionPoint containing parameters, that cannot be tainted, and the weakness type to report. Boolean method isCandidate looks up constant pool for the names of taint sink classes and return true if present.

TaintDetector is currently used to detect command, SQL, LDAP and script (for eval method of ScriptEngine) injections and unvalidated redirect. More bug types and taint sinks should follow soon. Test results are looking quite promising so far. Inter-procedural analysis (not restricted to a method scope) should be the next big improvement, which could make this analysis really helpful. Then everything should be tested with a large amount of real code to iron out the kinks. You can see the discussed classes in taintanalysis package and try the new version of FindSecurityBugs.

FindBugs GUI

In the previous article, I was describing the creation of a new FindBugs detector for hard-coded passwords and cryptographic keys. I also mentioned some imperfections and I have decided to learn more about FindBugs and improve the detection.

Java virtual machine has a stack architecture – operands must be pushed on the stack before method is invoked, given number of stack values is consumed during invocation and produced return value (if any) is pushed subsequently. My detector class extends OpcodeStackDetector, which implements abstract interpretation technique to collect approximated information about values at the operand stack for each code location. These pieces of information (usually called facts) are kept only for those locations, where a derived value of the fact does not depend on the preceding control flow (for example, the value is the same for each possible branch executed in earlier conditional statements).

One of the facts available for stack values is the actual value of a number or String (related to constant propagation performed by compilers during optimization). We can use this to detect hard-coded values – known constant means hard-coded value. However, we also need to track other other data types besides Strings (numbers can be ignored) to detect passwords in a char array and hard-coded keys. In addition, there is one more issue with this approach…

Tracking concrete values is unnecessarily complicated and the value often becomes unknown – we only need to know whether the value is constant, not which constant is on the stack. Consider a piece of code like this:

private Connection getConnection(String user) {
    String password;
    if ("root".equals(user)) {
        password = "superSecurePassword";
    } else {
        password = "differentPassword";
    }
    return DriverManager.getConnection(DB_URL, user, password);
}

The constant value in the password variable is known inside both branches, but these values are forgotten after the conditional statement, since the values differ. For this reason, weakness like this was not reported by the previous version of the detector nor by the original FindBugs detector for constant database passwords. Even if there is only one possible constant, analysis can fail because of null values, see this code (looking a bit unreal, but demonstrating the problem):

String pwd = null;
if (shouldConnect()) {
    pwd = "hardcoded";
}
if (pwd != null) {
    Connection connection = DriverManager.getConnection(url, user, pwd);
    // some code working with database
}

We can easily see that the password variable has always value “hardcoded”, but the performed analysis is linear and the fact is forgotten right after the first conditional statement. Second condition cannot return the forgotten constant back and weakness is not detected again.

Fortunately, these issues can be solved by setting and reading user value fact, which OpcodeStackDetector allows (if annotation CustomUserValue is added to the extending class). Our fact has only one value for hard-coded stack items or it is null to indicate unknown state (default). We can periodically check, whether a value on the stack is a known constant or null and set the user value for it if it is, propagation is done automatically. If then analysis merges facts from different control flow branches with different constants (or null), user value is the same and not reset to default. Custom user value is also used to mark hard-coded passwords and keys with the other password and key data types, detection of those objects remains similar as in the previous version of the detector. Weakness is reported if sink parameter has non-null user value and stack value is not null (null passwords are not considered to be hard-coded).

The detector is using proper flow analysis after this improvement; however, it is restricted to a method scope and hard-coded values in fields are reported only if used in the same class. Inter-method and inter-class analysis is a future challenge, but I have kept reporting of hard-coded fields with suspicious names and unknown sink not to miss important positives. In contrast to the previous version, these fields are reported with lower priority and only if not reported before using proper flow analysis technique to prevent duplicate warning. Moreover, all fields are reported in a single warning for a class to make possible false positives less distracting.

Another improvement is the possibility to configure more method parameters as password or key sinks. If more than one parameter is hard-coded, only single warning is produced and the parameters are mentioned in the detailed message. The last important change is that hard-coded cryptographic keys are reported in a separated bug pattern since both hard-coded passwords and keys have a different CWE identifier (259 and 321) and are equally important. Decision between reported warnings is done automatically based on the data types of hard-coded parameters.

I have tested the detector with Juliet Test Suite and using proper analysis it can reveal both types of weakness in 17 flow variants (out of 37) and all sink variants with no false positives. Original FindBugs detector reveals weaknesses in 10 flow variants and only for database passwords, other password methods and hard-coded keys are not detected in any variant.

You can see the detector class on GitHub. Happy coding with no hard-coded passwords!

software bug

FindBugs is a great open source tool for detection of software bugs in Java. It uses static analysis to search compiled classes for hundreds of bug patterns and even more can be found using FindSecurityBugs and fb-contrib plugins. However, before my recent contribution there was no general detector for hard-coded passwords and cryptographic keys. Hard-coded password are identical for each installation and can be easily extracted, which is likely to be exploited (see CWE-259 for more information). FindBugs and FindSecurityBugs could already detect this vulnerability, but only for constant database passwords and two very specific cases. I have created a detector (accepted to FindSecurityBugs), which is able to find hard-coded values of Strings, char and byte arrays or BigIntegers used as an input parameter for one of the configured methods such as KeyStore.load or KeySpec constructors.

To add a new detector, we have to create a class that implements Detector or extends a prepared class with helper functionality (I have used OpcodeStackDetector). An instance of BugReporter passed in constructor and its method reportBug are used to report problems (BugInstance objects). We also need to add the class name to findbugs.xml to be executed and edit messages.xml for information about detections. Good start for thinking about the detection logic is writing a bunch of flawed code samples and looking to their bytecode (plugin for IDEA can be used). We can write unit tests for them by mocking BugReporter.

Detector class can use visitor design pattern (if it implements Visitor) to react on events while analyzed class is scanned. I have started by overwriting method sawOpcode, which is called every time an instruction is read. Since we are interested in invocations of specific methods, we need to check if it is one of the invoke instructions and get full called method name such as java/security/KeyStore.load(Ljava/io/InputStream;[C)V, which contains method class with package, argument types ([C is char array, parameters of object types starts with L and ends with semicolon) and return type (V for void). Method name can be obtained by calling methods getClassConstantOperand, getNameConstantOperand and getNameSigOperand inherited from DismantleByteCode. If it is one of the problematic methods (loaded from resource files), we can create a BugInstance, add current analyzed line plus some info to it and report the bug. Now we have a password usage detector but not hard coded password detector, so it is time to eliminate false positives (detections that are not real bugs).

For String passwords, we can utilize OpcodeStackDetector and check nullness of stack.getStackItem(0).getConstant() to detect usage of constant String as a first method parameter. Unfortunately, it is not so easy for the other variable types. To detect that an array is initialized with hard-coded values, I am checking whether instruction for new array creation is followed by push and array store instructions while for example not calling methods. Constant arrays are also converted from constant Strings using methods toCharArray and getBytes. After implementing this, we can detect BigIntegers too, since they can be constructed from Strings or byte arrays.

In terms of so called taint analysis, we are able to detect vulnerability source (hard-coded data) and sink (usage of password or cryptographic function), but bug should be reported only if there is a flow from source to sink (we cannot be sure that hard-coded password is really a password until it is used as a password parameter). In the current implementation, no complex flow analysis is performed, we assume that a taint source followed by a taint sink of a matching type inside the same method body are always related. For this reason, false positives are easy to demonstrate, but are quite uncommon in practice. On the other hand, local hard-coded declarations are forgotten while another method is analyzed (visit method is overwritten to reset the state), so passwords are not detected if they are passed as a parameter and used in another method.

Class fields are also taken into an account – if constant data is stored to them, we remember that and consider it as a taint source when they are read. Because of this, the order of the methods matters and as static initializer section is added to the class end by compiler, its analysis is run ‘manually’ by calling doVisitMethod when class analysis starts. In addition, if the field stores hard-coded data and its name is suspicious (like password, secret or aesKey), the bug is reported immediately, since there is a high bug confidence and if it was used in a different class, it would not be reported otherwise (one the other hand, it can be reported twice now).

You can see the whole code on GitHub. I have mentioned some imperfections, but I think the detector is working quite well. Unfortunately, there is not much information about writing detectors, so creating them can be just a matter of trial and error. If you have an idea for improvement or a new detector, don’t hesitate to contact me or pull the code directly. 🙂

sdl-shield-greenNowadays, security becomes an important aspect of almost every software system. Unfortunately, it does not necessary mean that security is adequately considered in every piece of software. What does “adequately” mean? For me, security measures are adequate if an investment to them is less than loss caused if these measures were not implemented. I would say this explanation is clear if there is no unknown variable there, i.e., the potential loss.

If a company has no security incidents so far, does it mean that its security measures are adequate and it should not put additional investments in security? It is really hard to say yes or no. The software may change (e.g., it may introduces new features with new attack vectors), the market can change (e.g., more people can start using the software, so it may become more attractive to attackers), and the environment where the software is used may change (e.g., from an intranet solution it may become a public cloud based solution).

The history gives us a generous hint. No matter what, security incidents happen. The question is only when a what impact it will have on the customers and the company.

“Security is a process, not a product”
– Bruce Schneier

In order to create secure products, a company should regularly evolve and adapt its development processes to changes that continuously take place around us.
Here, in Y Soft, we decided to examine and try in one of our development teams the Microsoft Secure Development Lifecycle (SDL) that is a process aiming to help to build more secure software.

In this series of posts, I would like to:

  • Look at Microsoft SDL in more details,
  • Describe its advantage/disadvantages as we perceive them here in Y Soft,
  • Describe obstacles we meet and how we overcome them.

Secure Development Lifecycle (SDL) is a process that helps to build more secure software. Microsoft introduced its SDL in 2004, successfully uses and regularly updates it. Microsoft SDL covers all phases of development process, e.g., Requirements, Design, Implementation, Verification and Release.

Training → Requirements → Design → Implementation → Verification → Release → Response

Microsoft SDL describes tasks that should be completed at each phase and also provides tools and training materials that are available free of charge.
Key features of Microsoft SDL, as I perceive them, are:

  1. Security is considered systematically. Developers and managers can be certain to some degree that important security issues were not overlooked and they will not emerge at release (or after release) phase where their resolution is often not cost effective.
  2. Microsoft SDL is designed to be a part of software development process. And this is probably the most important thing about it. A bullet-proof system does not exist. New attacks appear all the time. And it is important that new defenses that come after are in place and in time.
  3. Security is considered early in the product lifecycle. The process starts from the requirements phase. It means that design reflects requirements on security features and features that need to be secure. The process that starts from the requirements phase, among other, decreases the risk of security design flaws in the later phases and hence decreases the total development cost.
  4. Process is primarily focused on developers. Most of the tasks from the Microsoft SDL are completed by developers themselves. The tasks include among others, threat modeling, secure design, code analysis using static analysis tools, etc. Security experts are those who are consulting, reviewing and helping developers to get their job done right.
  5. It can be integrated gradually. Microsoft SDL defines four maturity levels (Basic, Standardized, Advanced and Dynamic) where each level has additional tasks (that should be completed) in comparison to the previous one.
  6. It naturally fits the software development process. Tasks and activities defined in the Microsoft SDL logically follow or extend tasks that are normally done by developers. For example, for systematic threat analysis, a developer can utilize the Microsoft SDL Threat Modeling Tool. This tool uses Data Flow Diagrams (DFDs) that resemble container/component diagram from C4 model that is used by our developing teams.

Next time, we will look at the training and requirements phase, and discuss activities and tasks we decided to implement.