ECOS Solver for Java
We are excited to announce that ecos4j (ECOS Solver for Java) is now released and open-source! It provides an interface from the Java programming language to the native open source mathematical programming solver ECOS. The library invokes the native solver through Java’s new Foreign Function and Memory (FFM) API.
ecos4j allows solving second-order cone programs (SOCPs). Many convex optimization problems from a wide range of disciplines such as AI, finance, machine learning, and statistics may be formulated as SOCPs, and modern conic optimizers such as ECOS provide a robust and efficient way to solve these problems.
Convex Optimization with ecos4j
Mathematically, ecos4j solves SOCPs in the standard form
\[\begin{align*} & \text{minimize} & & c^T x \\ & \text{subject to} & & A x = b \\ & & & G x + s = h, \qquad s \in \mathcal{K} \end{align*}\]
where \(x \in \mathbf{R}^n\) are the primal variables, \(s \in \mathbf{R}^m\) are slack variables, \(c \in \mathbf{R}^n, \, A \in \mathbf{R}^{p \times n}, \, b \in \mathbf{R}^p, \, G \in \mathbf{R}^{m \times n}, \, h \in \mathbf{R}^m\) are the problem data, and \(\mathcal{K}\) is the convex cone. The cone \(\mathcal{K}\) is the Cartesian product of the following primitive cones
\[\begin{align*} & \text{positive orthant cone} & & \{s \; | \; s \geq 0 \} \\ & \text{second-order cone} & & \{ (t, s) \in \mathbf{R} \times \mathbf{R}^k \; | \; \| s \|_2 \leq t \} \\ & \text{exponential cone} & & \{ (x, y, z) \in \mathbf{R}^3 \; | \; y e^{x/y} \leq z, \, y > 0 \} \end{align*}\]
As an example of convex optimization in quantitative finance consider the Markowitz portfolio optimization problem, where a long-only investor wishes to maximize the expected portfolio return given a limit on the portfolio risk
\[\begin{align*} & \text{maximize} & & \mu^T x \\ & \text{subject to} & & x^T \Sigma x \leq \sigma^2 \\ & & & \mathbf{1} x = 1 \\ & & & x \geq 0 \end{align*}\]
where \(x\) is the unknown vector of portfolio allocations, \(\mu\) is the estimated expected return vector, \(\Sigma\) is the estimated covariance matrix, and \(\sigma\) is the given limit on the portfolio risk.
In order to solve the Markowitz portfolio optimization problem with ecos4j, it is necessary to convert it to a SOCP
\[\begin{align*} & \text{minimize} & & -\mu^T x \\ & \text{subject to} & & \mathbf{1} x = 1 \\ & & & x \geq 0 \\ & & & t \leq \sigma \\ & & & \| G^T x \|_2 \leq t \end{align*}\]
where \(G\) is such that \(\Sigma = G G^T\). \(G\) may be found e.g. by the Cholesky decomposition. The first two inequalities may be modeled with a positive orthant, the third with a second-order cone.
Using ecos4j in Java
Setup
To use ecos4j, the latest (at the time of writing) JDK 22 is needed as ecos4j depends on Java’s new FFM API.
The library is available at
Maven Central. For Maven add the latest version to your pom.xml
<dependency>
<groupId>com.ustermetrics</groupId>
<artifactId>ecos4j</artifactId>
<version>x.y.z</version>
</dependency>
or using Gradle
implementation 'com.ustermetrics:ecos4j:x.y.z'
For Windows or Linux users, there is a separate library ecos4j-native that wraps the native shared library binaries.
Using Maven add the latest version from Maven Central to your pom.xml
<dependency>
<groupId>com.ustermetrics</groupId>
<artifactId>ecos4j-native</artifactId>
<version>x.y.z</version>
<scope>runtime</scope>
</dependency>
or using Gradle
runtimeOnly 'com.ustermetrics:ecos4j-native:x.y.z'
Alternatively, install the native ECOS solver on the machine and add the location to the java.library.path
.
ecos4j dynamically loads the native solver.
Since ecos4j invokes some restricted methods of the FFM API, use the --enable-native-access=com.ustermetrics.ecos4j
or --enable-native-access=ALL-UNNAMED
(if you are not using the Java Platform Module System) VM options when running the Java code to avoid warnings from the Java run-time.
Using ecos4j
As an example consider the Markowitz portfolio optimization problem.
First, the data of the problem needs to be prepared. We are using the Efficient Java Matrix Library (EJML) to define the portfolio optimization problem and to convert it into a SOCP
// Define portfolio optimization problem
val mu = new SimpleMatrix(new double[]{0.05, 0.09, 0.07, 0.06});
val sigma = new SimpleMatrix(
4, 4, true,
0.0016, 0.0006, 0.0008, -0.0004,
0.0006, 0.0225, 0.0015, -0.0015,
0.0008, 0.0015, 0.0025, -0.001,
-0.0004, -0.0015, -0.001, 0.01
);
val sigmaLimit = 0.06;
// Problem dimension
val n = mu.getNumRows();
// Compute Cholesky decomposition of sigma
val chol = DecompositionFactory_DDRM.chol(n, true);
if (!chol.decompose(sigma.copy().getMatrix()))
throw new IllegalArgumentException("Cholesky decomposition failed");
val upTriMat = SimpleMatrix.wrap(chol.getT(null)).transpose();
// Define second-order cone program
val cMat = mu.negative()
.concatRows(new SimpleMatrix(1, 1));
System.out.println("\ncMat");
cMat.print();
val aMat = SimpleMatrix.ones(1, n)
.concatColumns(new SimpleMatrix(1, 1));
System.out.println("\naMat");
aMat.print();
val bMat = SimpleMatrix.ones(1, 1);
System.out.println("\nbMat");
bMat.print();
val gMatPosOrt = SimpleMatrix.identity(n)
.negative()
.concatColumns(new SimpleMatrix(n, 1))
.concatRows(new SimpleMatrix(1, n).concatColumns(SimpleMatrix.ones(1, 1)));
val gMatSoc = new SimpleMatrix(1, n)
.concatColumns(SimpleMatrix.filled(1, 1, -1.0))
.concatRows(upTriMat.negative().concatColumns(new SimpleMatrix(n, 1)));
val gMat = gMatPosOrt.concatRows(gMatSoc);
System.out.println("\ngMat");
gMat.print();
val hMat = new SimpleMatrix(2 * n + 2, 1);
hMat.set(n, 0, sigmaLimit);
System.out.println("\nhMat");
hMat.print();
// ecos4j needs sparse aMat and gMat
val tol = 1e-8;
val aSpMat = DConvertMatrixStruct.convert(aMat.getDDRM(), (DMatrixSparseCSC) null, tol);
System.out.println("\naSpMat");
aSpMat.print();
val gSpMat = DConvertMatrixStruct.convert(gMat.getDDRM(), (DMatrixSparseCSC) null, tol);
System.out.println("\ngSpMat");
gSpMat.print();
Then, the SOCP can be solved with ecos4j
// Create model
try (val model = new Model()) {
// Set up model
model.setup(n + 1, new long[]{n + 1}, 0, getNzValues(gSpMat), getColIdx(gSpMat),
getNzRows(gSpMat), cMat.getDDRM().data, hMat.getDDRM().data, getNzValues(aSpMat),
getColIdx(aSpMat), getNzRows(aSpMat), bMat.getDDRM().data);
// Create and set parameters
val parameters = Parameters.builder()
.verbose(true)
.build();
model.setParameters(parameters);
// Optimize model
val status = model.optimize();
if (status != OPTIMAL) {
throw new IllegalStateException("Optimization failed");
}
// Get solution
val xMat = new SimpleMatrix(model.x())
.extractMatrix(0, n, 0, 1);
System.out.println("xMat");
xMat.print();
}
where some helper methods are used to ensure that the three arrays representing a CSC sparse matrix have the right dimension and type, and that the row index is sorted within columns
private static double[] getNzValues(DMatrixSparseCSC matrix) {
if (!matrix.isIndicesSorted()) matrix.sortIndices(null);
if (matrix.nz_values.length == matrix.nz_length) {
return matrix.nz_values;
} else {
return Arrays.copyOfRange(matrix.nz_values, 0, matrix.nz_length);
}
}
private static long[] getNzRows(DMatrixSparseCSC matrix) {
if (!matrix.isIndicesSorted()) matrix.sortIndices(null);
if (matrix.nz_rows.length == matrix.nz_length) {
return toLongArray(matrix.nz_rows);
} else {
return toLongArray(Arrays.copyOfRange(matrix.nz_rows, 0, matrix.nz_length));
}
}
private static long[] getColIdx(DMatrixSparseCSC matrix) {
return toLongArray(matrix.col_idx);
}
private static long[] toLongArray(int[] arr) {
return Arrays.stream(arr).asLongStream().toArray();
}
Running the code prints the SOCP, the output of ECOS, and the solution
cMat
Type = DDRM , rows = 5 , cols = 1
-.05
-.09
-.07
-.06
0
aMat
Type = DDRM , rows = 1 , cols = 5
1 1 1 1 0
bMat
Type = DDRM , rows = 1 , cols = 1
1
gMat
Type = DDRM , rows = 10 , cols = 5
-1 -0 -0 -0 0
-0 -1 -0 -0 0
-0 -0 -1 -0 0
-0 -0 -0 -1 0
0 0 0 0 1
0 0 0 0 -1
-.04 -.015 -.02 .01 0
-0 -.149248116 -.008040303 .00904534 0
-0 -0 -.045114893 .016120458 0
-0 -0 -0 -.097766623 0
hMat
Type = DDRM , rows = 10 , cols = 1
0
0
0
0
.06
0
0
0
0
0
aSpMat
Type = DSCC , rows = 1 , cols = 5 , nz_length = 4
1 1 1 1 *
gSpMat
Type = DSCC , rows = 10 , cols = 5 , nz_length = 16
-1 * * * *
* -1 * * *
* * -1 * *
* * * -1 *
* * * * 1
* * * * -1
-.04 -.015 -.02 .01 *
* -.149248116 -.008040303 .00904534 *
* * -.045114893 .016120458 *
* * * -.097766623 *
ECOS 2.0.10 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS
It pcost dcost gap pres dres k/t mu step sigma IR | BT
0 -6.744e-02 -1.288e-01 +7e+00 6e-01 6e-01 1e+00 1e+00 --- --- 1 1 - | - -
1 -6.814e-02 -1.684e-01 +2e+00 6e-02 7e-02 4e-03 3e-01 0.8383 7e-02 1 1 1 | 0 0
2 -6.998e-02 -7.496e-02 +2e-01 4e-03 5e-03 2e-03 2e-02 0.9359 3e-02 1 1 1 | 0 0
3 -7.435e-02 -7.525e-02 +3e-02 6e-04 1e-03 4e-04 5e-03 0.8417 4e-02 1 2 2 | 0 0
4 -7.536e-02 -7.569e-02 +1e-02 2e-04 4e-04 1e-04 2e-03 0.7725 2e-01 1 1 1 | 0 0
5 -7.546e-02 -7.552e-02 +2e-03 4e-05 7e-05 3e-05 3e-04 0.9885 2e-01 1 1 1 | 0 0
6 -7.554e-02 -7.554e-02 +4e-05 8e-07 1e-06 7e-07 7e-06 0.9840 7e-03 1 1 1 | 0 0
7 -7.554e-02 -7.554e-02 +4e-06 7e-08 1e-07 6e-08 6e-07 0.9199 2e-03 1 1 1 | 0 0
8 -7.554e-02 -7.554e-02 +2e-07 3e-09 6e-09 3e-09 3e-08 0.9890 4e-02 1 1 1 | 0 0
9 -7.554e-02 -7.554e-02 +3e-09 7e-11 1e-10 6e-11 6e-10 0.9799 1e-04 1 1 1 | 0 0
OPTIMAL (within feastol=1.2e-10, reltol=4.6e-08, abstol=3.4e-09).
Runtime: 0.001459 seconds.
xMat
Type = DDRM , rows = 4 , cols = 1
3.4290E-09
.296759479
.663419471
.039821047
Using ecos4j in Kotlin and Kotlin Notebooks
Since Kotlin is 100% interoperable with Java, ecos4j may also be used in Kotlin and Kotlin Notebooks. Kotlin Notebooks are interactive editors that integrate code, graphics, text, and markdown text including math formulas in a single environment. When using a Notebook, you can run code cells and immediately see the output. If hosted on Github, then the results from running the Notebook are available as preview.
We created a Kotlin Notebook that contains an extended version of the portfolio optimization example including the computation of the efficient frontier on Github.
Conclusion
This post presents ecos4j, a powerful Java library for solving many convex optimization problems from a wide range of disciplines such as AI, finance, machine learning, and statistics. As an illustration this post shows how to use ecos4j to solve the Markowitz portfolio optimization problem from quant finance.
The source code of ecos4j, ecos4j-native, and the portfolio optimization example may be found on Github: https://github.com/atraplet/ecos4j, https://github.com/atraplet/ecos4j-native, and https://github.com/atraplet/portfolio-optimization.