Run Solr in Openshift

Get a running solr instance on Openshift with s2i images

Albert Lacambra BasilAlbert Lacambra Basil

Create docker file

FROM solr:8.2.0

MAINTAINER  Albert Lacambra Basil "albert@lacambra.tech"
USER root
ENV STI_SCRIPTS_PATH=/usr/libexec/s2i

LABEL io.k8s.description="Run SOLR search in OpenShift" \
      io.k8s.display-name="SOLR 6.6" \
      io.openshift.expose-services="8983:http" \
      io.openshift.tags="builder,solr,solr8.2.0" \
      io.openshift.s2i.scripts-url="image:///${STI_SCRIPTS_PATH}"

COPY ./s2i/bin/. ${STI_SCRIPTS_PATH}
RUN chmod -R a+rx ${STI_SCRIPTS_PATH}

# If we need to add files as part of every SOLR conf, they'd go here
# COPY ./solr-config/ /tmp/solr-config
# Give the SOLR directory to root group (not root user)
# https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-origin-specific-guidelines

RUN chgrp -R 0 /opt/solr \
  && chmod -R g+rwX /opt/solr

RUN chgrp -R 0 /opt/docker-solr \
  && chmod -R g+rwX /opt/docker-solr

USER 8983

Build Dockerimage

#!/bin/bash
SCRIPT_DIR=$(dirname $0)

docker build -t 'alacambra/openshift-solr:8.2.0' -f ${SCRIPT_DIR}/Dockerfile ${SCRIPT_DIR}

Import ImageStream

oc import-image s2i-solr --from="alacambra/openshift-solr:8.2.0" --confirm

Create App

create new app with solr config files: lacambra.tech/blog-build/s2i-solr/solr
oc new-app s2i-solr:8.2.0~/{path/to/s2i-solr/solr/} --name=solr-blog

Reabuild app

rebuild app with solr config files: lacambra.tech/blog-build/s2i-solr/solr
oc start-build --from-dir . solr-blog

After update docker file or s2i scripts

./build.sh && \
docker push alacambra/openshift-solr:8.2.0 && \
oc import-image s2i-solr --from="alacambra/openshift-solr:8.2.0" --confirm && \
oc start-build --from-dir ./solr/ solr-blog

Send a query request

Response body:

{
    "responseHeader":{
        "status":0,
        "QTime":34,
        "params":{
            "q":"a"
        }
    },
    "response":{
        "numFound":0,
        "start":0,
        "docs":[
        ]
    }
}

Run docker commands on all or some containers

execute a docker command over containers matchin some given expression

Albert Lacambra BasilAlbert Lacambra Basil

Use the ps command with the q param to fetch the insance id and then apply the command:

docker {COMMAND TO RUN} $(docker ps -a -q)  -> for containers
docker {COMMAND TO RUN} $(docker images -q) -> for images

Examples:

Delete all stopped containers
docker rm $(docker ps -a -q)
Delete all unused images
docker rmi $(docker images -q)
Start all stopped containers:
docker start $(docker ps -a -q)
Restart all containers:
docker restart $(docker ps -a -q)
Print names of containers. This commands avoids column name using NR>1 and prints the last column. So it uses the fact that las column is the one we are interested.
docker ps | awk "{if(NR>1) print $NF}"
Reformat ps output format. Using format modifier we directly print only desired columns separating them using a given separetor (\t on this case)
docker ps –format "table {{.Names}}\t{{.Image}}"
Print only containers names without headers using format and tail. Print only column names using format modifier and avoid column names using tail command to print from to line 2
docker ps -a –format "table {{.Names}}" | tail -n +2
For more elavorated filters. awk "{ print $1 }" captures the container id value.
docker  {COMMAND TO RUN} $(docker ps -a |grep {TEXT-TO-CAPTURE}|awk "{ print $1 }")
Remove all images with "pattern" in name
docker rmi $(docker images -f="reference=*pattern*" -q)

ThinWar with Widlfy

Create thin wars with wildfly. Ship dependencies with the application servers.

Albert Lacambra BasilAlbert Lacambra Basil

I have heard the concept of ThinWars several years before in one of the airhacks workshop from Adam Bien.

Basically, the idea is that in any project packaged into a war should contain only business logic. No jar dependencies should be packaged with the business code. That reduces drastically the size of the final war file, improving build, package, delivery and start-up times.

So, to be able to remove all dependencies from our war artifact, we need to change the scope from compiled to provided, and then add those dependencies directly to our application server. In a naive first approach, I have tried just to add those dependencies to the java classpath. However, the wildfly’s class loading is much more complex than that.

To add new dependencies into the Widlfly application server, we need to add them creating a new module. That means that we must create a new module.xml file, declaring included jars and dependencies.

So, to create a new module we must follow the next steps.

  • Create the module directory:

mkdir -p $JBOSS_HOME/modules/my/module/name/main
  • Create a module.xml file with including our depenedencies. Required jars can be directly downloaded form https://mvnrepository.com

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="my.module.name">
   <resources>
      <resource-root path="{my-dependency-jar1.jar}" />
      <resource-root path="{my-dependency-jar2.jar}" />
   </resources>
</module>
  • Declare the module as a global module into settings.xml so that it can be used by any deployed application. Under the <subsystem> section add:

<global-modules>
    <modulename="my.module.name"slot="main"/>
</global-modules>
Albert Lacambra BasilAlbert Lacambra Basil

Since Java API for JSON Processing (JSR 374) version 1.1, it is possible to use JosnPointer.

JsonPointer is a specification of rfc6901 and as we can read on it, JSON Pointer defines a string syntax for identifying a specific value within a JavaScript Object Notation (JSON) document. In other words, it is possible now to evaluate and change values from our JsonObjects using a pointer string instead to go through the whole chain of calls and recreating an object builder at the end.

So instead of that:

String nameWithObject = jsonObject.getJsonArray("user_mentions").getJsonObject(0).getString("name");

we can do that:

String nameWithPointer = ((JsonString)Json.createPointer("/user_mentions/

We can easily see, that the use of pointers make easier to know which element we are fetching and more intuitive to write. However, since the pointer is returning a JsonValue, we need to use a cast to be able to fetch the final value.

Why JsonPointer is not providing methods to directly get java types like JsonObject is doing, is something I do not really know.

So, what can we do with the JSON pointer?

We can not only get values from a JsonStructure using pointer notation but also modify the object without the need to reconvert it into its builder equivalent. So specifically we can:

  • add a value to a JsonStructure

  • check if a value is contained into a JsonStructure

  • remove a value from a jsonStructure

  • replace a value into a JsonStructre

Let’s see some examples. For the examples, I will use the the following json object:

{
    "id":1,
    "user":{
        "name":"some-name",
        "lastname":"some-lastname"
    },
    "user_mentions":[
        {
            "name":"Twitter API",
            "indices":[
                4,
                15
            ],
            "screen_name":"twitterapi",
            "id":6253282,
            "id_str":"6253282"
        }
    ]
}
Get a simple value from an object:
JsonNumber id = ((JsonNumber) Json.createPointer("/id").getValue(example));
Get an object from an object.
JsonObject user = Json.createPointer("/user").getValue(example).asJsonObject();
get an array from an object
JsonArray userMentions = Json.createPointer("/user_mentions").getValue(example).asJsonArray();
get an element from an array
JsonObject mention = Json.createPointer("/user_mentions/0").getValue(example).asJsonObject();
String mentionName = ((JsonString) Json.createPointer("/user_mentions/0/name").getValue(example)).getString();
int mentionIndex0 = ((JsonNumber) Json.createPointer("/user_mentions/0/indices/1").getValue(example)).intValue();
check if an object contains an element
Assertions.assertTrue(Json.createPointer("/id").containsValue(example));
Assertions.assertTrue(Json.createPointer("/user_mentions/0/indices/0").containsValue(example));
Assertions.assertTrue(Json.createPointer("/user_mentions/0/indices/1").containsValue(example));
Assertions.assertFalse(Json.createPointer("/user_mentions/0/indices/2").containsValue(example));
Add a simple value
JsonObject extendedExample = Json.createPointer("/timestamp").add(example, Json.createValue(System.currentTimeMillis()));
Assertions.assertTrue(Json.createPointer("/timestamp").containsValue(extendedExample));
Add an element to a JsonArray. The pointer must point to the last_element + 1 index. Empty elements would produce an error.
extendedExample = Json.createPointer("/user_mentions/0/indices/2").add(extendedExample, Json.createValue(30));
Assertions.assertEquals(30, ((JsonNumber) Json.createPointer("/user_mentions/0/indices/2").getValue(extendedExample)).intValue());
Replace elements
example = Json.createPointer("/id").replace(example, Json.createValue(2));
Assertions.assertEquals(2, example.getInt("id"));
example = Json.createPointer("/user_mentions/0/indices/1").replace(example, Json.createValue(9999));
Assertions.assertEquals(9999, ((JsonNumber) Json.createPointer("/user_mentions/0/indices/1").getValue(example)).intValue());
Remove elements
example = Json.createPointer("/id").remove(example);
Assertions.assertFalse(example.containsKey("id"));

Source code on github

Albert Lacambra BasilAlbert Lacambra Basil

When testing and mocking, sometime we perform some assertions/verifications that are not covered by the standard Mockito or Hamcrest Matchers.

In these cases, we can create our own matchers. Your own matchers, can increase readability, perform more complex assertions or call verifications.

Let’s see an example.

In the following code we have an Article class. Our article is quite simple. It has an id, a title and a text.

public class Article {
  private int id;
  private String title;
  private String text;

  public Article(int id, String title, String text) {
    this.id = id;
    this.title = title;
    this.text = text;
  }

  public int getId() {
    return id;
  }

  public String getTitle() {
    return title;
  }

  public String getText() {
    return text;
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) return true;
    if (o == null || getClass() != o.getClass()) return false;
    Article article = (Article) o;
    return id == article.id;
  }

  @Override
  public int hashCode() {
    return Objects.hash(id);
  }
}

We can store this article in a store using the field id as its primary key. We use its methods equals and hashcode to identify two instances that represents the same stored article.

public class ArticleStore {

  private Map<Integer, Article> articles;
  private DuplicationService duplicationService;

  public ArticleStore(DuplicationService duplicationService) {
    this.duplicationService = duplicationService;
    articles = new HashMap<>();
  }

  public Article createArticle(String text, String title) {
    Article article = new Article(getNextId(), text, title);
    storeArticle(article);
    return article;
  }

  public Article createCopy(int id) {
    Article article = articles.get(id);
    Article copy = duplicationService.duplicate(getNextId(), article);
    storeArticle(copy);

    return copy;
  }

  public List<Article> getArticlesLike(Article article) {
    return articles.values()
        .stream()
        .filter(a -> a.getText().equalsIgnoreCase(article.getText()) && a.getTitle().equalsIgnoreCase(article.getTitle()))
        .collect(Collectors.toList());
  }

  void storeArticle(Article article) {
    articles.put(article.getId(), article);
  }

  private Integer getNextId() {
    return articles.size() + 1;
  }
}

We can test that an article has been created checking that the storeArticle method has been called with the desired article.

@Test
void createArticle() {
  ArticleStore articleStore = Mockito.spy(ArticleStore.class);
  articleStore.createArticle("someText", "title");

  Article article = new Article(1, "someText", "title");

  Mockito.verify(articleStore).storeArticle(Mockito.eq(article));

  //Or simply
  Mockito.verify(articleStore).storeArticle(article);
}

Now, we have a service that duplicates this articles. That means that the duplicate service will create a new article that contains a different id but the same title and text.

package tech.lacambla.blog.examples.matchers;

public class DuplicationService {

  public Article duplicate(int nextId, Article article) {
    return new Article(nextId, article.getTitle(), article.getText());
  }

}

If we want to verify that this method is correctly called, we must create a matcher, ArgumentMatcher.isDuplicateOf(Article article).

class ArticleMatcher implements ArgumentMatcher<Article> {


  public final Article article;


  public static Article isDuplicateOf(Article article) {

   // Register our matcher.

    mockingProgress().getArgumentMatcherStorage().reportMatcher(new ArticleMatcher(article));
    return null;
  }

  public ArticleMatcher(Article article) {
    this.article = article;
  }

  /**
   * Implements matches method with our matching logic.
   * @param article
   * @return
   */
  @Override
  public boolean matches(Article article) {
    return this.article.getText().equalsIgnoreCase(article.getText());
  }

  public String toString() {
    return "<ArticleMatcher>";
  }
}

Now we can use our ArgumentMatcher to create stubs and verify calls:

@Test
  void duplicateArticle() {

    Article article = articleStore.createArticle("someText", "title");
    articleStore.createCopy(article.getId());

    //2 times since the both articles have the same contents
    Mockito.verify(articleStore, times(2)).storeArticle(ArticleMatcher.isDuplicateOf(article));
  }