Author Archives: purpleblob

Let’s create a Vert.x docker image of our application

As you’ve probably realised from previous posts, I’m very much into the idea of deploying my server applications via Docker, so let’s take the VertxTest application and create a Docker image with it.

Thankfully we do not need to start our Dockerfile by creating a base OS then adding Java etc. The people writing Vert.x have kindly implemented a base image for us.

See Deploying a Java verticle in a docker container.

If we take the example Dockerfile from their website and make the changes for our application (just replace VERTICAL_NAME and VERTICLE_FILE with our main verticle and the JAR) we get the following file

FROM vertx/vertx3

ENV VERTICLE_NAME com.putridparrot.MainVerticle
ENV VERTICLE_FILE VertxTest-1.0-SNAPSHOT-fat.jar

ENV VERTICLE_HOME .

EXPOSE 8080

COPY $VERTICLE_FILE $VERTICLE_HOME/

WORKDIR $VERTICLE_HOME
ENTRYPOINT ["sh", "-c"]
CMD ["exec vertx run $VERTICLE_NAME -cp $VERTICLE_HOME/*"]

Place this Dockerfile into a folder along with the compiled JAR and then execute

docker build -t vertx_test  .

This creates an image name vertx_test which we can now execute. If you want to watch the output from our System.out.println execute

docker run -it -p 8080:8080 vertx_test

Once you see the HTTP server started on port 8080 outputs, the Vert.x application will be ready for you to access via your preferred web browser etc.

Vert.x event bus communicating between Verticles

In my previous post on Vert.x, I demonstrated how to create multiple Verticle’s but also how a Verticle needn’t actually be a “service” but in some cases is just a worker process or in the case of the previous post a “main” or startup process.

This leads us to better see how Vert.x is actor-like with regards to Verticles.

In situations where we have multiple Verticles and possible some of those Verticles are simply workers for the other Verticles, we would need some way of communicating across those different parts of the application. Hence Vert.x also comes with an event bus for communicating across Verticles. This is a global event bus across all our Verticles.

Let’s, for sake of simplicity change our previous example so that whenever a service is successfully routed to, an event is sent out of the EventBus to say which services was called.

So, taken the example from my previous post, add the following to the MainVerticle start method

EventBus eventBus = vertx.eventBus();

eventBus.consumer("/svc", m -> {
   System.out.println(m.body());
});

The String /svc is the topic or an address if you prefer that we publish and consume message from. In the example above, we’re listening to the event bus for the anything sent (or published) to the address /svc. How you name you addresses is totally upto you.

Don’t forget to import io.vertx.core.eventbus.EventBus to use the event bus.

Next we’ll add code to the services to send a message on the /svc address when we wish to communicate something. Here’s the code for the HelloVerticle start method with the event bus added.

@Override
public void start() {

   EventBus evBus = vertx.eventBus();

   router.route("/hello").handler(ctx -> {
      evBus.send("/svc", "hello called");

      ctx.response()
         .putHeader("content-type", "text/plain")
         .end("Hello " + ctx.queryParam("name"));
      });

      vertx.createHttpServer()
         .requestHandler(router::accept)
         .listen(8080);

   System.out.println("HTTP server started on port 8080");
}

As you can see from the highlighted lines adding an event bus and sending messages out on it, is very simple. If we updated the WorldVerticle from the previous post also (but sending “world called”) then when run, the MainVerticle will print a line to the console every time the HelloVerticle or WorldVerticle routes are called.

Publish and Send

In the example (above) I used the event buss method send to send messages out on the event bus, but send will only be delivered to at most one of the handlers of the message.

Therefore is we added the following code to each of our service Verticles

evBus.consumer("/svc", m -> {
   System.out.println("Hello " + m.body());
});

After the declaration of the evBus variable.

we’ve now got a requirement that the message on address /svc needs to be handled by multiple consumers. Running up the application as it currently stands would result in only one of the consumers receiving the messages. In my case this was the MainVerticle, however there’s no guarantee which Verticle would receive the messages as this is determined by Vert.x in a round robin fashion.

So here’s the difference between send and publish. If you change the code from send to publish in the service Verticles. Then messages will be routed to all consumers.

Multiple Verticles

In my previous post I looked at creating my first Verticle/Vert.x application. This had a single Verticle acting as a service and running up a server against port 8080. If we therefore assume that by default a Verticle represents a single service then to create multiple service we simply create multi-Verticles.

Let’s create three Verticle’s. The first will act as the “main” verticle (or run as an application) and it will register/deploy the Verticle’s that we’re using to host our two services, the HelloVerticle and WorldVerticle.

Each service will itself create/host an HttpServer. Traditionally this would tend to mean we’d have two ports exposed, one for each service, but Vert.x allows us to create multiple HttpServer’s on the same port and uses something like “round robin” to try to locate a valid route.

Check my previous post for the required pom.xml.

Here’s MainVerticle.java

package com.putridparrot;

import io.vertx.core.AbstractVerticle;
import io.vertx.ext.web.Router;

public class MainVerticle extends AbstractVerticle {
    @Override
    public void start() {
        Router router  = Router.router(vertx);

        vertx.deployVerticle(new HelloVerticle(router));
        vertx.deployVerticle(new WorldVerticle(router));
    }
}

Notice that this Verticle’s job is to both register the other Verticle’s and acts as the main entry point to our application.

You do not need to pass the Router around in this way unless we intend to have multiple routes on the same port. So in this case we’re creating the router that will be used by both services on the same port.

Let’s look at the HelloVerticle.java service (both HelloVerticle and WorldVerticle, in this simple example, are basically the same, but I’ll reproduce all code here anyway).

package com.putridparrot;

import io.vertx.core.AbstractVerticle;
import io.vertx.ext.web.Router;

public class HelloVerticle extends AbstractVerticle {

    private Router router;

    public HelloVerticle(Router router) {
        this.router = router;
    }

    @Override
    public void start() {
        router.route("/hello").handler(ctx -> {
            ctx.response()
                    .putHeader("content-type", "text/plain")
                    .end("Hello " + ctx.queryParam("name"));
        });

        vertx.createHttpServer()
                .requestHandler(router::accept)
                .listen(8080);

        System.out.println("HTTP Hello server started on port 8080");
    }
}

There’s nothing particularly special or different here, compared to my previous posts code. So let’s look at the WorldVerticle.java

package com.putridparrot;

import io.vertx.core.AbstractVerticle;
import io.vertx.ext.web.Router;

public class WorldVerticle extends AbstractVerticle {

    private Router router;

    public WorldVerticle(Router router) {
        this.router = router;
    }

    @Override
    public void start() {
        router.route("/world").handler(ctx -> {
            ctx.response()
                    .putHeader("content-type", "text/plain")
                    .end("World " + ctx.queryParam("name"));
        });

        vertx.createHttpServer()
                .requestHandler(router::accept)
                .listen(8080);

        System.out.println("HTTP World server started on port 8080");
    }
}

What’s interesting is that both HelloVerticle and WorldVerticle create the HTTP server on the same port. As stated earlier Vert.x simply handles this for us without a port conflict and because we’re using a single router across these two Verticle’s Vert.x can correctly located the service for each of the following URL’s

http://localhost:8080/hello?name=Hello
http://localhost:8080/world?name=World

First look at Eclipse Vert.x

Eclipse Vert.x is a lightweight framework for developing web applications, microservices and more.

I’ve created a project using IntelliJ named VertxTest based upon a Maven project to try out the routing functionality in Vert.x.

Dependencies – the pom

Our pom.xml looks like this

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>VertxTest</groupId>
    <artifactId>VertxTest</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <java.version>1.8</java.version>
        <vertx.version>3.5.0</vertx.version>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <manifestEntries>
                                        <Main-Class>io.vertx.core.Launcher</Main-Class>
                                        <Main-Verticle>com.putridparrot.MainVerticle</Main-Verticle>
                                    </manifestEntries>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/services/io.vertx.core.spi.VerticleFactory</resource>
                                </transformer>
                            </transformers>
                            <artifactSet>
                            </artifactSet>
                            <outputFile>${project.build.directory}/${project.artifactId}-${project.version}-fat.jar</outputFile>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.5.0</version>
                <configuration>
                    <mainClass>io.vertx.core.Launcher</mainClass>
                    <arguments>
                        <argument>run</argument>
                        <argument>com.putridparrot.MainVerticle</argument>
                    </arguments>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-core</artifactId>
            <version>${vertx.version}</version>
        </dependency>
        <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-unit</artifactId>
            <version>${vertx.version}</version>
        </dependency>
        <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-web</artifactId>
            <version>${vertx.version}</version>
        </dependency>
    </dependencies>
</project>

The Verticle

As you can see from this my class MainVerticle (which extends AbstractVerticle) is created in the package com.putridparrot and the code looks like this

package com.putridparrot;

import io.vertx.core.AbstractVerticle;
import io.vertx.ext.web.Router;
import io.vertx.ext.web.handler.BodyHandler;

public class MainVerticle extends AbstractVerticle {
    @Override
    public void start() {

        Router router = Router.router(vertx);

        router.route("/hello").handler(ctx -> {
            ctx.response()
                .putHeader("content-type", "text/plain")
                .end("Hello " + ctx.queryParam("name"));
        });

        vertx.createHttpServer()
            .requestHandler(router::accept)
            .listen(8080);

        System.out.println("HTTP server started on port 8080");
    }
}

Note: queryParam actually returns a List so in this example we would probably want to actually check the result and then get items from the list.

We extend AbstractVerticle overriding the start method to handle the setting up of our routes etc.

Building and Running

We execute mvn clean install and then mvn package to generate the executable jar. Finally we can run this using java -jar target/VertxTest-1.0-SNAPSHOT-fat.jar or similar in a configuration in an IDE (for example a JAR Application configuration within Intelli J just requires the path to the JAR listed above).

Testing and more

Now we should be able to run the following from your preferred web browser or Intelli J’s REST client (or similar in another IDE).

http://localhost:8080/hello?name=Mark

We can easily add further routes and/or handle GET or POST HTTP methods. For example here’s the addition of an error route using the get method

router.get("/error")
   .handler(this::handleGetError);

and

private void handleGetError(RoutingContext ctx) {
   ctx.response()
      .setStatusCode(500)
      .end();
}

Now browsing to

http://localhost:8080/error

will display a 500 error.

Static pages

We can also add static pages by declaring a route such as

 router.route("/pages/*")
   .handler((StaticHandler.create("pages")));

where the pages directory is located off the root off of the project folder (in this example). Simply place your HTML pages in this folder and you can access them using the following URL

http://localhost:8080/pages/index.html

Routing based upon mime type

Along with routine based upon the specific query/resource requested we can also route based upon the Content-Type within a request. For example if using HTTP POST with Content-Type application/json we would want to return JSON, likewise if Content-Type is application/xml we’d expect XML return (and so on).

To route based upon the mime type we simply write the following

router.route()
   .consumes("*/json")
   .handler(ctx -> {
      System.out.println("JSON request");
});

router.route()
   .consumes("*/xml")
   .handler(ctx -> {
      System.out.println("XML request");
});

Obviously in the examples above we’re simply routing off of the root “/” URL. Supply a path within the route method, as per previous example for other routing scenarios.

Running our Vertx application

We’ve seen we can create and run the JAR, but we can also create a Vertx application by creating a main method that looks something like this

public static void main(String[] args) {
   Vertx vertx = Vertx.vertx();

   vertx.deployVerticle(new MainVerticle());
}

We use deployVerticle along with our Verticle and Vertx does the rest.

Creating an ansible dockerfile

I was trying to run the ansible docker image on Docker hub, but it kept failing, so I went through the steps listed in http://docs.ansible.com/ansible/latest/intro_installation.html inside of an Ubunto docker image

  • apt-get update
  • apt-get install software-properties-common
  • apt-add-repository ppa:ansible/ansible
  • apt-get update
  • apt-get install ansible

Then running ansible –version demonstrated everything was working.

I decided to use the same commands to create a Dockerfile and hence create my own image of ansible.

  • Create an ansible folder
  • Create the file Dockerfile in the ansible folder

Please note, this is not a complete implementation of a Docker image of ansible so much as a starting point for experimenting with ansible.

Now we’ll create a fairly simple image based on ubuntu and install all the libraries etc. as above. So the Dockerfile should have the following contents

FROM ubuntu:latest

RUN apt-get -y update
RUN apt-get -y install software-properties-common
RUN apt-add-repository ppa:ansible/ansible
RUN apt-get -y update
RUN apt-get -y install ansible

From the ansible folder run sudo docker build -t ansible ., if all goes well you’ll have an image name ansible created. Just run sudo docker images to see it listed.

Now run sudo docker run -it ansible to run the image in interactive mode and then within the container run ansible –version to see if everything worked.

Now we’re up and running there’s obviously more to do to really use ansible and hopefully I’ll cover some of those topics in subsequent posts on ansible.

Sorting a range on specific column of data using VBA in Excel

In my previous post I imported a CSV file into my spreadsheet and displayed the rows after my headings row. I now want to sort the rows by a column of data, but I do not want the heading to be part of the sort, so I basically want to sort the rows based upon the column starting at the row below the column headings – I hope that makes sense.

Anyway, we’re going to need to find all the rows and columns that make our range of data and then select a slice of the data which are our value items to sort. Here’s the code

rows = ws.Cells(Rows.Count, "A").End(xlUp).Row
columns = ws.Cells(3, Columns.Count).End(xlToLeft).Column
    
Set dataRange = ws.Range(Cells(3, 1), Cells(rows, columns))
Set sortColumn = ws.Range(Cells(3, 5), Cells(rows, 5))
dataRange.Sort key1:=sortColumn, order1:=xlDescending

In this example we set rows to be the row count and columns to be the number of columns starting at row 3 (where our data is imported to). Next we turn these values into a range object (mine’s named dataRange) again using row 3 as our data’s starting point. Next we create a sortColumn range to be the actual column we want to use as our sort key, in this case I’m using column 5 data. Finally we sort the first data range using the second and mine sort is descending in this example.

Loading an CSV file into a workseet in Excel

Let’s assume we have an Excel worksheet with some nicely formatted headers and just above the row containing the headers we’ve placed a button which will we use to load a csv file into the rows below header row (without losing our formatting etc.)

Go to the Developer tab in Excel and select Visual Basic then create a subroutine like this

Sub LoadFile()
Dim fileName As String
   
   fileName = "c:\myfile.csv"

   Set ws = ActiveWorkbook.Sheets("Import")

   ws.Rows("3:" & Rows.Count).ClearContents
    
   With ws.QueryTables.Add(Connection:="TEXT;" & fileName, Destination:=ws.Range("A3"))
        .TextFileParseType = xlDelimited
        .TextFileCommaDelimiter = True
        .AdjustColumnWidth = False
        .Refresh
    End With
    
End Sub

I’ve simplified the code to have an explicit filename, instead of getting the filename via a folder selection dialog box or from a cell on the spreadsheet.

Simply get the active worksheet (mine’s named Import) and from this we’ll clear all row’s below our heading row (my rows start as row 3).

Next we use the QueryTables object and add the “Connection” supplying the filename that is to be imported. The destination is the start column/row where we want to place our data.

AdjustColumnWidth = False is useful to stop the import resizing your columns, the Refresh procedure will cause the spreadsheet to redraw itself.

Now this will load the CSV into the area below the column headings and not resize the columns to match the data size (hence not ruin existing formatting).

cxf-codegen-plugin not generating proxies everytime

In case this helps others out…

I am working on a Java project at the moment using Maven to build the code and run tests etc. I’m also using the cxf-codegen-plugin plugin (as below) to generate some proxies for a soap/wsdl file I have.

<plugin>
   <groupId>org.apache.cxf</groupId>
   <artifactId>cxf-codegen-plugin</artifactId>
   <version>3.2.2</version>
   <executions>
      <execution>
         <id>generate-sources</id>
         <phase>generate-sources</phase>
         <configuration>
         <wsdlOptions>
            <wsdlOption>
               <wsdl>${project.basedir}/src/main/resources/wsdl/Svc.wsdl</wsdl>
               <wsdlLocation>classpath:wsdl/Svc.wsdl</wsdlLocation>
               <extraargs>
                  <extraarg>-verbose</extraarg>
                  <extraarg>-client</extraarg>
                  <extraarg>-p</extraarg>
                  <extraarg>com.putridparrot.soap</extraarg>
               </extraargs>
            </wsdlOption>
         </wsdlOptions>
         </configuration>
         <goals>
            <goal>wsdl2java</goal>
         </goals>
      </execution>
   </executions>
</plugin>

I found that when I tried to either use mvn generate-sources mvn install the generated source folder was deleted and recreated, however the proxies folder had no proxy code/files (i.e. the files were deleted and not regenerated).

If I ran mvn clean install the proxy code would re-appear, but subsequent calls to generate-source of install would again delete the proxy code/files.

To cut a long story short it appeared that another plugin was clearing out the generated-sources folder but the cxf-codegen-plugin plugin would not regenerate it’s proxy code/files because of another file which the plugin created in the cxf-codegen-plugin-markers folder.

Within this folder is a file with a UUID style name. This was being plugin used by the plugin to check whether it needed to regenerate it’s proxy code/files (it doesn’t check if the files exist or simply regenerate them each time it’s run).

You can recreate the problem easily by running mvn generate-sources to create the proxies, then delete the generated-sources folder and run mvn generate-sources again. No proxy files are created. Now delete the marker file and the proxies will get regenerated.

Deleting this folder (or just the file within it) will fix the problem.

To stop this happening again you can either ensure the folder or file are deleted when the generate-sources folder is deleted or even simpler just write your proxies to another folder, so this would now get deleted by the other plugins.

i.e.

<configuration>
   <sourceRoot>${project.build.directory}/cxf</sourceRoot>
   <!-- other config -->
</configuration>

Running a maven plugin

Occasionally you might want to run a maven plugin without building/installing etc. at the same time – for example I am using wsdl2java to generate proxies for webservice and every now and then I want to just regenerate those proxies.

We can simply run

mvn antrun:run@your_execution_id

Embedding files and/or templates within templates using Velcocity

We can embed a file or other templates within Velocity templates – this is obviously useful for reusing snippets of code/text or whatever in more than file. We still need to create the context mappings for all the template file that form out overall template – in other words we supply the mappings the container template and all of the embedded templates.

To embed a template we simply use the #include or #parse directives. #include is used to pull in a template or file without transforming it using the template engine, i.e. any variables etc. will not be transformed or rendered by Velocity whereas #parse will transform any Velocity variables or code.

Taking the template (template.txt.vm) from my previous post, which looked like this

Hello $name,

This is a $template_name template.

we might break this template into the following two templates

salutation.txt.vm

Hello $name,

and template.txt.vm becomes

#parse("templates/salutation.txt.vm")

This is a $template_name template.

This will result in the combined and transformed template.