Friday, 1 March 2019

AWS, Spring, Localstack

Using the AWS Java client is very straightforward.  Unit testing is also quite simple by just mocking the AWS classes.  However, integration testing is more complicated.  Here is an example of using LocalStack, TestContainers and Spring to wire AWS objects to point to the LocalStack instance.

Localstack: An implementation of AWS which runs locally with natively or in a docker container
TestContainers: A java library that lets a docker container be run locally for testing

Here TestContainers is used to start the localstack docker image so that the AWS calls can be made against it.


Maven dependencies

Using the v2 dependencies for the AWS library requires bringing in the AWS bom (bill of materials) so that any dependency can just be declared and the bom takes care of getting the correct versions of each dependency.  In the example below the S3 and SQS dependencies are configured.

Dependency management & dependencies

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>bom</artifactId>
        <version>2.4.11</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
    </dependency>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>sqs</artifactId>
    </dependency>

Test dependencies:

    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers</artifactId>
      <version>1.10.6</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>localstack</artifactId>
      <version>1.10.6</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>cloud.localstack</groupId>
      <artifactId>localstack-utils</artifactId>
      <version>0.1.18</version>
      <scope>test</scope>
    </dependency>


AWS Configuration

The normal spring configuration for aws clients is very straight forward.  Here is an example of an S3Client and an SqsClient using the AWS v2 objects.

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.sqs.SqsClient;

@Configuration
public class AwsConfiguration {

  @Bean
  public S3Client s3Client(){
    return S3Client.builder().region(Region.EU_WEST_1).build();
  }

  @Bean
  public SqsClient sqsClient(){
    return SqsClient.builder().region(Region.EU_WEST_1).build();
  }
}

These objects will use the default AwsCredentialProvider but this can be overridden here.


TestConfiguration

To create the test configuration we need to start Localstack using TestContainers.  This test configuration starts the Localstack and then uses it to configure the S3Client and SqsClient to point to localstack

import static org.testcontainers.containers.localstack.LocalStackContainer.Service.S3;
import static org.testcontainers.containers.localstack.LocalStackContainer.Service.SQS;

import java.net.URI;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.testcontainers.containers.localstack.LocalStackContainer;
import org.testcontainers.containers.wait.strategy.DockerHealthcheckWaitStrategy;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
import software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider;
import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.CreateQueueRequest;

@TestConfiguration
public class AwsConfigurationTest {

  @Bean
  public LocalStackContainer localStackContainer() {
    LocalStackContainer localStackContainer = new LocalStackContainer().withServices(SQS, S3);
    localStackContainer.start();
    return localStackContainer;
  }

  @Bean
  public S3Client s3Client() {

    final S3Client client = S3Client.builder()
        .endpointOverride(URI.create(localStackContainer().getEndpointConfiguration(S3).getServiceEndpoint()))
        .build();

    client.createBucket(CreateBucketRequest.builder().bucket("test_bucket").build());

    return client;
  }

  @Bean
  public SqsClient sqsClient() {

    final SqsClient sqs = SqsClient.builder()
        .endpointOverride(URI.create(localStackContainer().getEndpointConfiguration(SQS).getServiceEndpoint()))
        .build();

    sqs.createQueue(CreateQueueRequest.builder().queueName("test_queue").build());

    return sqs;
  }
}




Monday, 26 November 2018

Enzyme Cheat Sheet

Here is a reminder of the enzyme tips I've picked up so I don't forget!

Basics

Mount a react component in a test

import React from 'react'
import { mount } from 'enzyme'

describe('MyReactComponent', () => {
  const $ = mount(
     <MyReactComponent />
  )

  ...
})

Now different expectations and finds can be used to test,

A React Class
expect($.find('MyReactClass').exists()).toBe(false)

An html element
expect($.find('button').exists()).toBe(false)

An html id
expect($.find('#my-id').exists()).toBe(false)

A style
expect($.find('.some-style').exists()).toBe(false)

Html properties
expect($.find('[aria-label="Save"]').exists()).toBe(false)
expect($.find('[title="Save"]').exists()).toBe(false)

Combination - Html element & Property
expect($.find('button[title="Save"]').exists()).toBe(false)

HTML / Text / Contains

Text content.  Find the first of many 'myClass' css classes and check the text of the element
expect($.find('.myclass').text()).toContain("My Value")
expect($.find('.myclass
').text()).not.toContain("My Value")

As above but with an exact text match
expect($.find('.myclass
').text()).toBe("The exact text")
As above but getting the whole html of the element rather than just the text
expect($.find('.myclass').at(0).html()).toBe("<div>something</div>")

Focus

Focus a field with enzyme
const inputField = $.find('#myInput')
inputField.getDOMNode().focus()

Validate that a field has focus
expect(document.activeElement.id).toBe('the id expected')

Props

Find an element and get the html properties
const button = $.find('[aria-label="My Button"]')
expect(button.props().disabled).not.toBe(true)

Once an element has been found the react props can be used for expectations by also using props()
expect($.find('MyReactClass').props().options).toEqual([{option: 'option1'}, {option: 'option2'}])






Monday, 12 November 2018

Formik

Formik is a brilliant React form builder.  It is simple and intuitive.  It includes validation using a third party - I'm using yup below.

Basic Formik

This example below shows the basics of using Formik.  The internal properties and functions of formik take care of all the plumbing.  All you need to do is wire the onChange and onBlur functions to the individual 

<div>
  <Formik
    initialValues={{
      title: '',
      firstName: '',
      surname: ''
    }}
    validationSchema={Yup.object().shape({
      title: Yup.string()
        .trim()
        .required('Please enter a title')
        .max(5, 'Too many characters (Maximum 5 allowed)'),
      firstName: Yup.string()
        .trim()
        .required('Please enter a firstName')
        .max(100, 'Too many characters (Maximum 100 allowed)'),
      surname: Yup.string()
        .trim()
        .required('Please enter a surname')
        .max(100, 'Too many characters (Maximum 100 allowed)'),
      
    })}
    onSubmit={values => alert(values)}
  >
    {formikProps => {
      const {
        values,
        touched,
        errors,
        isSubmitting,
        handleChange,
        handleBlur,
        handleSubmit,
        setFieldValue,
        setFieldTouched
      } = formikProps
      return (
        <form>
          <div>
            <label htmlFor="title">
              <span className="mandatory">Title</span>
              <input
                id="title"
                type="text"
                value={values.title}
                onChange={handleChange}
                onBlur={handleBlur}
                className={`${errors.title && touched.title ? error : ''}`}
              />      
              {errors.title && touched.title && <div>{errors.title}</div>}
            </label>
          </div>

          <div>
            <label htmlFor="firstName">
              <span className="mandatory">First Name</span>
              <input
                id="firstName"
                type="text"
                value={values.firstName}
                onChange={handleChange}
                onBlur={handleBlur}
                className={`${errors.firstName && touched.firstName ? error : ''}`}
              />      
              {errors.firstName && touched.firstName && <div>{errors.firstName}</div>}
            </label>
          </div>

          <div>
            <label htmlFor="surname">
              <span className="mandatory">Surname</span>
              <input
                id="surname"
                type="text"
                value={values.surname}
                onChange={handleChange}
                onBlur={handleBlur}
                className={`${errors.surname && touched.surname ? error : ''}`}
              />      
              {errors.surname && touched.surname && <div>{errors.surname}</div>}
            </label>
          </div>
        </form>
      )
    }}
  </Formik>
</div>


Validate on Edit

By default Formik will validate once the first blur has occurred and then everytime on change.  This gives the user the chance to get the content right first without being bothered by error messages.  However, should you want to validate all changes from the off you can change your inputs to include an onInput function,


            <label htmlFor="surname">

              <span className="mandatory">Surname</span>
              <input
                id="surname"
                type="text"
                value={values.surname}

                onInput={() => setFieldTouched('surname', true)} // Validate as the user types
                onChange={handleChange}
                onBlur={handleBlur}
                className={`${errors.surname && touched.surname ? error : ''}`}
              />      
              {errors.surname && touched.surname && <div>{errors.surname}</div>}
            </label>


Validate Function

Validation doesn't have to be done by a schema.  This is quite limiting as it doesn't allow dynamic messages to be generated.  You can still use a schema as this is really easy syntax but just recreate it each time.  If you use the validate function the Formik 'values' object is passed to it.  This can be used to generate a schema with the current values and adjust the error messages accordingly.  In the example below the validate function calls for a validationSchema.  The schema uses the values and the ` string template indicator to create dynamic error messaages.

getValidateSchema = values => 
  Yup.object().shape({
    title: Yup.string()
      .trim()
      .required('Please enter a title')
      .max(5, `Too many characters (${values.title.length} entered, Maximum 5 allowed)`),
    firstName: Yup.string()
      .trim()
      .required('Please enter a firstName')
      .max(100, `Too many characters (${values.firstName.length} entered, Maximum 100 allowed)`),
    surname: Yup.string()
      .trim()
      .required('Please enter a surname')
      .max(100, `Too many characters (${values.surname.length} entered, Maximum 100 allowed)`),
  })

validate = values => {
  try {
    validateYupSchema(values, this.getValidateSchema(values), true, {})
    return {}
  } catch (error) {
    return yupToFormErrors(error)
  }
}

render = () => 
  <div>
    <Formik
      initialValues={{
        title: '',
        firstName: '',
        surname: ''
      }}
      validate={values => this.validate(values)}
      onSubmit={values => alert(values)}
    >
      {formikProps => {
        const {
          values,
          touched,
          errors,
          isSubmitting,
          handleChange,
          handleBlur,
          handleSubmit,
          setFieldValue,
          setFieldTouched
        } = formikProps
        return (
          <form>
            <div>
              <label htmlFor="title">
                <span className="mandatory">Title</span>
                <input
                  id="title"
                  type="text"
                  value={values.title}
                  onInput={() => setFieldTouched('title', true)}
                  onChange={handleChange}
                  onBlur={handleBlur}
                  className={`${errors.title && touched.title ? error : ''}`}
                />      
                {errors.title && touched.title && <div>{errors.title}</div>}
              </label>
            </div>

            <div>
              <label htmlFor="firstName">
                <span className="mandatory">First Name</span>
                <input
                  id="firstName"
                  type="text"
                  value={values.firstName}
                  onInput={() => setFieldTouched('firstName', true)}
                  onChange={handleChange}
                  onBlur={handleBlur}
                  className={`${errors.firstName && touched.firstName ? error : ''}`}
                />      
                {errors.firstName && touched.firstName && <div>{errors.firstName}</div>}
              </label>
            </div>

            <div>
              <label htmlFor="surname">
                <span className="mandatory">Surname</span>
                <input
                  id="surname"
                  type="text"
                  value={values.surname}
                  onInput={() => setFieldTouched('surname', true)}
                  onChange={handleChange}
                  onBlur={handleBlur}
                  className={`${errors.surname && touched.surname ? error : ''}`}
                />      
                {errors.surname && touched.surname && <div>{errors.surname}</div>}
              </label>
            </div>
          </form>
        )
      }}
    </Formik>
  </div>

Wednesday, 26 September 2018

JSON Date / Time Formatting

The default behaviour when serialising json from a Java Date object is to break it down to its constituent parts eg,

"startDate" : {
    "year" : 2014,
    "month" : "MARCH",
    "dayOfMonth" : 1,
    "dayOfWeek" : "FRIDAY",
    "dayOfYear" : 1,
    "monthValue" : 1,
    "hour" : 2,
    "minute" : 2,
    "second" : 0,
    "nano" : 0,
    "chronology" : {
      "id" : "ISO",
      "calendarType" : "iso8601"
    }
  }

Spring Boot 2

Spring boot already contains the dependencies,

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.datatype</groupId>
        <artifactId>jackson-datatype-jsr310 </artifactId>
    </dependency>

and it'll wire the jsr310 dependency automatically for any ObjectMapper that is @Autowired.  All that is necessary then is to add the property,

    spring.jackson.serialization.WRITE_DATES_AS_TIMESTAMPS=false

Standalone

For standalone code the dependencies above will need to be added.  In the code when the object mapper is created the following is necessary,

    final ObjectMapper objectMapper = new ObjectMapper();
    objectMapper.enable(SerializationFeature.INDENT_OUTPUT);
    objectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
    objectMapper.registerModule(new JavaTimeModule());


Friday, 17 August 2018

PACTs

PACT testing is a Consumer Driven Contract implementation.

Here is a quick example for a Consumer test and a Provider test.

Consumer

This is some sample code for the consumer.  This will create a 'pact' json file in the target folder after a maven build.  This json is what will get sent to the Pact Broker or can be used to validate the pacts locally instead.
 
public class MyPactConsumerTest {  

  @Rule
  public PactProviderRuleMk2 pactProviderRule = new PactProviderRuleMk2("<NAME_OF_PROVIDER>", "localhost", PACT_SERVER_PORT, PactSpecVersion.V2, this);

  @Pact(consumer = "<NAME_OF_CONSUMER>", provider = "<NAME_OF_PROVIDER")
  public RequestResponsePact doSomething(final PactDslWithProvider pactDslWithProvider) {

    return pactDslWithProvider.given("<STATE>")
      .uponReceiving("<DESCRIPTION")
      .path("<URL>")
      .method("POST")
      .body(getExpectedRequestBody())
      .headers("Content-Type", "<TYPE>")
      .willRespondWith()
      .status(HttpStatus.NO_CONTENT.value())
      .toPact();
  }

  @Test
  @PactVerification(fragment = "doSomething") <THIS IS THE METHOD NAME ABOVE>
  public void testDoSomething() {
    someJavaClassToCall.someMethod();
  }

  private DslPart getExpectedResponseBody() {
    return new PactDslJsonBody().stringMatcher("dataAttribute", ".*", "");
  }
}

Provider

The provider can then validate the pact json created by the consumer above.  If the Pacts are published then it is necessary to validate against the Broker with the @PactBroker annotation.  If you want to just validate against a pact json file then copy the file into a folder in /src/test/resources/pact (or something) and just add the @PactFolder("pact") annotation instead.


@Provider("<NAME_OF_PROVIDER>")
@PactBroker
@IgnoreNoPactsToVerify
//@PactFolder("pact") <A FOLDER CAN BE USED INSTEAD OF THE BROKER>
public class MyPactProviderTest {

  @State("<STATE>")
  public void methodName() {
    when(mockService.mockMethod(any())).thenReturn(new Data()); <THIS RETURNS DATA THAT MATCHES THE RESPONSE BODY ABOVE>
  }

}

Friday, 16 March 2018

sed Crib Sheet

sed (stream editor) is a linux command that can manipulate files.  It is particularly useful for doing match and replace on a whole file. Below is the linux syntax

Change from 'something' to 'something else

sample.txt

hello out
there
how are
you today


Simple single line match
> sed -i s/hello/goodbye/ sample.txt
goodbye out
there
how are
you today


Multiline replace.  Match on 'out'.  N adds the next line to the 'match space' and then there is a normal substitution on the 'match space'.
> sed -i "/out/{N;s/there/fred/}" sample.txt
hello out
fred
how are
you today

Thursday, 8 March 2018

Kubernetes / kubectl Crib Sheet


Get and Describe

Generally the get and describe can be used for 'deployments', 'nodes', 'pods', 'services', 'secrets' etc

// Get a list of pods
> kubectl get pods

// Interact with the pod by getting a shell
> kubectl exec -it <pod-nane> /bin/bash

// Get a list of services
> kubectl get services

// Describe a service to see how to connect to it
> kubectl describe service <service-name>

Contexts

// Allow kubectl to see multiple config files - this isn't really a merge in that the files stay separate
> export KUBECONFIG=config:config-other:config-different

// List the Contexts (* = current)
> kubectl config get-contexts

// View all the config details
> kubectl config view

// Use a particular contextsa
> kubectl config use-context <context-name>

Secrets

Create a secret
> kubectl create secret generic ssh-key-secret --from-file=id_rsa=~/.ssh/id_rsa --from-file=id_rsa.pub=~/.ssh/id_rsa.pub