Using GitHub Gist to host and share your Groovy scripts

Not only does GitHub offer the best git hosting available, it also offers another very useful (and free) service that allows you to easily share snippets of code or text - GitHub Gists.

I already use Gist quite a bit to store notes, instructions, useful code snippets or scripts. In this post I'll explain how you can use gist to store a groovy script (as in Groovy Language) and use groovy to run that script locally on your machine via the gist URL for the script.

1. Getting a gister

Gist provides a nice web interface that allows you to edit your gists but there are a number of command line interfaces (CLIs) available enabling you to use the editor of your choice. The best of the CLIs I’ve used is gist, which was created by Chris Wanstrath the cofounder of GitHub.

The gist cli is a ruby gem so if you are running ruby go ahead and install the gem by:

> gem install gist

Conveniently gist is also available in Homebrew if you are running on Mac OSX:

> brew install gist

Now that gist is installed you will want to initialize it and link it to your GitHub account so that your Gists can be stored. Run the gist login command providing your GitHub username and password - N.B. gist will not store your password and only requires it this once in order to get an OAuth2 token:

> gist --login

If you have 2 factor authentication set up on your GitHub account (which you should) you will also be asked for an additional auth code that will be texted to you via SMS.

2. Groovy csv to json converter

The Gist we are going to create is a simple groovy script that takes a csv (comma separated file) and converts it into JSON format.

To create the gist use the command below:

gist csvToJSON.groovy

After the gist is created gist will print the unique id to your gist like below:

https://gist.github.com/8679353

If you need to update the gist run the command below referencing the gist to be replaced/edited:

gist csvToJSON.groovy -u 8679353

Many other options exist for gist and you can get more information from the gist --help command or on the defunkt/gist repository page.

3. Running your Groovy Gist remotely

When you are happy with your gist you can now run it remotely via the standard groovy command. First you need to get the raw URL that points to your script resource on GitHub. Simply go to your gists page, select the 'View Raw' option and copy the raw url provided. Now simply run the file as below passing in required input arguments an options as normal:

groovy https://gist.github.com/stephenhardy/8679353/raw/9c3b78f758aac3d6af8cb31b51f2edb06680999d/csvToJSON.groovy afr.csv afr.json

4. Converting csv to json

As an aside here is a little information about the script itself. We use the Jackson JSON Library and additionally the jackson-dataformat-csv extenstion in order to read a csv file containing OHLCV (Open, High, Low, Close, Volume) data, transform it into JSON format and then store it in a file (the first line of the csv is expected to contain the column headings for the csv delimited data). Jackson is a very fast JSON processor for the JVM and I'll be posting more about in in the future.

Below provides a high level overview of the main parts of the script.

a. Use Grape to grab your dependencies

@Grab(group = 'com.fasterxml.jackson.dataformat', module= 'jackson-dataformat-csv', version = '2.3.0')

b. Define the POJO to bind your csv data

I've defined a class within the groovy script called Candle that defines the attributes that will be mapped from the csv file being defined and then subsequently serialized into JSON.

Annotate each of your attributes that you wish to be JSON properties using the JsonProperty Annotation. You can define "" if the property name is the same as the JSON property to bind to.

@JsonProperty("date") String date @JsonProperty("ticker") String ticker @JsonProperty("open") String open @JsonProperty("high") String high @JsonProperty("low") String low @JsonProperty("close") String close @JsonProperty("volume") String volume

Define a constructor annotated with the JsonCreator Annotation that will be used to create your Pojo from a csv row.

@JsonCreator public Candle( @JsonProperty("date") String date, @JsonProperty("ticker") String ticker, @JsonProperty("open") String open, @JsonProperty("high") String high, @JsonProperty("low") String low, @JsonProperty("close") String close, @JsonProperty("volume") String volume) { this.date = date this.ticker = ticker this.open = open this.high = high this.low = low this.close = close this.volume = volume }

c. Read the csv file and map to Pojos

Create a CsvMapper and define a CsvSchema using the first line of the csv file (header line):

CsvMapper mapper = new CsvMapper() CsvSchema schema = CsvSchema.emptySchema().withHeader()

Read the csv file using a ObjectReader - this gives us a MappingIterator containing all our mapped Candle Pojos with data from the csv rows:

MappingIterator<Candle> it = mapper.reader(Candle).with(schema).readValues(new File(csvFile))

d. Write the Pojos as JSON to the output file

Create the ObjectWriter to serialize our Pojos to JSON:

ObjectWriter ow = new ObjectMapper().writer().without(SerializationFeature.FAIL_ON_EMPTY_BEANS)

Iterate the MappingIterator of Candle Pojos and serialize them into the output file:

while (it.hasNext()){ row = it.next() out.append(ow.writeValueAsString(row)) if (it.hasNext()) { out.append("\n") } }

It is a pretty simple script that is not doing too much but I will probably do a post in the future showing how to use the Jackson Streaming API to handle much larger data sets.

5. Conclusion

As we have shown here gist provides a simple solution for sharing your scripts with others or for yourself across machines.