The Brief Note: I am on week two of running a HomeAssistant service so I’m still learning. If you have any suggestions or improvements please let me know. I want to be able to set my Slack presence when I’m... more
How do you have a long polling mechanism that can be cancelled even during it’s sleep? THis is a question I found myself facing recently with a CLI app that is polling AWS Cloud Watch Let’s say you have a... more
Full disclosure, this is a pretty niche blog post and the prerequisites for getting it working were enough to make me consider not posting. So what is it about? When I start a Zoom call, I want to dim the... more
I’m familiar with Redis as a fast cache and have used it as such in the past. A few days ago I saw someone using Redis Insights to interact with RedisGraph and it looked pretty cool. I wasn’t aware that... more
In defsec, we have a large number of Go objects which represent real world Cloud entities. A documented schema is required, but due to the number, it would be nice to automate the generation. In this post, I’m going to... more
An introduction For those who don’t know, I’m one of the original creators of tfsec and I now work on Trivy as an open source engineer at Aqua. Together with @liam_galvin I am working on adding scanning support for Azure... more
UPDATED 21st Sept 2022: v1.0.0 Release View lazytrivy on GitHub Trivy is the go to scanner for vulnerabilities and is rapidly becoming the go to for all your scanning needs. lazytrivy just makes the experience even easier. You can run... more
A bit of background I’m working on a tool that uses the AWS Go SDK to pull back a lot of information about an account using the API. Some services require several secondary API calls - for example to get... more
What is this about? This blog post is kind of a learning in public exercise. The goal is to run Prometheus in my local K8s cluster and export Node metrics to it. I’m using a cluster generated by my local_k8s... more
I have had an on/off relationship with Kubernetes for a number of years and want to spend some time really getting to know how to use it, run it and secure it. I previously wrote some Anisble playbooks to run... more
I am having a baptism of fire with UI development updating a stalled knowledge base at work. One of the entirely new technologies I’m picking up is sass for style sheets; the workflow of compile -> minify seemed clunky and... more
Hot on the heels of tfsec, we have been working on cfsec for scanning your CloudFormation templates. cfsec will scan both yaml and json template files and gives you the same rich information about any issues that it finds. The... more
A few weeks ago I wrote tfsec-pr-commenter-action , this is a ready to go Github Action that you can drop into your Terraform code repository and have each PR checked for tfsec security issues. If you don’t know anything about... more
In October 2020, the Thoughtworks Tech Radar promoted tfsec to Trial status. In the past few months, there has been a flurry of activity to improve performance, rewrite the parsing and add new features. One of the recent additions is... more
We use Terraform for all our deployment automation needs. Thanks to it’s fantastic extensibility, if there isn’t a provider available to do what we need, it’s very easy to create one. Terraform, for those who haven’t used it before, lets... more
This website is generated using Jekyll, a static site generator. Basically, I write the posts as Markdown files, run it through Jekyll and out pops this website, owenrumney.co.uk. For as long as I have been building the site in this... more
The AWS SDK provides a number of waiters that allow you to block your code while waiting for a process to complete. One that we make use of in our managed ECS container rollout is the services_stable waiter. This will... more
I have been using using Docker and Kubernetes off and on for work and personal for a few years now, but I was recently shown a feature in the Docker file that I wasn’t aware of. A bit of background... more
Last year I created a diagram tool specifically for AWS diagrams. It was essentially a hack of the underlying code draw.io uses. The reality is, if you’re going to be doing diagrams you might as well use the proper draw.io... more
Sometimes you inherit or even create a huge Makefile which is unwieldly and difficult to understand. The longer it is, the more complicated it can be to find out which targets are available and what they do. This post covers... more
Overview I’ve just spent a bit of time trying to work out how to group a Spark Dataframe by a given column then aggregate up the rows into a single ArrayType column. Given the input; transaction_id item 1 a 1... more
Overview As part of my journey into using Scala I have had to get used to the ScalaTest and the wealth of functionality it offers. One of the enduring headaches with unit testing is find a clean way to test... more
Overview At my current workplace we are using Databricks with much success. Having recently activated the Security Operations Package I was keen to implement the Single SignOn (SSO) functionality. The documentation provided by Databricks doesn’t seem to cover integrating with... more
This is less of a post and more of a CIDR to mask cheatsheet to remind me how many IPs I get for a given block. CIDR Subnet Total IPs Usable IPs /32 255.255.255.255 1 1 /31 255.255.255.254 2 0... more
This is pretty short post. H/T to @prokopp for telling me know about this. Git allows you to add aliases in your global config - this is the first one I’ve actually added and all it does is a concise,... more
I recently posted an entry announcing www.aws-glossary.info. In this post I want to cover how it actually works. Basic Architecture The logical diagram below details the basic structure of the application. The core components are; Component Purpose Github Static content... more
Yesterday I created a scraper for the AWS documentation landing page to give an aws glossary of all of the services that have available documentation. Originally I posted it as a page in this site, I’ve now moved it to... more
This morning I was trying to solve an issue where http://owenrumney.co.uk has an SSL error when trying to access my blog but https://www.owenrumney.co.uk is fine! (its still not resolved). As I was poking around in my github account I looked... more
Overview We have just replaced our aging Canon MX885 Multi Function printer with a new colour Laser printer, an HP M281fdw multi function. One of the scanning options is to “Scan to Email”. This is something I’ve found really useful... more
Assumptions I’m assuming for anyone who is doing this that you have your /etc/krb5.conf in order and that isn’t going to get in your way. One thing you’re going to want to know is what your permitted and default enctypes... more
This post covers the steps to create a simple dockerised flask app to cover some of the basic steps required when creating a REST(ish) service that can be run as a Docker container. The App Rather than go with the... more
In this post I’m going to cover the basics of implementing retry logic using urllib3. There is probably a solid argument saying “why aren’t you just using requests?”, as it happens, requests uses urllib3 and it’s Retry functionality. For the... more
In a recent post I covered an using RefreshingAWSCredentials within .NET AWS SDK to solve an issue with the way my current organisation has configured SingleSignOn (SSO) and temporary credentials. Essentially, the solution involves a background process updating a credenial... more
Where I am currently working we have Single Sign On for AWS API calls and need to use task accounts to connect and get temporary credentials. To that end, its not very easy to have long running processes making calls... more
Python is one of those languages where if you can concieve it, there is probably already a library for it. One great library is Faker - this makes the generation of sensible test data much easier and removes a lot... more
When working with EC2 instances, you really want to be right sizing the instance from the outset. With Amazon regularly bringing out new classes of instance it is hard to keep track of what is available now and what the... more
Back in 2014 I wrote an article about borrowing the bing wallpaper to use on my Macbook. Not long after that, I found that I was missing the odd “Bing of the Day” so I decided to run it in... more
Last year I managed 3 blog posts so this year my aim is to beat that by at the very least doubling it. At the end of 2018 I completed my AWS DevOps Engineer - Professional certification, to that end... more
The first version of gitsearch-cli is now available. This command line interface allows you to search github repositories and users using keywords and (currently) a handful of additional criteria. Installation To install git search you can use pip3 with the... more
We need to access ElasticSearch in a namespace within minikube and the other Pods can’t connect to 9200. It turns out that from the box its limited to localhost and the network.host property needs updating. Setting network.host in the elasticsearch.yml... more
Regularly when writing a shell script I find that I want to be able to pass an argument into the script but only sometimes. For example if I want the script to output to /tmp folder for the most part... more
Its reasonably easy to run a Spark job against HBase using the newAPIHadoopRDD available on the SparkContext. The general steps are, create an HBaseConfiguration create a SparkContext create a newAPIHadoopRDD perform job action To get this working, you’re going to... more
If you have committed some code to git (or in the current case, BitBucket) and you have made an error in the commit message (in the current case, referenced the wrong Jira ticket), all is not lost. To replace the... more
Following on from part 1 of this mini series. I’ve got my local environment ready to go and I have pulled down some test data to work with. The next step is to start having a look at some of... more
I’m hoping this will be a reasonably accurate account of my play with the TfL Cycling DataSets. I’m still forming my plan, however loosely I think I want to end up with a visualisation where the bike points are highlighted... more
Now I’m writing almost all documentation in markdown then using Pandoc to convert it to Mediawiki or docx as required, I needed to finds an easier way to quickly create my tables. It doesn’t do anything fancy, but I created... more
I’m having a bit of a dabble with Go, as a by product of working with Elastic Search beats. One thing I’ve been looking at today is the channels to allow two go routines to communicate with each other and... more
I’ve moved to a new role where I will be doing a lot more “devops” type work which hopefully comes with more interesting subjects to start writing about. Oozie and AWS Simple Email Services The Cloudera cluster I have started... more
Today someone showed me a handle word cloud tool called wordle. I put the skills from my CV into it and had a play with the output… I think it looks quite cool. more
For a while Garmin have been adding “wearable technology” functionality to their watches - notifications from the phone etc. I got a Garmin Fenix 3 for Christmas and had a play with the vast number of watch faces and widgets... more
Today I’m at my first Scala eXchange conference - the 5th annual one to be precise. The day started with the keynote session by @jessitron which was both inspiring and enlightening. The general tone being that code should be clear... more
I need to write a couple of unit tests for some code to add a log entry into HDFS but I don’t want to have to rely on having access to full blown HDFS cluster or a local install to... more
Here we are in June, some five months since the last post and I finally have some time and content to sit and write a post. In April 2013 I started working with Hadoop, the plan was to suck in... more
All the work I have been doing with AWS has been using Python, specifically boto3 the rework of boto. One of the intentions is to limit bandwidth when transferring data to S3 the idea is to send periodic snapshots then... more
Towards the end of 2014 Amazon released the KMS service to provide a cheaper cut down offering for Key Management Services than those provided with the CloudHSM solutions (although it still uses hardware HSM underneath). KMS service can be accessed... more
Some time ago, back in September, I wrote a post on starting my adventures with Spark but didn’t progress things very far. On thing that was holding me back was a reasonably real world problem to use as a learning... more
I’m working with a new AWS account and I am moving to testing Boto3 to use the KMS service. I needed to make sure that the AWS account and secret keys were updated so ran aws configure to quickly update... more
I’m not sure if this is because we’re behind a proxy, the network has issues or my work laptop isn’t great, but for some reason the git clones very often hang during the unpacking of objects. remote: Counting objects: 21,... more
For 18 months I’ve been working with Hadoop. Initially it was Hortonworks HDP on Windows then Hortonworks HDP on CentOS and for production we settled on Cloudera CDH5 on Red Hat. Recently we’ve been introduced to Spark and subsequently Scala... more
I’m not a huge fan of Bing search engine, I’ve tried to use it but I don’t like the format of the search results and I don’t think it’s particularly good at finding relevant results either. I do like Bing... more
I know that it is incredibly lazy and a non problem but I find it quite tedious in Windows 8 to go digging for the system environment variable GUI whenever I need to add or update something. Generally I’m already... more
I’ve just had an interesting problem with an SFTP account that suddenly stopped working from a cron job. When the account was used directly from the bash prompt the response was simply Connection closed immediately. Every thing was set up... more
As part of a project I’m working on I need to create a cross join of users against symbols; the idea being that any of the given users might look at any of the given symbols. I have a list... more
I have been setting up a couple of Red Hat boxes and I need to have a cron task to mount a network share from an AD domain, copy the files then dismount. In the past I have successfully accessed... more
I have been doing some work for a company who have a user base who primarily use iPads to access the SharePoint site. There was a requirement to allow those users to easily access the team site without getting their... more
I got the result and feedback for the first assignment of the final Open University project today. Its been a long week waiting for it, not least because a significant number of others on the course have been reporting back... more
At the eleventh hour I finally settled on the project that I am going to be working on for the next 8 months. I had previously intended to work on a project that would allow me to play with some... more
For the final project of my open university degree I am looking at creating an application to bring together a proprietary dataset of information about available workers and cross it with social media data. The project is in the early... more
Since November 2013 I’ve been working out in my garage with a cross trainer and timed interval training. I did have a membership to Virgin Active, but between my commute, Open University degree and family I just don’t have time... more
I’m setting up a tactical farm of Linux servers on some desktops till we can get the permanent kit installed in the data center. The new servers are running on trader desktops so they’re reasonably good kit. Too satisfy security... more
The SharePoint developer dashboard, when switched on, will render at the bottom of the page in SharePoint 2010 and appear as an icon in the top right hand corner of SharePoint 2013. If the DisplayLevel is set to OnDemand for... more
For the project I am currently working on, we have an order for a Hadoop Appliance from our supplier to be placed in the Mega Data Centre for processing all of our log files. The order is in, but with... more
For one reason or another, my SharePoint 2013 development environment became corrupt and the quickest way to get it up and running again was to start a fresh with SharePoint. The problem I faced was when running the Products and... more
Not too long ago I moved my website to a static page structure using Sandra.Snow. At the time, I was aware that it was based on the idea of Jekyll with a Microsoft lilt using Nancy to generate the static... more
I sat with my kids while they watched Justin Time (they love it) on New Years day and I started to clear up the disk in my Dell XPS 13. Its a 256GB SSD and while its not tiny it... more
When an update has been made to /etc/fstab the changes won’t immediately and automatically take effect so there is a need to refresh them manually. This can be done either by rebooting, which is a pain or just use the... more
I have been doing some revision on my WPF, particularly around ModelView-View-Model pattern. One thing that I’ve always found a bit difficult to get right is the handling of events on the window and getting them back to the view... more
One of the requirements for a piece of SharePoint/Office 365 work I’m working on at the moment is for easy finger usage when accessing with an iPad. The user base coming in on iPad are performing a specific task so... more
Since starting to use Azure more and more I’ve found myself revisiting suddenly enthused by the web and cloud. I’ve had the obligatory Amazon free tier but done little with it. (I’ve recently talked with Amazon about a project at... more
Now I’ve got into the Azure platform I’ve decided to start making the most of the MSDN voucher I get for free credits. , I am a serial “new project” creator, some of these projects make it as far as... more
An interesting chain of events happened last week; @thecodejunkie relaunched his blog, I happen to follow his twitter feed and he happened to mention it. The chain continued, I followed the link to his blog for a look and read... more