Running additional VSTS agents in Azure

When working in a smaller team or on a small project, you usually aim to reduce costs of your development pipeline. This is also true for huge projects, but there we're talking about bigger budget and money. When working with a free VSTS subscription, you're limited to only 1 hosted pipeline(and 4 hours / month) to run your builds and release. To be honest - it's nothing when you have real CI/CD set up and working. However, if you consider a fact, that you also have 1 private pipeline, it's possible to mitigate this problem.

Hosting your own agent

VSTS allows you to host and connect your own agents running on your machines. It's a great way to take advantage of a private pipeline when hitting a limit of 4 hours and allows you to have a backup when you need more builds and releases in a short period. To use an agent you need two things:

  • a machine which will host it
  • an installer

A good idea to get a machine is to use your existing Azure subscription(especially when you have MSDN/BizSpark) and create a VM, which can be started and stopped easily. Another thing are capabilities of your machine, which have to match requirements for your build(e.g. MSBuild, Grunt, gulp and so on). 

Get agent screen will help to both determine what is needed to install it and how to do it

Because of capabilities, using different(you can choose from Windows/OSX/Linux) agents can be easier or more difficult. In the future I will show you how to set up e.g. Linux agent, for now we'll focus on the Windows one.

The easiest way is to create a Visual Studio VM from the marketplace in Azure Portal. That way you will have most components needed for a build already installed and configured. 

Usinga Visual Studio virtual machine will ease whole process a lot

To get an installer you have to go to Agent queues screen in VSTS and click the Download button. As present in a one from the screens above, information needed to install and configure it are already there so I won't reinvent a wheel and just ask you to go through it by yourself.

Configuration

Configuration of your agent is pretty straightforward. When you enter following command:

/
C:\your_agent_directory> .\config.cmd

You'll be asked a couple of questions regarding your VSTS account, a name for an agent, agent pool and a method of authentication. The easiest way to authenticate is to use PAT(Personal Access Token) which can be generated here: https://{your_account}.visualstudio.com/_details/security/tokens.

Once you've configured your agent, you can run it with a .\run.cmd command. If everything's all rights, when you go once more to the Agent queues screen, you'll see your build agent available in the selected agent pool:

Building and releasing on your agent

So far so good - we have additional build agent, which can be used in our private pipeline. But what we have to do to schedule a build and a release? 

Builds

Go to the Builds screen. When you click on the Queue new build... button, you will be asked to select a couple options related to a build. The one we're interested in is Queue. Just selected the one which have your build agent configured and click Ok. Your build should start though you've used all available minutes from the hosted pipeline.

Selecting a different queue will allow you to use a different pipeline

Releases

Releases are a bit tricky. When triggering a release you have no option to specify a queue. To do so you have to go to a specific release definition and find Run on agent link above release steps. When you click on it, it will show you another screen, when you can find Deployment queue list, from which your private pipeline can be selected.

Somehow hidden Run on agent screen allow you to run your releases on your own agent

Summary

In above solution you have to consider costs generated by using a VM to perform builds and releases. Currently you can buy another hosted pipeline for 40$ so it's two times less than a minimum VM needed for a Visual Studio. On the other hand, you can automate it so it works only in your work hours = saving some money. If you have an available subscription like BizSpark, you can save even more money. Both solutions are viable but as you can see, starting another build agent is a very easy task, and can help you when you either want to control usage of your pipelines or to build something, what couldn't be built by a hosted agent.

 

 

EventStore on Azure and Ubuntu - it's a piece of cake! #4

Finally - the last but not least post about setting your own EventStore instance using Ubuntu on Azure. We've already prepared the majority of work needed here, so it shouldn't be difficult to adjust is just a bit to use DNS instead of hardcoded IPs.

Configuration

Currently our configuration looks like this:

/
---
RunProjections: None
ClusterSize: 3
GossipSeed: 10.0.3.6:2112,10.0.3.5:2112
DiscoverViaDns: False
ExtIp: 10.0.3.4
IntIp: 10.0.3.4

Clearly the first change is to get rid of the DiscoverViaDns property. The reason why we're going to remove it is the fact, that it's set to true by default. However, it appears that we need two more properties: ClusterDns and ClusterGossipPort. Additionally we'll remove the GossipSeed property as it also won't work here anymore. Let's get to work!

Network, DNS and Azure

When we created Ubuntu VMs in Azure we're given a single instance of a virtual network. You can think about it as a logical representation of your network in Azure - you manage IPs, DNS and other settings without installing physical devices. It gives you isolation and security - if you want to, you can forbid both inbound and outbound traffic. What we're interested in right now is its DNS capability. Currently you have two options:

  • use a DNS provided by Azure
  • use your own DNS 

Unfortunately using the former won't work here - we have to add records manually, what is not allowed when using Azure DNS. Obtaining and configuring a DNS server is beyond the scope of this post - if you're interested take a look here. The good thing is that it's still possible within Azure and additional tools are required. Once you have your DNS, the configuration should look similar to:

/
---
RunProjections: None
ClusterSize: 3
ExtIp: 10.0.3.4
IntIp: 10.0.3.4
IntTcpPort: 1111
ExtTcpPort: 1112
IntHttpPort: 2113
ExtHttpPort: 2114
ClusterDns: domain.com
ClusterGossipPort: 2113

DNS entries

The tricky thing here is to set correct entries in your DNS server. What you have to do here is to add an A entry pointing to your private IPs inside a network. Note that there's no concerns in doing this - it's a common practice. The one issue here is that it describes a little how your local network looks like - while accessing domain from an Azure network will point to the correct machine, when one tries to access it from an external network, he will be redirected to the private IP being the same as the one used in an entry.

Summary

This is it - we've went through installing, configuring and managing EventStore using Ubuntu and Azure. I strongly encourage you to discover other OSS solution, which could be run using such configuration and play with them, it becomes more and more fun.