I’ve been working on a project at the office recently which involves interoperating with an API written in WCF using message contracts.
Basically, the API manages the lifecycle of lists of securities in much the same way as one would have a watch list of eBay items. The project revolves around linking this API with another external helper API such that in normal operation the one would talk to the other and exchange some useful information in servicing a customer’s request – such as creating such a list. This is the integration I'm talking about. We’ve effectively become contractors to modify their API accordingly so as to make it talk to the 2nd external API(helper API). The API is a traditional on-prem service using Oracle, however, it's moving to AWS using RDS. Not that this affects me too much as most of that transition affects the lower level code and I’m much higher in the business logic.
Most of my work is behind the scenes and this should not affect the customer’s view of the service they’ve traditionally expected to work in a certain way. This requires that the API calls deliver the same types of messages back etc. ie. the interface should remain but the implementation will change to facilitate calling out additionally to a further helper API.
With all this being said, there are reams of considerations.
Firstly, that we don’t break the API such that customer’s existing apps break. To this end, the API does have some automated tests which should highlight these issues. That being said we still need to wire these tests into our src copy of the API. This will be useful. The other is that we can modify the code in such a way that what we’ve introduced doesn’t compromise the integrity of the existing codebase – i.e introduce nasty bugs that we have to fix down the line. This is more of an architectural issue about how we should structure our new code changes to minimize the impact of our code while still delivering the new code and introducing the new functionality. Ideally, the new code will be contained and fairly isolated within a new sub-project which will then be used throughout the API. We’ve called this the ‘service’ project and it contains all our integration code.
Initially, we’ve taken a moderately pragmatic starting approach – basically make the changes in any way we can that basically gets the basic ideas to work. This is a risk mitigation strategy so that we can quantify the amount of work to just do this, without taking into account any other considerations and merely proving the possibility that it can be done.
Once we’ve established that a pragmatic, naïve and architecture-less implementation solves the imminent tasks, then the risk shifts to making sure that we introduce a better architecture that is less ‘as-we-go’ and more reasoned. To this end, we need to ‘design’ our code into an architecture that mitigates against future bugs and reduces the surface area of code that we introduce, making changes less problematic and affecting less of the overall systems than absolutely necessary.
The less coupling our code with the existing codebase, the less the risk of a change we introduce changes that ripple throughout the system. But like most things, this is a journey: from naïve implementation to realising an architecture to contain it.
As part of introducing a way to reduce the amount of code we change, we also need a mechanism to test the code we’ve introduced. To reduce the amount of code that we change or at least its impact, we need to change the code in one place and ideally reference this one place throughout the other code as necessary. This is so that we don’t duplicate code in multiple places and what code we do use can be tested in isolation to the places that use it.
This is where the unit and integration tests come in. I'm a great proponent of unit testing - ever since setting seeing the benefits while designing my C API with integrated tests. Unit tests are great for ensuring assumptions about small units of code remain the same, irrespective of code changes elsewhere in the code. This helps ensure that by changing the seemingly unrelated code, doesn’t break the assumptions that the unit tests cover for other code.
Issues with unit tests for legacy systems is that existing systems aren’t usually designed to be testable, that is the codebase doesn’t make it easy to run code in isolation to the whole application being run. There are ways to mock out or fake and otherwise difficult subsystems that would usually require the application as a whole to already be up and running. Also, it can be more pain mocking these dependencies that is practical. However, we can introduce our new code such that we can unit test it. The difficult parts that we need to interoperate with can be covered through an integration test, which tests the system as a whole – basically black-box tests where you test the outcomes of running the systems and make sure that they output correct results.
Another useful thing to do is to try and use the API as a customer – this is the black-box type integration testing mentioned above. So calling the API and ensuring it does what is expected by the customer as well as the changes introduced are doing the right thing – such that the helper API is being called and being used correctly. To this end, we’d call the API and then also call the helper API so see that the interoperation took place and that the results are as expected. These types are changes are next to impossible with white box/unit testing.
A lot of this type of thinking is about risk mitigation, it's about increasing one's chances of success. Attempting to see into the future and trying to circumvent issues that might and probably will come up is far better than a wait-and-see approach, but inevitably it usually always starts out this way as I mentioned ealier. It also allows you to be more confident and professional about the work you do. This is why modern software development is as much to do with testing and planning and adaption to change than ever before. Probably why AGILE is so useful these days.
Anyway, I’ve started a unit test project and an external cmdline app to integration test the API from a customer’s point of view. I’ve also implemented an internal integration test suite that calls only our service project and allows it to call out to the helper API. So basically two types of integration tests are in place for this project: external customer-API-view and internal-view of our code and its effects on the helper API. The internal integration tests are great because we don't have to run the full API. There is nothing worse than being on a deadline and the tools you use are slowing you down more than the code itself!
As mentioned previously, to ensure that the code we introduce is not scattered all over the show, I’ve ensured that the new code resides in a ‘service’ project' where our code is the service being offered to the API, that code is responsible for calling the helper API.
A good thing about putting all our code in this project is that we a unit test it and we can re-use it throughout the project. Changes in the service project should uniformly change in other areas that use the service project - replication not duplication. Another great thing about integration tests is that while they exercise the internal service project code, they don't have to have the main API running as mentioned before, particularly if you’re just connecting to the helper API. Otherwise, code change would require and re-deploy each time(rebuild an run) and that takes time. Another benefit of isolating any code(all out code) introduced via the services project is that we can add timing functions to just the services project to see how long it takes to run the certain code. Integration tests that test the API from the customer’s point of view can time how long the calls take now in contrast to how long they took before without the code.
One thing that is useful is being able to turn off the new functionality without affecting the product, such that the product doesn’t need the new functionality. This kill-switch design should be implemented within the architecture of the newly introduced code i.e. the service project. So theoretically you can just turn off the new functionality and the same level of service that previously was in place and expected by the API resumes as per normal. This obviously requires a design consideration.
So in a nutshell, my integration work has revolved around:
- Architecture for newly introduced code (isolation, testability, low coupling etc)
- Service project
- Kill-switch functionality
- External Integration tests – customer expectations are met and that the API still works as before. (Cmdline)
- Internal integration tests – run our code without having to run the main API while still talking to the new helper API (unit test project)
- Timing functions – make sure we’re not slowing things down, we can quantify code changes in terms of their performance.
The other thing we’re doing is introducing a mechanism whereby the helper API, if its used independent of the main API, then that the main API gets notified of this usage. To this end, we’re putting together a AWS lambda function that will be pushed notifications from the helper API and then those notifications will be subscribed to by the main API. Something like this:
The difficulty in being a 3rd party contractor is learning how to get the existing build system to work for you, and in our case to deploy the AWS Lambda. It at times feels like we’re build engineers too – not my favourite past-time as I prefer to be working on the code not figuring out how configuring build scripts in a system I don’t know. Though that’s the point I guess – you need to learn to know. This is what's involved in coming in 'cold' from the outside - contractors need to know how to figure stuff out. Its stacked up against you otherwise.
I’ve designed the lambda to receive notifications through HTTP endpoints using API gateway and learn how the internal build system publishes lambdas using “serverless” scripts. So I’ve had to learn how to do that and as it turns out, serverless is a useful system.
The external integration tests, as well as the lambda, will talk to the main API like any other customer – through direct calls to the API via WCF/Http basic. I’ve incorporated this API calling code into a library that both can use so I don’t have to duplicate effort to connect to the API. The external integration tests are run using a cmd line app written in .net core 2 and so is the library as well as the lambda. I've come to love https://www.nuget.org/packages/Microsoft.Extensions.CommandLineUtils/ to help with this. I added some boiler-plate code here that contains the skeleton code for coding up a cmd line app like this in minutes. check it out here. Here is what it basically looks like:
public static int Main(string[] args)
{
var app = new CommandLineApplication();
app.Command("Command1", target =>
{
target.Name = "Command1";
target.Description = "Command1 Description";
target.OnExecute(() =>
{
Console.WriteLine("Command1 not ready yet");
return -1;
});
target.HelpOption("-?|-h|--help");
});
app.Command("Command2", target =>
{
target.Name = "Command2";
target.Description = "Command2 Description";
target.OnExecute(() =>
{
//work
return 0;
});
target.HelpOption("-?|-h|--help");
});
app.Command("Command3", target =>
{
var argPortfolioNameOrCode = target.Argument("Portfolio name/code", "The portfolio name or code");
var argPortfolioType = target.Argument("Portfolio type", "The portfolio type");
target.Description = "Command3 Description";
target.OnExecute(() =>
{
if (MissingMandatoryArguments(target, argPortfolioNameOrCode, argPortfolioType))
{
return -1;
}
//work
return 0;
});
target.HelpOption("-?|-h|--help");
});I added some handling for mandatory arguments because it appears that this is not provided by default. The nice thing about this is that you add commands, and then you can have integrated help for each command, much like many linux utils like git for example.
Anyway getting back oon track now, the internal integration tests are unit tests within the main API and just test the service project code. The external integration tests are via a cmd line .net core app.
One thing that has become apparent is that things take longer than expected when interfacing with people/systems that you don’t know. And sometimes I feel that people aren’t as responsive to your requests as they might otherwise be if you were part of the company(we’re contractors). The other is that is always more useful to take a top-down, wide bird-eye-view of your solution’s impact on the system. This allows you to envelop the problem and attack it from all areas and more importantly see all the areas. Once you know all ways in, you can focus on those ways in.
I’m also thinking that it might be a useful time to introduce a mechanism that reduces the impact of code changes in the service project, things like factories etc but I'm not sure what I need yet.
The point of all this upfront cost/work/time is that we now just need to focus on delivering the functional business requirements and hopefully just add new tests and re-run old tests as we go and this should ensure that we’re introducing working code that doesn’t break each time we make new changes. We will also have confidence that the API still works as expected via integration tests and that our code is testable and isolated.
I’m also reading an interesting book about Investing by Tim Hale called “Smarter Investing – simpler decisions for better results”, which is interesting and I’m also working on my investments project. I’ve recently implemented a means for a user to define their own entities and associate them with investments. Its still in progress but interesting none-the-less. It's on its own branch at the moment until I get it fully integrated. I've also done some Running in the heat.
I met up with my old colleagues at Citrix over the weekend to swap stories. We met up in Gerrard's cross at the Grey Hound. It's nice to hear what they are doing now. I then took a nice leisurely run back down my old running route and it felt like I was right back running from the office as per usual. That's one of the great things about this life is that you can run whenever you want. I even waited at the same bus stop shelter and took the same bus home. Quite surreal really.
Anyway more to come as the trials and tribulations around API integrations, software design decisions and who knows even a little running.