Recently I wrote a small Dist::Zilla plugin to help with the problem of dependencies being old in modules I author.
I use update-cpanfile to check dependencies in projects I write using a cpanfile (and carton); I wanted a tool to do the same thing with libraries I write.
Using it is really simple you just add [Test::Prereqs::Latest] in your dist.ini and then when you run dzil test it will add and run a test file (xt/author/prereqs-latest.t) which checks the version of each module in the [Prereqs] secion of the dist.ini and fail if the version you have specified is lower than the latest CPAN.
The code is pretty simple, using existing work by HITODE whose update-cpanfile I use regularly to help keep dependencies up to date in projects I have that use cpanfile and carton.
It's simplistic, but works well. If the version in the dist.ini is lower than the version on CPAN it fails the test and tells you as much.
Once written, I was easily able to test it against a couple of modules I maintain by installing the package with dzil install then in the other module I add [Test::Prereqs::Latest] to the dist.ini and ran dzil test and it worked.
Once I had done basic testing locally; was able to create a trial release and upload to CPAN with dizil release --trial which built and uploaded the distribution.
Of course CPAN is amazing, so shortly afterwards the cpan testers started discovering the module and testing that it built on a variety of versions of Perl and a variety of platforms. People love GitHub actions but cpan testeers was first and I did literally nothing to get all this amazing testing without any configuration work, nothing... it just happens. It's amazing, seriously amazing.
The module is not ready for a proper release, but it's been nice to "scratch my own itch" so to speak.
In my last tale I outlined how I used Dist::Zilla to do much of the heavy lifting for packaging and releasing a Perl module to CPAN. In this tale I want to outline how quick and easy it was to write that module; wrapping a proprietary undocumented API so that I could easily use my personal data in ways the company did/does not provide.
The module in question is WebService::SmartRow. SmartRow is a physical device and app that provides power and other performance data from a indoor rowing machine (specifically the WaterRower) machines.
It is a bluetooth device, that sends data to a web service via a mobile application. This data is then accessed via the app, or the website. The data includes things like the power in watts, stokes per minute, heart rate, etc.
The specific data I wanted to see, was/is how my average power changes over time. Specifically, across a series of 5000 meter rows, am I generating more power over time. Also I wanted to see my personal best times, over time. I.e. how much better was this 5000m over my previous best.
The API
SmartRow do not provide a developer API (at the time of writing, and as far as I was able to see). What they do have is a website that is mainly driven by JavaScript and a JSON API. As logged in user my browser makes calls to the API and then creates pretty HTML from the JSON returned.
The API is really quite tidy and easy to understand. Which is nice! I have previous experience wrapping JSON APIs and it can be really ugly; thankfully the SmartRow team have made a clean, easy to understand API.
The API relies on you being logged in, but quick exploration showed that basic auth is possible on the API endpoint. Meaning I did not need to worry about OAuth or keys and so forth.
The web requests
Knowing that I could get away with basic auth, that meant that the code I needed could work simply by including the username and password in the URL I make a HTTPS request to.
I turned to HTTP::Tiny and was quickly able to craft a get request that looked a little like this:
Then (after some basic tests to make sure I got a 200), I could parse the $response->{content} from JSON to a Perl data structure using Cpanel::JSON::XS.
This gave me a data structure looking like this:
[
{
"accessory_mac" => E,
"account" => E,
"ave_bpm" => E,
"ave_power" => E,
"calc_ave_split" => E,
"calc_avg_stroke_rate" => E,
"calc_avg_stroke_work" => E,
"calories" => E,
"confirmed" => E,
"created" => E,
"curve" => E,
"device_mac" => E,
"distance" => E,
"elapsed_seconds" => E,
"extra_millies" => E,
"id" => E,
"mod" => E,
"option" => E,
"option_distance" => E,
"option_time" => E,
"p_ave" => E,
"protocol_version" => E,
"public_id" => E,
"race" => E,
"strava_id" => E,
"stroke_count" => E,
"time" => E,
"user_age" => E,
"user_max_hr" => E,
"user_weight" => E,
"watt_kg" => E,
"watt_per_beat" => E,
},
...
]
(The above code is actually taken from the Test2::V0.t file I wrote to confirm the structure.)
So you can see it's pretty easy to understand, the keys are all mainly understandable. In my case I wanted the p_ave and distance, so I could filter on 5000 meters and build an array of all the average power values.
Module creation
At first this could have been a simple script, but I wanted to make this something portable and usable by anyone wanting to work with their personal data.
So after I proved the approach would work, I started a module (using dzil new WebService::SmartRow). This is currently a single file with little refinement.
I used Moo for some simple OOPness and structure. This allowed me to specify the attributes I want:
has username => ( is => 'ro', required => 1 );
has password => ( is => 'ro', required => 1 );
There are pretty self-explanatory, to use the API you need those, so add them as required attributes.
Next I added an http attribute:
has http => (
is => 'ro',
default => sub {
returnHTTP::Tiny->new();
},
);
The default here creates a HTTP::Tiny object which I can later use in methods via $self, which meant my earlier get request changes to look like this:
You can set your own http attribute when creating via WebService::SmartRow->new() so if you need to do something like change the user agent, or have a preferred module, you can inject it easily (assuming the methods match HTTP::Tiny).
Testing
Currently the module is pretty simple, three attributes and 4 public methods. The module has little smarts so the t directory is pretty spartan as the object is pretty simple.
I am using the xt directory to hold tests that talk to the API and as such require an internet connection and credentials.
Not wanting to include my personal credentials in the repo, I have a private sub in the class that gets the username and password from environment variables. Which is good as it means I can commit my tests, and if someone using this module does not need to commit their credentials in code either.
Perl makes the environment variables easy to work with, so the small sub that handles it looks like this:
So if you have instantiated the module with username or password it will use those. If they are not present it will use SMARTROW_USERNAME or SMARTROW_PASSWORD.
Then (and I know I can make this a bit smarter) my get_workouts() method, has a call to my ( $user, $pass ) = $self->_credentials_via_env; prior to calling the URL.
And it will go and connect to the API and execute tests for me by getting real data from the API.
Earlier I mentioned I am using Test2::V0 for tests, so in xt I have a growing collection of files that primarily confirm the structure of the data returned from the API.
Mainly they use E() to prove hash element exists, some could/should/do test the content. For example in one test I just wanted to confirm that the data and distribution fields are arrays before later testing the content of those arrays. So I have a test like this:
is $leaderboard, {
distribution => {
ageMode => E,
data => array {
etc(),
},
max => E,
mean => E,
min => E,
range => E,
userPercentile => E,
},
id => E,
mod => E,
records => array {
etc(),
},
},
'Leaderboard structure is as expected';
It is mainly E() checking that is exists, then array to confirm it contains an array. etc() tell the testing code to assume some rows, but not to check the content of them. They just have to exist. As etc() is the only thing in there, as long at data and records contain arrays with some rows, the tests pass.
Having tests like this is really helpful when wrapping someone elses API previous pain has taught me. If the structure of the return data changes, I can easily confirm what has changed.
When you are wrapping an API this way it is an inevitability that the API will change so having the tests is one of the only ways to ensure you can maintain it over time. My Webservice::Judobase has had this happen a multitude of times in the past 5 or so years.
Summary
As you can see from the brevity of this tale, wrapping an API is pretty easy to to. I know from previous experience that doing it helps others build things with Perl. So it's the sort of project that helps both the service and our language.
Perl is a famous "glue code" language and this is a great example of why. Two CPAN modules do almost all the heavy lifting and our fantastic test tooling means I was easily able to write tests that will make it easy to maintain.
Dist::Zilla (as per previous post) I use to automate most of the publishing work. I will write something up about that another day. It has lots of plugins that make sure your module is up to scratch.
The JSON API and JavaScript front-end trend has meant that lots of websites have a JSON endpoint that you could wrap. This means you could as I have create a tool that uses your data in a way that the website may never build for you.
It also gives me confidence that I could pull my personal data out of the service and store it in my backups (I am doing that next), so that if the company ever goes bust, I have my data there and have it backed up securely and independently. I can do with it what I please as it is just data on my disk(s), either as plain files or maybe pushed into another database.
If you use a website and there is no module for it on CPAN, maybe it has an API you can wrap too?
Perl is a mature language, businesses have been relying on it for decades, a common refrain is that Perl is battle tested. One of the strengths of a mature language is that the community builds tooling around the language. Perl has some well known tools and some less well known ones. In this tale I want to talk about some of the tooling I like and perhaps are less well known.
Having a good set of tools helps developers build software better and more easily.
Well known tools
Here are some of the netter known tools that many Perl developers use everyday.
Perl::Tidy
Consistent formatting is surprisingly important and time consuming. Especially in a team environment running Perl::Tidy is helpful for visually formatting the code so that no matter who writes the changes they are formatted consistently. Be that tabs vs. spaces or aligning the "=" signs in variable assignments.
Perltidy is similar to gofmt (GoLang), prettifier (Node, etc) or elm-format (Elm).
Perl::Critic
perlcritic is a static analysis tool, it is a very flexible and configurable tool that allows you to define and detect rules on things you don't want in your code base.
For example, a common rule is that all sub routines need an explicit return; perlctitic can enforce this across your code base. There are over 100 "policies" on CPAN for perlcritic. Another one Perl::Critic::Policy::logicLAB::RequireParamsValidate insists that the Params::Validate module is used to validate all parameters to your subroutines.
Perl::Critic::Policy::Variables::ProhibitUnusedVarsStricter prevents you defining a variable that is never used. This is a pretty common thing to find in a large code base, either because the variable is no longer used or there is a typo somewhere.
prove
Perl has a strong and well developed testing culture and tooling. prove is the test runner most developers are used to using.
There are a wide selection of Test:: modules that you can use for mocking, unit testing, BDD, even enforcing Perl::Critic policies.
perlbrew
Perlbrew is relatively well known, though perhaps less often used than it might deserve to be.
Perlbrew allows an easy way to install various versions of Perl and switch between them. It is often used on your development machine especially if you need to work on specific versions of Perl so you can support legacy applications. Increasingly we are seeing it used in the building of Docker container.
Less well known tools
carton
If you have written any node you will understand carton. With carton you describe your module dependencies in a file called cpanfile and then install them with carton install This installs all the modules in a ./local directory and you can then run the perl application with carton exec and it runs the code with all the dependencies in the correct path.
This is particularly helpful for when you have multiple projects and they use differing versions of the same dependencies.
Because I use carton with almost all my projects now, I have the following two aliases setup:
perl='carton exec perl'
prove='carton exec prove'
These mean that if I forget to type carton exec perl some_script.pl and type perl some_script.pl it works as expected using the local dependencies. The prove alias is handy as I almost never remember to type carti exec prove.
update-cpanfile
This command-line tool is really helpful for maintenance of your dependencies. When you run it with update-cpanfile pin it will pin your dependencies in the cpanfile. carton install will then install the specific versions in the file. This keeps you dependencies consistent; but you could do that by hand in the cpanfile.
Where cpanfile-update is really helpful is when you run it with update-cpanfile update, this will make the tool check cpan.org for the latest version of modules and will update the cpanfile for you with the new version.
If you maintain a variety of projects or have lots of dependencies update-cpanfile is a real time saver.
yath
The simplest way to describe yath is to say that yath is the new prove... but better.
The venerable prove has been used for a long time, yath is relatively new (2016) and can be used pretty much to do all the things prove does.
However, it's able to do some really interesting things such as running in a daemon mode saving startup times... give it a try.
perlvars and perlimports
These two tools are super handy for analyzing your code base to ensure that some undesirable code smells.
perlvars identifies unused variables in your code.
perlimports identifies unused modules you may have included in your code but are not using.
Removing unused variables and modules helps keep your code "tidy" and improve memory consumption and protect against such things as methods from an unused module causing issues when another module has a method with same name for example.
What am I missing?
This is just a short list of tools I wanted to mention, in part to invite people to let me know what they use.
So if you have some tools you use all the time, drop me an email or a tweet.
This week I tried a new OBS setup to livecode the solution on Twitch.tv/lancew. The setup is thanks to Martin Wimpress who shared his setup ( Creating the Slim.AI OBS Studio set ) and this broadcast is extra special as I was able to bring Mohammad Anwar into the live stream; which was awesome.
This week I solved the "Chowla Numbers" task of the Perl Weekly Challenge, first in Perl; then in Elm.
Both I did as TDD-ish as I could, some cheating where I just Data::Dumper'd to STDOUT in the Perl version. The Elm one, I used the tests a bit more as I am less familiar in the language, and to be frank I didn't know an easy way to dump to the screen. ;-)
The Elm version I ended up with this:
module Chowla exposing (chowla, n, nums)
import List
nums : Int -> List Int
nums max =
List.range 1 max
|> List.map n
n : Int -> Int
n num =
let
denominators =
List.range 2 (num - 1)
numbers =
List.map (chowla num) denominators
in
List.foldl (+) 0 numbers
chowla : Int -> Int -> Int
chowla num denom =
case remainderBy denom num of
0 ->
denom
_ ->
0
The Perl looks like this:
package Chowla;
use Moo;
sub list {
my ($self,$max) = @_;
my@chowla_numbers;
formy$n (1..$max) {
push@chowla_numbers, $self->n($n);
}
return \@chowla_numbers;
}
sub n {
my ($self,$num) = @_;
my$total = 0;
formy$n (2..$num-1) {
if ($num % $n == 0) {
$total += $n;
}
}
return$total;
}
1;
Neither are particularly long, the Elm without type signatures is 15 lines. The Perl 21 lines (though if you formatted it differently it would be about the same).
The Elm code, not having a for loop relies on map and fold (specifically foldl). I could/should try the Perl with a map. There is no equivalent of foldl in Perl. foldl is in interesting command, basically like a map, but collapsing in on itself via a function, in my case + so just summing all the numbers.
Testing the code in Perl was as ever easy, Perl's Test2:V0 is a great tool. On the Elm side I am getting more comfortable with the testing, I did not use any of the built in fuzzing on this one, I will have to explore it more.
Over the next week I have some more Go code I want to work on for https:://fantasy-judo.com and I'd like to work on modernising my https://www.vwjl.net/ a little more, the combat simulation code is where my attention should go. Given that, I may not live stream the weekly challenge this week, but might stream some general hacking about.