Posts

Dask’s dashboard is great, but temporary Dask has a great set of built-in diagnostic tools. In particular, Dask’s dashboard provides a real-time look into the state of a cluster by displaying the tasks which are current running, the CPU/memory load on the scheduler and workers, scheduler and worker logs, and more. It’s an invaluable tool in helping diagnose the performance of Dask computations. However, part of the real-time nature of the dashboard is that it goes away when the cluster shuts down....

A couple of weeks ago I sat down with Hugo Bowne-Anderson to discuss the ins and outs of Coiled Cloud, a deployment-as-a-service project for scalable computing that we’re building at Coiled. YouTube recording: https://www.youtube.com/watch?v=wKTkjYHCoKo Materials presented: https://github.com/jrbourbeau/st-tour-de-coiled Coiled documentation: https://docs.coiled.io

A couple of notes about interacting with continuous integration (CI) services that I find useful when working on open source projects. Using an empty commit to trigger CI Sometimes it’s useful to open a pull request to simply trigger any CI services for a project. Until recently I had done this by making some small change (e.g. adding a blank line to a README). Then I found out git commit has an --allow-empty option which lets you make an empty commit:...

This post originally appeared in the Quansight Labs blog This post provides an update on some recent Dask-related activities the Quansight Labs team has been working on. Dask community work order Through a community work order (CWO) with the D. E. Shaw group, the Quansight Labs team has been able to dedicate developer time towards bug fixes and feature requests for Dask. This work has touched on several portions of the Dask codebase, but generally have centered around using Dask Arrays with the distributed scheduler....

This week I learned about mocking code behavior when writing tests. This is really cool, so I thought I would write a little about it. Recently, I wrote a Python wrapper for the Open Brewery DB API. At first, the tests I wrote for my Python wrapper included code that actually requested data from the Open Brewery DB external server. This wasn’t ideal because the tests took a long time to run (a few seconds for each test, but this can add up during development when you’re running tests frequently)....